How to Tell If Your Recommender Will Fail (Before You Spend All That Money On It)

Awhile back, we started exploring how individual interest graphs, powered by Primal’s data service, can be used to improve the performance of recommendation engines.

Our surprising conclusion: Marketers and technology buyers are sold on the promise of personalized recommendations. Unfortunately, they don’t know how to tell recommenders apart, even when they’re built on radically different technical approaches. And the results are mixed, at best. End users are left frustrated, wondering when this promise of personalized recommendations will actually be delivered.

In this post, we’re going to show you how to ask the right questions of your technology provider, to make an informed choice based on business considerations, not technical jargon. We’ll highlight some of the common risks and pitfalls, and conclude with a statement of what to expect from your technology provider. 

DISCLAIMER: My company, Primal, creates data about user interests that can improve the performance of recommendation systems. We don’t build or sell recommenders directly, but we love making recommendations more relevant to individual users. No recommendation systems were hurt in the production of this blog post.

Sorry, Your Recommender Sucks. Yes, Yours…

For context, a recent survey conducted by The Economist Intelligence Unit (EIU) concluded that marketers are confused about personalization technologies.

Some of the problems cited:

• Lack of budget. 50{069d078e67762c42581366636271aa561827dd9d3f7d93a54ca42b15a162055d} of marketing executives said that they have inadequate budgets for digital marketing/database management. We can’t afford expensive experiments and failures.
• Misalignment with consumers on critical issues of privacy. Nearly half of consumers (49{069d078e67762c42581366636271aa561827dd9d3f7d93a54ca42b15a162055d}) are concerned about the threat of privacy online, but marketing executives believe only 23{069d078e67762c42581366636271aa561827dd9d3f7d93a54ca42b15a162055d} of their customers are very concerned.
• Disillusionment among consumers. 70{069d078e67762c42581366636271aa561827dd9d3f7d93a54ca42b15a162055d} feeling jaded and believe “attempts at personalization are superficial”; 33{069d078e67762c42581366636271aa561827dd9d3f7d93a54ca42b15a162055d} of consumers cite superficial personalization as one of their top annoyances.

These aren’t small failings. These are issues that threaten to undermine the whole enterprise of personalization technologies.

What’s gone wrong?

Background: Recommenders Are Based On Some Understanding of the World

When technologists tell you all about their gee-whiz-bang technology, they’ll often overwhelm you with the technical details. (Believe me, I’m my own worst offender in this category.)

When it comes to recommenders, the first question you need to ask is, What is the underlying knowledge that is being used to power the recommendation system? This may seem abstract on the surface, but the technologist should give you a concrete answer.

Is the knowledge is based on some universal or authoritative source? “Our knowledge model was developed by a team of experts from your industry.” The recommendations will reflect the knowledge, as captured by these experts.

Other times, the technologist will point to some external domain of knowledge. “Our knowledge model is derived from this collection of documents from Wikipedia or from that database.” The recommendations will reflect the knowledge that’s encoded in these sources.

Is the knowledge model learned by the recommendation system, through examples? “Our knowledge model is trained using examples drawn from your existing users or people that act like your users.” The recommendations will reflect the past activity of these users.

Question #1: Whose Knowledge Is It, Anyway?

The recommender will be making recommendations on your behalf. If you’re hiring a technology to make recommendations for you, you want a technology that reflects your knowledge and product goals.

Just as you wouldn’t hire an employee whose experience is misaligned with your business, you shouldn’t hire a knowledge technology that represents knowledge that’s misaligned with your business.

Many times, the source of the knowledge is just that, misaligned with your needs. The technology may be driven by social networks, but that network has no knowledge of your business. Or it might be driven by a body of knowledge that is at best tangentially related to your business.

If you don’t buy into the fundamental premise behind the knowledge technology, don’t go any further!

One way to mitigate this risk is to use technologies that have proved themselves within markets that are very similar to yours. This is analogous to hiring employees that have work experience in your industry.

Question #2: What’s the Baseline Performance?

Unfortunately, recommenders are often very difficult to evaluate without an implementation effort.

A common tactic among purveyors of personalization is to claim remarkable productivity improvements in past implementations.

“Our personalization technology has improved user engagement by 200{069d078e67762c42581366636271aa561827dd9d3f7d93a54ca42b15a162055d}!”

While this sounds great on the surface, it can be misleading. What’s the baseline for the calculation? Is the improvement based on an improvement over “do nothing”? That is, is the technology competing against the alternative of using no recommender system at all?

Imagine if you were considering a new car, and the car salesman said, “This car will go 200{069d078e67762c42581366636271aa561827dd9d3f7d93a54ca42b15a162055d} faster than walking!” It doesn’t sound as impressive, does it.

Often times, it begs the question of how you can test it, but your technologist should be willing to put their money where their mouth is. For example, running A/B comparisons with other approaches can give you the perspective you need to make an informed buying decision.

Question #3: What are the data and privacy risks?

Many personalization technologies are premised on detailed data profiles of your individual consumers.

If you’re buying into a technology that requires a large amount of user data acquisition and retention, you need to be very concerned about the privacy implications and assumed risks. Ask the technologist about these risks.

“It isn’t a risk for us; our technology is bulletproof. We have this-that-and-the-other security certification to prove it!”

“We have very high standards and ethics with respect to our use of consumer data. Trust us!”

Facebook can give you 6,000,000 reasons why this is a huge risk. If you think your brand is immune from the risks, you’re kidding yourself.

If you’re sensitive to this risk (and you should be!) you can mitigate the risk by using technologies that don’t need to acquire or retain a lot of user data.

How Does Primal Fit In All This?

As disclosed at the top of this post, we’re not in the business of building recommendation systems, but we are in the business of helping personalization technologies reach their full potential.

If I’ve persuaded you that your recommendation system might need fixing, I encourage you to explore a recommender use case, powered by Primal.

To the questions above, we would answer:

The knowledge we model relates to the specific interests and intentions of your end-users. Primal-powered applications are driven by the knowledge of your individual end users; their unique interests and intentions. Our goal is for your end-users to feel like the experience is unique to them, as individuals.

In evaluating the performance of our data, we encourage and support A/B testing. Our technology is unique in its ease of integration. You can have a Primal-powered solution up-and-running quickly, with no significant time or financial outlay. And since Primal runs on top of your existing assets, there’s no ambiguity about the baseline performance against your existing search, recommendation, or personalization technology.

And to the question of user privacy, Primal is unparalleled in its sparse consumption of user data, and we have no need to retain that data on your behalf. We don’t burden you with user privacy risks because we don’t burden you with large quantities of user data.

Getting Started

If you’d like to build your own Primal-powered recommendation system, check out our developers site to get started.