How can I optimize my store's product recommendation engine?

This interview is part of Kameleoon's Expert FAQs series, where we interview leaders in data-driven CX optimization and experimentation. Michael Schrage is an expert in the behavioral economics of iterative innovation with an emphasis on the design of rapid experiments. He is a research fellow with MIT Sloan school’s initiative on the digital economy and the author of multiple award-winning books, such as The Innovator’s Hypothesis and Recommendation Engines.

How can an organization gain a competitive advantage by improving their product recommendation engine?

This is precisely the wrong question. Your focus should be on improving recommendations, so customers and prospects like, trust and respect your recommendations. In turn, they’ll buy more from you and happily boost your NPS. That design sensibility and business perspective works for mutual benefit. Meaningful recommendation differentiation comes from what you stand for and the quality of your advice.

Think strategically, not tactically. Ask yourself, ‘What is the true purpose of your recommendations? What is the point for you and your customers?’ While there’s nothing wrong with wanting to sell more, don’t kid yourself—if that’s your recommender’s real mission, your customers will react accordingly. Your customers aren’t fools.

Sustainable competitive advantage goes beyond better data and smarter analytics; it comes from making people feel inspired by your recommendations and advice. Are you measuring that? Great recommenders and great recommendations should be integral to great CX. That’s as true for Amazon and LinkedIn as for TikTok and Spotify. Otherwise, recommenders are little more than algorithmic gimmicks to get people to buy more stuff. That’s not sustainable.

What data or segmentation should I avoid to ensure our product recommendation algorithm isn’t discriminatory or creepy?

Well, if you’re a B2B business, that’s probably not a burning issue. But no one wants to be lumped in with ‘people like you’ who are creeps or jerks. If you take UX seriously, recommenders create many opportunities to make people feel curious, special, and good about themselves. If recommenders say, "Parents like you…." instead of "People like you…" or "Adventurers like you…." rather than "Travellers like you…," you’d be using language that invites people to look at themselves and their choices in more intimate ways—rhetoric matters. Nobel Prize-winning behavioral economics research from Daniel Kahneman, Richard Thaler, and others affirms how you ‘frame’ choices powerfully influence decision behavior. So you want to think hard about what kind of customer segmentation and categorization gets folks engaged rather than concerned. Again, those are fairly straightforward ‘customer journey’ and CX design issues.

Yes, we should assess how biased data and algorithms might inadvertently or even deliberately discriminate against people with different genders, ethnicities, and identities. Be familiar with MNAR—that horrible technical acronym for data Missing Not At Random— because, by definition, those datasets will be biased. Biased data can lead to biased outcomes, and biased outcomes can lead to counterproductive outcomes. Again, this largely depends on what you want from your recommenders.

First and foremost, obey local laws.

Your organization’s purpose and mission statements should also influence your recommender design, i.e., you may algorithmically "weight" sustainable products and services more highly or choose to value and thus recommend minority-owned businesses or partnerships more frequently. Fine tuning recommenders to compensate for—and inject—bias into your digital advice is becoming a new normal. Do so intentionally and transparently. If you, your data science, or business intelligence teams can’t explicitly identify your dataset, MNAR, and algorithmic biases to legal counsel, you’re not doing it right.

What is a “filter bubble,” and how might it impact my ecommerce business?

Filter bubbles affirm and facilitate confirmation bias. With apologies to Eli Pariser, who coined the phrase, filter bubbles are what you get when your recommender primarily reinforces people’s preferences, prejudices, and preconceptions. ‘More of the same’ recommendations dominate, and unless you are a politics junkie, a zealot, and/or an ideologue—that gets boring pretty fast. Filter bubbles are bad for business.

Academic research and real-world empirical experience over the past decade demonstrate that the most effective recommenders are novel, diverse, and serendipitous. You’re not going to excite, inspire or motivate people with SO-SO, same old recommendations. If your recommendations don’t pleasantly surprise your prospects, they’re failing.

Again, the core business issue revolves around what role you want recommendations to play in a customer/client relationship. For example, Stitch Fix serves two types of customers:

1. People with a set ‘style’ who are always looking for apparel that reflects and respects that.
2. People who like to push their fashion boundaries and make edgy choices.

Their motives, fashion sense, and willingness to explore are different. You need recommenders that can affirm a known sensibility from a novel angle and twist a known boundary at an unexpected angle. Did Stitch Fix do tons of experimentation and analytics to build its business on these insights? You bet.

For me, this describes one of the great and important tensions between ‘personalization’ and ’segmentation.’ Some folks want inspiration, insight, and alignment from ‘people like them;’ others want to be as unique as humanly/digitally possible. Recommenders that blow filter bubbles undermine this vital process of self-awareness and self-discovery.

What elements of consumer behavior or behavioral economics should I consider when developing product recommendations?

This question makes me smirk. The only possible answer is "all of them."

You want to understand everything you reasonably can about what informs, influences, and impacts your customers around choice. That’s why you gather data and do analyses. You crave actionable insight into customer and prospect behavior.

For me, the best way to approach this is through the fantastic framing of "choice architectures," first described by Cass Sunstein and Richard Thaler in their best-seller Nudge. Choice architecture is best understood as ‘the practice of influencing choice by organizing the context in which people make decisions.’ How choices are designed, sequenced, packaged, and presented powerfully shapes people's decisions. What kind of ‘choice architect’ are you? Who ‘owns’ choice architecture in your organization?

Every serious CMO, CRO, CCO, and UX designer needs more than a passing familiarity with choice architecture literature. I’d argue that choice architecture forces serious organizations to rethink and redesign UX, customer journey, conversion, and recommendation experiments and strategies. Virtually every time I’ve facilitated a choice architecture design workshop with CX, marketing, and data science folks, the result has been better collaboration and more intentional efforts to boost CLV.

Choice architectures, like recommenders, are a means to an end: not just better choices and recommendations but better and more profitable outcomes for the firm and its customers.

What impact do product recommendation engines have on consumer decision-making?

One of my favorite stories in my book came from Amazon’s Greg Linden, who pioneered the company’s nascent recommenders. He recounted executive resistance to the very idea of recommendation engines. Their concern—fear, actually—was why distract people with recommendations when they’re in Amazon’s sales funnel? Never get in the way of someone making a buy!

Fortunately for Greg and Amazon’s market cap, Bezos’ Amazon had a strong experimentation culture. Instead of deferring to executive wishes, they designed a set of A/B tests to track if recommendations really would distract people from purchases. Instead of items abandoned in shopping carts, the experiments found that, on average, recommenders uplifted sales. The benefits of recommendations significantly outweighed their cost and risks.

Having written a book on experimentation and recommenders, I can confidently assert that recommendation engines offer an embarrassment of experimental opportunities for organizations that want to learn about their customers and what motivates them. I’d argue the world’s most successful digital firms (Alibaba, TikTok, Amazon, Uber, Spotify, Google, etc.) all create a virtuous cycle between experimentation and recommendation. To wit, how can we experiment with recommendations, and then what recommendation for experiments do those outcomes/results suggest? To me, enabling those kinds of data-driven virtuous cycles is what digital transformation is all about.

I heard you like reading books about improvisational theater and cartoon animation. Have these subjects influenced your thinking around innovation?

Ha! I’ve loved animated cartoons ever since I was a child. I was a huge fan of Walt Disney and Chuck Jones and bitterly resented the sad fact that I couldn’t draw. That said, thinking in terms of ‘storyboards’ and ‘illustrated narratives’ and how it took 24 frames a second to give something the moving illusion of life had an enormous impact on my thinking. How do we measure, manage and illustrate the ‘moments in time’ that matter most? That’s what I got from my obsession/infatuation with animated characters and worlds. In an earlier life, I did a piece on Disney’s ‘Tron’—the pioneering ‘computer graphics’ movie—for the Smithsonian magazine. Gosh, I’m old.

Growing up in Chicago, I got involved in Second City–the Improvisational comedy troupe–during my high school years. I met and interacted with people like Del Close and Shelley Long. I never acted/improvised on stage, but I did have a flair for suggesting scenes. I shut up, paid attention, and learned a lot about collaboration, creativity, generosity, ensemble behavior, improvisation, humor, and live audiences. You can think you’re hilarious, but does the audience laugh? How does your energy shape the audiences? How does the audience’s energy influence your own? Is anybody surprised I ended up writing books on experimentation, collaboration, and advice?