Shanelle Mullin & Alex Birkett on A/B Testing & Personalization [Interview]

May 18, 2017

To get some insights on A/B Testing & Personalzation, we had a chance to interview Shanelle Mullin (Content/Growth marketer) and Alex Birkett (Growth Marketing manager at CXL). Among the CRO aficionados, one blog rules all the others: CXL (or you might know them as ConversionXL). They've historically been making some of the best content on CRO and show no signs of slowing down as they've launched online courses & certifications.

The interview

Question: How long have you been working at CXL and what does your job consist of?

Shanelle: I’ve been working at CXL for about two years now. I do top-of-the-funnel content and growth. So, that’s everything from writing blog posts to selling conference tickets. Anything that relates to attracting attention, collecting emails and sending people down the funnel to CXL Institute.

Alex: I’ve been at CXL for two years. I do product marketing for CXL Institute and write some blog posts for the CXL blog.

Q: So CXL is first a world class CRO agency and blog, but you guys are now also providing MOOCs & Certifications. What made you go in this direction? Can you share a couple of interesting things you learned through CXL Institute’s implementation and management?

Shanelle: The optimization industry is very young, according to our most recent State of the Industry Report. Almost 20% of respondents have been working in their optimization role for less than a year, most companies still don’t have a structured, documented optimization process, and 26% of teams don’t even meet regularly to talk optimization and growth. When the Internet is full of optimization and growth bullshit like “101 things to test right now” and “99 growth hacks to try”, you realize there’s a big opportunity for quality training.

Alex: In addition to what Shanelle said about the lack of good training on data-driven marketing, there’s sort of gap left by universities, too. I loved my college education and wouldn't change it for the world, but it didn’t necessarily prepare me well for the fast-paced and increasingly technical world of digital marketing. More importantly, graduate programs for marketing are more expensive and arguably less tactical than undergraduate programs, so I feel (and I’m not speaking for CXL or for our mission here, just personally), that we’re filling an important gap in how people look at further education and training. You can become a world class marketer, network with other world-class marketers, and have a reputable company’s certificate to prove you know your stuff. And you don’t have to go broke to do it.

Q: How did your agency become so well-known, what separates you from the other agencies out there?

Alex: I think our agency was well-known by the time I started at CXL and it was mainly because we freely gave away such detailed and helpful content. Peep’s blogging was different than the rest out there because he clearly knew his stuff and didn’t jump on goofy trends or do it for the SEO or virality. There’s an authenticity there that we still try to maintain. Our agency got so well-known because we freely gave away detailed & helpful content.

Q: Before we dive in: for the marketers out there not so familiar with A/B testing & Web personalization, how do you define them and what do you use them for?

Alex: A/B testing is a controlled online experiment. Every decision we make is made under a large amount of uncertainty, and A/B testing is a way to mitigate risk while encouraging innovation. It seeks to explore potential experiences and exploit the winning variation to maximize both insights and ROI.

Shanelle: Web personalization is the real-time individualization of a site to suit each visitor’s unique needs and guide them through a custom conversion funnel. People are drowning in information and options. Personalization reduces the amount of information and the number of options to help guide visitors through a funnel that’s designed just for them and their needs. According to Evergage’s most recent industry report, 88% of marketers think that leads and customers expect a personalized experience. Yet, only 45% think marketers are getting personalization right. This is an important aspect of optimization to pay attention to in the coming years.

Q: What are your go-to ways to identify optimization opportunities?

Shanelle: We use the ResearchXL model, which covers: heuristic analysis, technical analysis, web analytics analysis, user testing, qualitative surveys, and mouse tracking analysis. My personal favorite is web analytics analysis. I used to be the type of person who would open up Google Analytics every month, record a few out of the box metrics and prepare a report. So, I know it’s a real issue. There’s so much hiding below the surface in Google Analytics (and any other analytics tool). Diving into the data with a specific problem to solve or question to answer is straight up fun, but it’s also super insightful. Learning to slice and dice your data in new and interesting ways to move beyond that out of the box experience is incredibly valuable. Plus, web analytics analysis covers a lot. Cross-browser and cross-device testing, broken links, site speed, holes in your funnel, mining internal search, etc.

Alex: We use ResearchXL. I like user experience testing and digital analytics the best for optimization opportunities.

Q: About A/B testing, how do you use it to grow your own website & blog? How often? How many tests do you have running right now?

Alex: We haven’t run many content-focused tests recently, but we’re always running at least 1 or 2 tests around CXL Institute on-site. Whatever the traffic allows. I also run a ton of experiments with our email automation - for leads, users, churned customers, etc. That’s probably my biggest experimentation focus area right now.

Q: Could you share details on an A/B test that worked really well?

Alex: Social proof unsurprisingly performed really well on our CXL Institute landing page.

Q: What do you think makes a good A/B testing tool?

Shanelle: For me, it comes down to three factors…

Is it easy to use or unnecessarily complex / technical?
Does it promote valid results or does it set testing newbies up to fail?
Does it integrate with other tools easily?

Alex: Easy integrations are important. Also, I like to own my data, so the ability to pull the raw experiment data is great. Finally, I don’t like to use what I consider a “black box,” especially when it comes to more predictive-based tools. But in general, I like to know what’s going on under the hood, or at least have access to the raw data.

Q: What’s something you’d like the current A/B tools on the market to have that’s missing?

Shanelle: Deep integration with analytics tools is what I think has been missing. If the A/B testing tools and analytics tools could work together more closely, it would allow for more advanced targeting, more advanced reporting, more advanced conversion tracking, etc. Some tools are beginning to move in this direction and I’m really excited to see how the concept evolves.

Q: Web Personalization has been announced as THE next thing for quite some time now, what do you guys see with your clients?

Alex: It’s technically very easy to do personalization. The most difficult part of it is the complexity and management problem that comes with a variety of targeted experiences delivered with a certain assumption based on hopefully an adequate amount of data. And all wins are perishable, so when does a targeted experience become ripe for experimentation? And by what assumptions are we delivering experiences using predictive targeting? These aren’t easy questions or problems to solve, and better technology doesn’t necessarily solve them. Vendors and agencies alike need to provide education on what makes up a good mature personalization program.

Q: What main reasons preventing companies to implement personalization did you encounter? How did you overcome them?

Shanelle: Most marketers have content personalization, not web personalization. What that means is that personalization has become synonymous with “the right message at the right time to the right visitor”. While that’s certainly part of it, that’s not the be-all-end-all of user experience. Most marketers have content personalization, not web personalization. You have to change your mindset from “what message will I deliver and when” to “how can I individualize the actual user experience”. Consider how you can…

Provide more context to visitors.
Provide an experience that’s less machine-like.
Help visitors find what they’re looking for faster.
Make it easier for the visitor to do what you want them to do.
Reduce the amount of irrelevant or secondary information.

Q: What importance do you think personalization will have in businesses optimization strategy (if any)? A/B testing is almost a commodity now, do you think web personalization will take a similar route?

Shanelle: I think the importance of, expectations of and praise for web personalization will continue to grow while adoption continues to lag behind. Marketers are still really struggling to move beyond content personalization and make web personalization work for them in a meaningful way. I think it’ll be at least a couple of years before personalization catches up to where A/B testing is now.

Alex: I think there needs to be a clearer picture of the ROI of personalization. At the margins, what does an X% effect mean over time on Y segment over time? What’s the opportunity cost, if any? It’s not too difficult to at least conceptualize these things when it comes to A/B testing, but the benefits of personalization, although large and apparent to those who have successfully enacted personalization, aren’t as obvious.

Q: Do you use web personalization to grow CXL website and blog? If yes, could you share details on a personalization that worked really well? If no, why not?

Alex: We do some really basic behavioral automation with our emails as well as some dynamic messaging. Nothing out of the ordinary. We’re currently working on a more robust personalization program for our leads, but it’s still in the works.

Q: Any parting words for marketers out there getting started with user experience optimization?

Shanelle: Define a structured, documented process early on and run with it. Optimization isn’t about tactics, running 101 tests or throwing things against the wall to see what sticks.

Alex: Experimentation is the safest way to push the boundaries and innovate in product design, marketing, messaging, and other business functions. And it’s fun! Convince your boss to invest more in optimization and you (and your business) won’t regret it.

And that concludes our interview!

Share it with your colleagues and if you want us to interview someone in particular who'd have interesting thoughts on A/B testing & Personalization, comment here or tweet us @kameleoonrocks! If you want more from today's interviewees, here are their latest articles: