This interview is part of Kameleoon's Expert FAQs series, where we interview leading experts in data-driven CX optimization and experimentation. James Flory has worked with numerous healthcare companies and others in regulated industries, so we wanted to find out how those worried about data privacy and compliance can do risk-free testing to achieve growth through optimization.

Hi James! For those that don't know, could you please tell our readers a bit about yourself and what you do?

I’m the Director of Experimentation Strategy at Widerfunnel. We’re a CRO and experimentation agency focused on providing an evidence-based approach to experimentation. We help organizations all across North America improve their digital experiences and their business decisions through rigorous experimentation systems. 


As Director of Experimentation Strategy, I lead our team of strategists and ensure the continued quality and effectiveness of the strategic experimentation services we provide to our clients. I also lead a few accounts of my own to keep a pulse on the industry and the services we are providing.

Experimentation and staying compliant

What is one A/B test you ran which, in terms of results and insights, especially surprised you or your client?

A good number of the A/B tests I’ve run over the years have had surprising results, that's what keeps me interested and challenged.

One such test that stands out is the complete checkout redesign experiment I conducted with an energy provider. This particular experiment concluded with a completely flat result—no change to user behaviours or signups whatsoever. With the very same client, we ran a simple CTA copy change experiment that resulted in an astonishing 74% increase in click-through-rate and a 24% increase in signups on their primary funnel’s conversion CTA.

Download the report to see how HCOs are using data insights to improve patient experience.

What stops teams and companies from scaling up their optimization programs? Even if they’ve tried before, and failed, what do you think would help them scale up?

There are a diverse range of reasons organizations struggle here.

There’s the “too much, too fast” problem - where organizations get too excited and want to test everything and as a result they end up with a messy, underwhelming program. These organizations will often run a multitude of small iteration copy tests that aren’t showing any impact. Not everything should be tested, and part of building a successful experimentation program is the knowledge and experience to know what constitutes a successful experiment design, and what does not. These experiments ultimately must move the needle one way or the other in a meaningful way, and a “too much, too fast” approach will often stop this from occurring.



Another obstacle teams commonly encounter is internal politics and friction to change within an organization. Experimentation has a tendency to ruffle feathers, especially in organizations with a long established “what we say, goes'' mentality. Data and test results that challenge decisions (and egos) tend to create more enemies than friends.

Finding pockets of acceptance by starting small and building a case for experimentation, especially among key stakeholders who face challenges that can be solved with optimization, is one tactic that I’ve seen help nurture a culture of experimentation within an organization. Often times this helps build awareness and new allies which has cascading benefits with a company's ability to embrace change.

The list goes on for reasons behind organizational struggle. Some other relevant challenges I’ve seen include resource issues, data issues, technological challenges, as well as real world traffic volumes and conversion rate constraints. The list really does go on! As their trusted partner, I just keep at it and help them through one challenge at a time.

You've worked with several health and wellness brands, which often struggle with data. How would you help someone feel less overwhelmed about data when it comes to optimization?

Interesting question. I think primarily focusing on the actionable data is the best place to start. By focusing on data that’s directly related to the goals of a user's digital experience, and then drilling it down even further to what's actionable given HIPAA and GDPR regulatory constraints, provides a great foundation to build off of. Many organizations get caught up in a lot of backend narrow customer data that in the grand scheme of things won’t really help them truly optimize a user’s experience. You’ve really got to mature to that level of efficacy and instead focus on realistically identifiable and actionable data that you can utilize in your digital experience optimization efforts, within regulatory guidelines.

To find out how banks can use data securely while making experiences to attract and retain customers, see the report below. 

Some industries blame regulatory concerns for why they can’t optimize as aggressively as they would like. Is this anxiety merited?

I think it's definitely merited. These organizations are heavily regulated for a reason. They handle sensitive and confidential personal information for their users that they want to ensure is treated with the appropriate level of care and protection. That said, there is a lot of non-sensitive and anonymized data that these organizations can work with that they may be overlooking. Anonymized IP addresses and hashed user IDs are pretty common features in tools now, and aggregating this anonymized data to optimize an experience doesn’t expose any PII risks for most organizations. You can then use aggregated behavioural data to optimize effectively for an enhanced user experience that their competitors will be missing.

Have you ever encountered a situation where you wanted to use PII but couldn’t because of regulatory restrictions or some security risk? How did you solve it?

I have, and one of the ways I’ve been able to address it is by leveraging anonymizing features in data collection tools. I’ve also successfully used the blur or obscure fields options in screen recording and heat mapping tools. We’ve had to get creative using proxy data for certain things but ultimately I see a lot of opportunity to work with non-identifiable data and still conduct some impactful experiments. I’ve consistently found that the most glaring issues throughout organizations that have a lot of digital experiences are not the result of any PII related data or interactions.

Pay it forward

What is one book not directly related to optimization that you would recommend to CROs?

I’m a big fan of behavioural science books. Understanding how people think and make decisions while interacting with the world around them actually has a lot of practical application in the world of CRO. Some of my personal favorites are Influence by Robert Cialdini, Thinking Fast and Slow by Daniel Kahneman, Predictably Irrational by Dan Ariely, and Blink by Malcolm Gladwell. I’d highly recommend them all.

You’re at a conference (remember those?)—what’s your go-to networking strategy? Any icebreakers?

Bad idea to ask a data guy for tips about social interaction. I’m a fan of non-work related ice breakers. Chat about the venue, the food, and share casual personal facts before you dive right into the networking talk.

Who do you think deserves to be featured for their work in experimentation?

I’m an advocate for all the underdogs out there championing data and experimentation in their organizations. If I had to make a special mention, I’d say James Hillin over at Direct Energy has been a real experimentation advocate for years and has helped champion some pretty fantastic experiments.