Is velocity overrated in A/B testing?

April 8, 2022

Test velocity. It's often touted as the indicator of optimization success. Simply put, if you can run more tests more often, the hypothesis is that you'll learn more often too.

And along the way, you'll build the muscles that comprise a healthy culture of experimentation.

But is test velocity overrated? On April 7, Kameleoon joined with 4 leaders in experimentation to settle the question. Read the highlights below or watch the replay here.

Ton Wesseling

From the perspective of raising an organization to become more A/B test aware, what is harder: To start with the perfect quality, well-researched, strong hypothesis way of testing? Or is it easier to start testing garbage in, garbage out, and once the system is working raise the quality? From my perspective, it’s easier to start with just the quantity.

Ton Wesseling

Founder, Online Dialogue

It’s easier to start with quantity, then move to quality. If you start with quality, everyone’s going to start criticizing the tests—”wrong hypotheses, wrong insights, wrong, wrong, wrong…” Don’t give that signal. Instead, just move forward. Create happy testers. If no insights arise, then slow down and move to fix the testing.

You get more winners by investing in quantity than complexity. When you receive resources to scale a testing program, the end goal is to maximize the number of real outcomes. You don’t want to maximize the percentage of winning tests. You want to have as many tests as you possibly can that achieve a high win-rate. If that win rate drops, then you should look at investing in quality.

Verdict: Not overrated

Els Aerts

If we teach people that button color testing is a valid way to do A/B testing, are we really implementing culture in the right way? Are we sending the right message if we let people run really bad quality tests to start with and focus on the quantity?

Els Aerts

Cofounder, AGConsult

Don’t incentivize quantity for quantity’s sake. While maximizing your testing potential is important, trying to find the balance between running as many tests as you can and also making sure that you’re running the most valuable tests is important from the get-go.

Quality ≠ complexity. You don’t need super intricate research to run high-quality tests. You can look at a website and, seeing basic data, come up with great testing hypotheses. Therefore, velocity and quality are not mutually exclusive.

Having a plan and process in place is extremely important. Even if you have low bandwidth for testing, if you do not get your planning right, and if you don’t have your next test ready when your first one is running out, you’re not maximizing that.

Verdict: Overrated

Marianne Stjernvall

To learn a lot you have to test a lot.

Marianne Stjernvall

Head of Customer Growth, Coop Sverige

If you’re new to testing, just start with something. It doesn’t have to be the best idea ever, but you just have to keep it going, find data, and start learning about your users. In the beginning you might not find those great big winners, but you will learn a lot.

Create a testing program that suits your organization. Not all tests are created with the goal of generating revenue. In some cases, the goal is to learn about customers, or test software rollouts. It makes sense to have different quantities of types of testing. For example, one portion are quick tests, to just learn about your customers, another would require longer time and research. Then some development tests.G

Get buy-in to start. Maybe you only have one CRO specialist who’s relying on the developers to get tests out. To get up to any velocity, you need buy-in from those developers first.

Verdict: Not overrated

Brian Massey

Are you learning something about your visitors? Are you running a test where you’re removing an obstacle that can increase the conversion rate or the sales rate?

Brian Massey

Managing Partner, Conversion Sciences

Don’t deal in tests, deal in ideas. It’s idea velocity that matters—how quickly you get ideas off the deck. A/B testing is just one way to get to ideas and designing experiments to prove them wrong.

Tests have a cost. A large stack of underperforming tests can get expensive, since they reduce your overall number of conversions. It can get expensive if every 1 in 3 or 4 tests is a “loser”, if your goal is to test arbitrarily fast.

Verdict: Overrated

Don't miss the next Live Session. Follow Kameleoon on LinkedIn to learn about upcoming webinars with the world's leading experts on experimentation.

Topics covered by this article