Should you run an A/A test?

If you run A/B Tests (or just got into it) you might have come across what is called an A/A test and wondered what the heck it is and when you should (if at all) run it. Well, I got you covered. Here is a short and sweet answer :)

What’s an A/A test

An A/A test is an experience where the variation is an exact copy of the original page.

Why do an A/A test

There is actually not much use to an A/A test except making sure your A/B testing solution is set up properly and the results you get are indeed accurate. If everything is right, you should get virtually the same conversion rate at the end (since both variations are identical). Should it not be the same, you got a serious problem on your hands.

How come your results can be inaccurate

Usually, results are very close. But it can very well happen to get a winner in your A/A test, even though you KNOW it’s not possible. What could be causing this mishap?

Your A/B Testing solution has a problem (it could be badly set up, not recording data properly, …).
You have a false positive (details on this in our article on interpreting A/B tests results), which with a statistical significance of 95% has 5% chance of happening.
You ended your test too early and got a fake result (if you need help on when to stop your test, we touched on this here).

Tip: A/A/B test

A/B Testing takes time to do properly, so if you have enough traffic, instead of “wasting” your time on an A/A test, you can do an A/A/B test. You set up an additional variation identical to your control so you can check the validity of your results while doing an actual experiment at the same time.

In the end, you should only run an A/A Test if you’re getting funky results (or need reassurance everything is set up properly).

Topics covered by this article