Unlock Smarter Decision-Making with Sequential Testing

We are excited to announce a true game-changing feature — Sequential Testing. It helps you check how things are going with your experiments without the usual wait. It's like a faster, smarter way to make decisions based on your experiments.

The pain point:

Peeking is a significant concern in the A/B testing industry. It occurs when experimenters check the data multiple times during the duration of the experiment, looking for statistically significant results before the sample size is sufficient. Unfortunately, the traditional statistical framework for A/B testing is only applicable when deciding on experiment outcomes at a single predefined point, typically when a predetermined sample size or duration is reached.

If the treatment effect, which represents the change in outcomes (such as conversion rate, click-through rate, or revenue) between visitors exposed to the control group (receiving the current version of a page) and the treatment group (receiving the new version being tested), is stronger than expected, there may be a temptation to stop the experiment early.

The solution:

This is where Sequential Testing comes in. It is our way of making this whole waiting game with experimentation much easier. Now, you can check your results whenever you want without making mistakes. It's like having a peek at your data without messing up the reliability of your results. Here's how it works:

Continuous monitoring, valid confidence intervals:

With Sequential Testing, users can consistently monitor their experiments without the need to wait for a predetermined sample size. Achieve valid confidence intervals for the average treatment effect, regardless of how many times you check your results.

Alerts for early intervention:

Sequential Testing offers alerting capabilities. It allows you to set up sequential reliability alerts to stop a test early if it is reliably underperforming. This ensures that you can take corrective action promptly, preventing unnecessary resource allocation to underperforming strategies.

No more guesswork with sample sizes:

Sequential Testing liberates users from the burden of predefining sample sizes, allowing for a more agile and adaptive testing approach.

Basically, Sequential Testing lets you make smart decisions quickly without messing with the results. It takes away the problems that usually come with regular testing, making experiments faster and easier to handle.

Sequential Testing lets you keep an eye on things whenever you want, get reliable info, and even get alerted early if things aren't going well. No need to worry about guessing the right sample size beforehand. It's a simpler, more flexible way to run your experiments and make decisions with confidence.

Want to learn more about Sequential Testing and how it integrates with our other powerful statistical tools? Read on here.

Join our Experimentation Community

We're excited to create opportunities for discussion and collaboration among experimentation professionals. Stay connected with us on LinkedIn for invitations to our ThinkAhead webinars, virtual live sessions, and updates on in-person events.