[Product Release] New A/B test result page
Indications of required action
We recommend actions on your A/B test, based on indicators relevant to your business so you can view them at a glance! To give an example, Kameleoon continuously analyses the time needed to obtain trustworthy results you can base your decisions upon.
If results aren’t ready yet, we’ll give an estimation of the time it‘ll take to identify the winning variation.
Automatic audience recommendation
You are probably aware that A/B test performances can vary depending on different visitor segments. In some cases, a variation that gives good results with a segment A won’t prove successful with a segment B. Showing the same variation to both would be sub-optimal. Our proprietary machine-learning algorithms enable you to turn all your tests in personalization opportunities.
They continuously scan and analyze your tests’ performances and identify the variations that work best for a given visitor segment. In 1-click, you can tailor what user experience you give to different segments or create new experiences for segments with high growth potential. This feature will be available very soon. Ask for a demo for detailed information.
New DataViz and enhanced results segmentation
The new result page includes a powerful dataviz and numerous segmentation possibilities perfectly fitting your analysis needs.New indicators for better decision-making
We added new A/B test performance indicators to our results page:- Confidence interval: We don’t just indicate the average improvement rate based on registered conversions. We also show the confidence interval for this improvement rate, indicating the probability of your improvement rate to be situated within the given interval.
- Results reliability: As a frequent A/B tester you know that your tests can’t be considered done as long as the confidence level hasn’t stabilized over time. This graph is still available, we improved it by adding a visual indicator showing the confidence level’s stabilization. Stability has been reached when all three boxes are highlighted.
New improved graphics
Many of our customers and users track several goals for the same test. That’s the right thing to do if you want to understand your optimization hypotheses’ impact on the user journey and your business goals. We gave the comparative view (the “cobweb view”) a make-over.
At a glance, you can see the impact your variations have on all of your goals. We also raised the number of available graphs: conversion rate, number of visits, number of conversions, confidence level, etc. And that’s not all: you can also break down results for your goal and view them in one graph. More improvements are on their way. Over the next months, we’ll add :
- Traffic allocation per variation
- Evolution of traffic distribution among variations over time
- Integration of heatmaps into result pages, and comparison
Advanced result segmentation and filters
The new result page offers more than a simple breakup of results by goal or new/returning visitors. You now have the possibility to filter or break up results using over 25 criteria, for in-depth understanding of different elements’ impact on your conversion rate (such as weather, traffic source, time of visit, page visited, etc.).
Differentiate between unique vs. multiple conversion and visits vs visitors
If you’re an e-merchant, for instance, a 10% increase in “Added to cart” value is completely different when looked at from the “visitor” angle as opposed to a “visit” angle. If your business model is based on affiliation or ad revenues (comparators, media, etc.) and you want to optimize click rates, it may be interesting to compare multiple conversions (per visitor) with the overall number of converted visits.Manage your tests in real time
Real time technology is at the very heart of our work, regarding the analysis as well as the activation of your data. Our statistics engine is improved continuously and keeps the inherent latency of data processing to the bare minimum. On the result page, real time provides the following advantages:- Immediate access to data upon A/B test launch: If you’re used to working with analytics tools you have certainly faced an empty page, waiting for your data to come in. With our new result page, no need to wait for hours, your latest data is available almost immediately.
- Real-time alert on test behavior: We know that you can’t constantly watch your dashboard, so we created alerts to keep you updated on the most important changes on your test. These alerts can be adapted to your specific needs. You can ask for notifications as soon as a variation turns out to be a sure winner, when the confidence level you defined is reached, when there are conversion anomalies or on any information essential for you.
- Viewing test performances even when out of the office: We know that you’re not always at your desk. The result page will soon be available on mobile devices, too. Which makes performance tracking even easier.
- Results adapted to your business goals: Our statistics engine’s major added value is its capacity to adapt to your specific needs. To give reliable answers to your optimization hypotheses, two statistical methods are used: the Frequentist and the Bayesian approach. Then former works well on tests with important traffic, the latter offers faster results even with less traffic, using probability, thus accelerating your decision process. However, you have to wait for the confidence level's stabilization before taking any business decisions!
Improved UX
To make the new result page an easy-to-use tool in your day-to-day work, we grouped information and features according to relevance: Parameters impacting the test data can be found on top of the page, everything regarding test setup is group in the right-hand panel.
Manage your tests directly via the result page
Edit traffic distribution or a variation, launch a simulation (targeting and control) or preview directly via the result page. No need to return to the graphic editor! We also added a timeline to each test, offering to track any prior action on the test.Easier sharing of A/B test results
Sharing test results internally is very important for the development of an optimization culture and the involvement of all your teams. Up to now, users needed to log into Kameleoon to view results. We added a public, password-protected URL that can easily be shared. You can access the beta version of our new result page via your Kameleoon dashboard. Watch out for all the new exciting features that will be added over the coming weeks.
