Kameleoon’s take on performance and rankings
For over a decade, Kameleoon has led the experimentation space by prioritizing performance. Our commitment to speed and seamless customer experiences earned us consistent top rankings. Recently, however, a sudden change in the ThirdPartyWeb rankings—driven by a vendor’s methodology update—caused a drop in our score. Let’s explore what this means, and why Kameleoon remains the best choice for high-performance experimentation.
A shift in methodology
On September 18, 2024, ThirdPartyWeb updated its rankings, showing that Kameleoon’s performance score had dropped from 245ms in August to 913ms in September. This was surprising, as we had made no changes to our platform that could explain such a drastic shift.
Upon investigation, we discovered that a competitor introduced changes to the ranking methodology. The specific change can be publicly viewed here. While this modification was approved, it raised questions about bias and ranking integrity. Kameleoon believes performance metrics should remain independent, and our priority continues to be real-world results over artificial rankings.
Understanding the Full Picture of Performance
Although ThirdPartyWeb provides useful insights into CPU impact from third-party JavaScript, it doesn’t capture the full picture, especially for experimentation platforms like Kameleoon. Real performance depends on balancing speed, accuracy, and delivering a seamless user experience.
This is where Kameleoon strikes the ideal balance: Our experiment snippet is designed to optimize both performance and user experience. If we were to disable our anti-flickering engine, we would be significantly faster, improving our ranking on performance charts. However, this would come at the expense of reliable test results. Our commitment remains ensuring customers receive accurate, actionable insights, even if it means a slightly higher resource usage to avoid flickering and maintain test reliability.
We tested disabling our anti-flicker engine and saw an immediate improvement in our ThirdPartyWeb ranking. However, this comes at the cost of test reliability—a trade-off we won’t make. Our priority remains delivering accurate, actionable insights for our customers, not chasing superficial metrics.
Combined with our anti-flicker engine, we eliminate flickering issues, ensuring smooth and reliable experiments. Additionally, our Deferred Experiment Loading feature offers flexibility, allowing customers to defer certain experiment configurations—like pop-ups or elements below the fold—that aren’t needed immediately upon page load. This helps optimize performance without sacrificing user experience.
Enterprise-Level Performance
For enterprise teams running large-scale experiments, performance is crucial. Kameleoon’s snippet is built with cutting-edge technology, making it asynchronous and fully cached, ensuring page loading is never blocked, even during CDN incidents (99.99% uptime). Our compression techniques (combining Typescript and Brotli) deliver a lightweight 28.4KB script for minimal resource load.
Additionally, features like Beacon calls (204) reduce network requests, and our Deferred Experiment Loading allows selective experiment inclusion for faster initial load times. All of this guarantees seamless performance without compromising functionality, making Kameleoon the ideal platform for enterprises focused on speed and user experience.
Looking ahead
Kameleoon has always been about delivering top-tier performance where it matters: in real-world use cases, not just in artificial benchmarks. While rankings may fluctuate, we’re continually improving our platform without compromising the accuracy or reliability of our tests.
ThirdPartyWeb’s open-source model allows us to analyze how rankings are generated, and we’re confident that with a few refinements, we’ll reclaim our rightful spot at the top. In the meantime, our commitment to customer success remains unwavering. If you’re looking for an experimentation platform that balances speed, accuracy, and reliable results, reach out to our team or explore how Kameleoon can help you achieve your business goals.
Stay tuned for more updates!
