Dos and don'ts of outsourcing A/B test development
The value of optimization is becoming increasingly clear. Companies—in every major market—that have a strong culture of experimentation are 36% more likely to report fast growth than those that don't. As A/B testing and optimization programs mature, the challenge becomes less about getting buy-in, strategy, and processes and more about scale and velocity. Simply put, mature programs want to move from 1 to 100 tests a year because of the ROI A/B testing delivers. The challenge then becomes how.
Outsourcing your A/B test development could be the answer.
But how do you decide whether outsourcing A/B testing makes sense for you, what tests to outsource, and when and how to do it?
This guide shares insights from some of the best A/B test developers on the main considerations when thinking about how to outsource A/B testing.
1 When to think about A/B test outsourcing
First off, should your company even think about outsourcing A/B testing and optimization programs? Abdullah Mashuk, the founder of Brillmark LLC, says it probably should. "Most of our clients find outsourcing test development extremely important in order to execute CRO and other marketing campaigns on a timely basis. Generally, their internal development resources are busy with engineering priorities, so they can execute on marketing priorities."
Of course, that's assuming your company has internal development resources. Tasin Reza, COO of EchoLogyx comments: "One of the biggest challenges I had at my previous company was that after finding good developers, and getting them to the point where they understood why we were coding in a specific way, they'd leave and get a job in full stack development. That wasn't very efficient."
That experience led to him starting to outsource to a couple of engineers in Bangladesh, to building a bigger team, and then to joining EchoLogyx to solve the outsourcing issue for other companies.
While supplementing an internal development team is one reason to outsource, Mashuk adds that there are a lot of other scenarios where A/B test outsourcing makes sense. Some of these include:
- You have to run complex tests, and need experts to help you
- You don't have the budget to hire a full time developer for marketing
- You're unfamiliar with advanced a/b testing tool features
- Your testing volume tends to fluctuate
2 What to outsource - simple or complex A/B tests?
Once you've decided that A/B test outsourcing is the right move, then the next issue is whether you should outsource simple tests or more complex ones. Brillmark's Mashuk says:
"We believe both simple and complex tests should be outsourced as internal teams become more focused on the website this way. Additionally, the outsourced A/B team can later implement the winning variation to the site more easily."
Reza of EchoLogyx agrees: "Some clients only give us complex tests to develop, but it's on a client by client basis. When we work with direct clients, for example, musicMagpie, we are pretty much developing all the tests. Some of them are simple, some of them are complex."
Right now, Brillmark isn't doing server side testing or in-app (mobile testing). EchoLogyx is, but says it's a small percentage of overall testing. "One of the reasons is the workflow effect of doing server side testing because the team needs to have access to their codebase and go through that pipeline. There's also the timing—the time it takes to develop client side tests versus the time it takes to develop server side tests is probably quicker."
3 Benefits of outsourcing A/B test development
Outsourcing A/B saveS money
Outsourcing A/B test development has a number of benefits, one of which is cost savings, says Reza: "The cost was one third of how much I would have had to pay in a place like London, without losing that quality aspect. I was able to get really senior engineers who are extremely sharp. They know what they're doing with really quick communication."
A real improveMent in efficiency
Mashuk says efficiency and scalability are other huge wins. "You get a team of experts who can build tests rapidly and efficiently. Clients can easily scale up or scale down the development resources according to their needs."
According to Brillmark, other benefits include:
- Harnessing the skills of experts
- Reducing costs through more effective testing
- Using the best tools
- Making it easy to pivot if you need to change strategy
- The ability to focus on growth
- Using the time you save for research and ideation
Mashuk adds: "Our customers conveyed that we are efficient and deliver the tasks within the timeframe which helps them to be on schedule with their CRO activities. Clients also mentioned that by outsourcing A/B test development they were able to increase the test velocity by 10x or more."
4 What about the costs of outsourcing A/B test program?
Reza says EchoLogyx has two pricing models, hourly and monthly. The hourly rate of $25 covers test development, QA and design work with an engineer assigned from the pool. Clients on a monthly rate of $3000 (12 month minimum) get a dedicated engineer for their experimentation program. Over time, the monthly rate is more cost-effective, working out to around $18 an hour. He adds: "Some clients start on an hourly basis and then hire dedicated resources from us whereas some clients hire full time dedicated resources—it really depends on their need and how mature their testing program is."
5 Best practices when outsourcing A/B test development
If you're going to go the outsourcing route, there are a few best practices to keep in mind.
Choose an expert in a/b testing
Outsourced developers aren't all alike, so it's essential to make the right choice. A/B test development requires specialized training, so developers can problem-solve, troubleshoot, and reverse-engineer websites to push through variants. Not all outsourced developer firms have this expertise, so ensure that your choice really knows A/B testing.
Don't just outsource A/B testing to any outsource development company.
Identify a single point of contact
Reza says it's essential to have an identified point of contact to work with the outsourced team to manage resources and delivery. "Based on my experience working with agencies, they have a couple of engineers in-house to go to meetings and manage the client side, but the bulk of the day-to-day work is done by our engineers. This works really well."
Don't put too many cooks in the kitchen or rotate bodies.
Create a thorough brief
If you want A/B test outsourcing to work effectively, the brief is a key tool. It's the only way to be sure everyone's clear about test creation, expectations, and deliverables. Brillmark says the ideal brief includes:
- Experiment name
- Hypothesis
- Experiment details
- Variations
- Functional requirements
- Design assets
- Tool settings (if any)
- Url targeting
- Experiment goals
- Audience targeting
- Traffic allocation
EchoLogyx's brief covers many of the same areas, but also pays attention to quality assurance. "The most important part is the QA scenarios for a particular test. For example, on a product details page test one of the scenarios could be that on the sale page, there's a strike through price with the old price and the new price. Ensure the layout is not breaking for those pages."
But they don't always insist on having every checklist item filled in. "For example, if a test is merely a CTA change, we don't need a lot of information. They might just give a screenshot of the changes and say can we run a very quick test, whereby you're just changing the CTA text from 'add to basket' to 'buy now'. But if it is a complex test then, we definitely do need all this information."
A test plan template is also essential, says Reza: "This is THE blueprint for the test that should contain all the necessary information about the test, targeting, use cases, QA scenarios based on which we can then assess and understand what needs to be developed and QA’d. That way, we can provide estimation and ETA of test delivery for our clients."
Don't leave key information out of the brief for complex projects.
Communicate and track progress with project management tools
Communication is key to making the outsourcing process work well. That means using the right tools—such as Jira, Trello or Teamwork—to track everything that happens with a test and document what works well. That also makes it easy to be clear about deliverables and responsibilities. Mashuk says it's better to overcommunicate rather than risk missing information.
One area where it's best to be aligned is on production code pushes. Reza comments: "Our clients typically manage the communication between their internal developers and what are being tested. In case if we notice that something has been changed in the middle of the test development, we notify that to our clients. Otherwise, the clients usually let the internal team know what tests are being developed and under which area."
Don’t send the brief and wait for delivery, especially upon getting started.
Share your vision
If you're planning to make changes to your site, one way to make sure you and the outsourced team are on the same page is to share your thinking via mockups and workflows. Mashuk says that this process differs according to the client. "A few of our clients supply a Balsamiq file, where clients provide a high-level overview, and we use our knowledge and experience in order to align it with the client's website theme/or brand guidelines. It is always good to have a PSD/Invision/Figma/Sketch file available that reduces the back and forth with the client and things can be done with pixel perfect precision most of the time. We also find it very useful if our clients provide a video explaining the tasks."
Reza added that having access to a design file is useful for engineers when looking at the changes. It lets them identify colors, fonts and styles without any guesswork.
Don't take it for granted everyone on the team is on the same page.
Pay attention to data security
One potential concern is how to handle access to your website and tools, from a security perspective. Brillmark usually gets collaborator access to mockup tools to allow them to export and work with designs. In terms of working with tools themselves, they add: "In most cases, clients give us access to their tools either to a shared email they created for us or to individual developers that work on their projects. We develop the code in our local environment and once the development is completed we transfer the code to the testing platform using the access shared by the client."
When working with agencies, Reza has this to say: "We would typically become their partners or part of their team. Usually agencies say to their clients 'we have got a partner agency who will be developing and cueing the tests for you'. So there is that transparency. We have got the direct relationship with the agency and it's totally down to them, how they want to give us access to the testing tools for their clients."
Don't forget to get access to the testing platform.
Pay attention to how testing tools support SPAs.
Outsourcing A/B testing for single page applications (SPAs) requires expert handling, comments Brillmark's Mashuk: "On dynamic web pages or SPAs, the content of a web page usually gets updated based on various events after the initial page has loaded. If there is an a/b test running on the page, the updated dynamic content can sometimes overwrite the changes made by the test. This hampers the consistency of the experiment. In order to address this problem, several a/b testing platforms use different approaches. Only developers experienced with developing tests with Single Page Application for different tools know what approaches to take to build the tests. Luckily, we are experts in building these kinds of tests and know the right approach to take while using different tools."
Reza adds that there are some additional considerations when testing SPAs. "Clients need to understand whether the testing tools support the SPAs and work on the client side. The second thing is what kind of library is being used with the SPA, and how they have used it. You can run a lot of the tests client side, but some of the tests require developing server side."
He adds: "With SPAs the testing platform is quite important, and also whether you need to think about a hybrid model where you are doing some tests server side, and some client side, or both together."
Don't accidentally overwrite changes.
Follow a structured process
The best way to keep on track with A/B test outsourcing is to have a process Reza comments: "With our A/B testing department, we follow quite a structured process. In some cases sometimes they've got a different process which we have to then embed within our process, but in most cases, we follow the process that we have got: test plan, test brief estimate, then develop pre QA, QA."
Don't leave the structure of the testing process to chance.
Ensure you’re achieving savings
One of the big benefits of outsourcing is that you're not having to provide a salary for an in-house developer. Instead, you can use experts when you need them, and only pay for that time. If your A/B test outsourcing solution isn't generating savings, then it's time to look for a new one.
Don't outsource A/B testing unless it's saving you time and money.
So, should you outsource A/B testing? If you follow the guidance and best practices from our experts, the answer is yes. And you might find new ways to use the developers with whom you're building relationships, says Mashuk:
"One of our ecommerce customers was using different agencies for A/B test development, Shopify development, marketing, etc. We started to work with them as their A/B test development partner but later on, after seeing our quality work & support they kept expanding our cooperation with different services such as Shopify store development work, integrations with Shopify Apps, etc. Now they use us for all of their development needs."
The best A/B test development agencies in 2021
All proudly run by optimizers and digital experimentation experts.
- Brillmark – based in the US.
- EchoLogyx – based in the UK.
- Optiphoenix – based in India.
Don’t let development scarcity slow down your testing velocity. If you have any concerns, questions, or challenges in developing tests, please reach out to us. We’re happy to share with you how our customers and others are using our partners to affordably build professional A/B tests. [email protected]
