Why Should Both Product and Marketing Teams Experiment?
This interview is part of Kameleoon's Expert FAQs series, where we interview leaders in data-driven CX optimization and experimentation. Lupin Campos is a longtime proponent of experimentation. She began as a Lean Startup consultant 15 years ago, assisting startups in finding product-market fit through experimentation. She recently led product growth teams and implemented company-wide experimentation programs at high-growth fintech startups Stash and Greenlight. She's currently on the lookout for her next adventure.
What is the key motivation driving product and marketing when it comes to experimentation?
Marketing and product teams share a common goal: to offer the best user experience possible. They just approach it differently.
Marketing teams guide users from their very first interaction with the company before they even see the product. They want users to understand what the product can do for them and how it can help. They run experiments to see how different messages and content affect user understanding, ultimately impacting their behavior.
On the other hand, product teams focus on creating a product that can solve user problems and keep people coming back. Product teams’ experiments help them understand how elements of a product impact user behavior.
In short, both teams care about the user experience, but they approach it in different ways. They also need each other to succeed.
How do test types differ between product and marketing teams, and why is there a difference?
Marketing and product teams differ in their test types due to the nature of their roles and goals.
Marketing is inherently experimental, focusing on various avenues for promoting the product. Growth marketing teams explore advertising platforms, affiliates, promotions, and partnerships. Meanwhile, CRM and content marketing teams experiment with different content and imagery in their communications. Product marketing takes a similar approach regarding landing pages and the product website. They aim to welcome users and explain the product effectively, primarily focusing on copy, imagery, and layout. A/B tests run by marketing tend to be faster and more frequent, mainly because these changes are easy to implement and receive higher traffic at the top of the funnel (i.e., landing pages).
In contrast, product teams prioritize experience enhancements. They do engage in content optimization, often collaborating with product marketing. However, product tests are usually more complex, often requiring backend development, feature flags, and extensive Quality Assurance (QA). These tests also take longer to run, as they target deeper stages of the user journey with fewer users, making it challenging to achieve statistical significance. This necessitates thorough planning, research, and low-effort tests to evaluate the viability of ideas before execution.
Finally, it's important to add that product experimentation, particularly at the top of the funnel, leverages the traffic generated by marketing campaigns. This allows for the creation of unique, tailored experiences that cater to different user segments.
Recent Kameleoon research found that high-growth organizations invest in web and feature experimentation (MLG and PLG). Why should both product and marketing teams experiment?
Experimentation is the key to gaining valuable insights for both product and marketing teams, and there are three main reasons why it's crucial:
- Understanding user reactions. When introducing new features or landing pages, no amount of research or pre-work can predict precisely how users will respond. Only by observing their actions compared to their previous behavior can you truly grasp the impact of your decisions and allow for necessary adjustments.
- Adapting to changing user behavior. In the acquisition and early lifecycle stages, especially for early-stage startups, your target audience evolves as your strategy changes. Different user segments react differently to various stimuli. As high-growth companies know, shifting from paid advertising-heavy acquisition to affiliate or organic methods brings about different user profiles. Consequently, you need to adapt your communication and user experience accordingly.
- Measuring impact. Sometimes, changes are unavoidable, whether driven by a broader strategy or regulatory requirement. These alterations can significantly affect the user experience. It's vital to comprehend how and why these changes impact users, enabling you to learn from the outcome.
What’s needed in companies performing backend experimentation and web testing?
In companies where both backend experimentation and web testing occur, several essential tools and features are needed:
User-friendly web experimentation tool: A user-friendly tool facilitates rapid web experimentation, especially for non-technical teams. This empowers teams to make impactful changes to the UI without depending on engineers. A server-side A/B testing tool is needed to run more complex tests that change product features or functionality (requiring changes to the code).
Clear roles and communication: with simultaneous testing in the front end and back end, clear boundaries, roles, open communication, and well-documented processes are vital. This prevents conflicts and ensures smooth collaboration between teams. Additionally, the ability to run mutually exclusive experiments is essential to avoid biased results and unnecessary delays.
Consistent end-to-end experiences: If a user encounters a certain feature on the landing page but leaves before creating an account, that feature should still be available when they return. Ensuring consistent experiences, including separating cohorts over time, is crucial for the success of most experiments.
Comprehensive measurement system: A robust measurement system should allow for the definition and tracking of cohorts over time, not only against primary metrics but also secondary and unforeseen metrics. This ensures that the team can assess experiments' long-term impact and potential side effects on various aspects they might not have initially considered.
In summary, companies engaged in both backend and web experimentation require user-friendly server-side testing tools, clear processes, consistent experiences, and comprehensive measurement systems to effectively conduct and analyze experiments across their products and marketing efforts.
How can organizations support both product and marketing experimentation?
For the success of the experimentation program, create a framework for orchestration, communication, and documentation of experiments.
This base work should be encouraged by the organization's leadership. You will also need to open the right communication channels for cross-functional teams to share ideas, brainstorm, and have discussions that will enhance the user experience.
It's crucial to provide teams with the right tooling that meets their needs. A unified tool should allow everyone to easily access and understand ongoing and past experiments, including details, results, ownership, and other pertinent information. This tool should be integrated with reliable data sources. Ensuring data intake is clean and refining attribution models before conducting experiments. Trustworthy data is a prerequisite for obtaining meaningful results.
And finally, but more importantly, is getting alignment at a strategic level. Defining shared KPIs, North Star metrics, or company-wide goals that all teams work toward is essential. When teams share common motivations and objectives, it reduces friction and enhances collaboration, ultimately leading to better results.
Lupin, you do a lot of youth career mentoring. What tips can you share with managers looking to develop new talent in their teams?
Give your team clear expectations, objectives, and goals. Work with them weekly to support their progress.
I often work with emerging leaders who express frustration when their managers only provide feedback during performance reviews. This approach is ineffective, especially if expectations were never clearly communicated and feedback was lacking throughout. As leaders, we must maintain a broader perspective encompassing our team members' career development.
While it's easy to get caught up in the day-to-day demands of managing a team, allocate dedicated time to work with your reports beyond project updates and problem-solving. This time should be used to offer feedback, understand their feelings about their role, discover their personal goals and motivations, and collaboratively create a plan to help them achieve those goals.
