How can I track and measure lagging metrics?

This interview is part of Kameleoon's Expert FAQs series, where we interview leaders in data-driven CX optimization and experimentation. Daniel Schmidt is the CEO and co-founder of DoubleLoop, the platform which helps product teams execute metrics-driven product strategies. 

How should I use lagging metrics to make product decisions? 

Lag measures can’t be influenced directly by product iterations, so you need to figure out the input metrics that predict the lag measures.

The North Star playbook provides a framework for organizing your metrics in terms of input metrics (lead measures), a North Star metric (a proxy for customer value), and business KPIs (lag measures).

If you can positively influence an input metric, you should also see the lag measure benefit. But our assumptions about the relationship between lead and lag measures are often wrong, so they need to be tested with correlation scores and experiments.

You can apply the North Star framework to your business and connect your data sources to see correlations between lead and lag measures with tools like DoubleLoop. People are often surprised that their input metrics are inversely correlated with their North Star.

Here’s an example of what a data-connected metric map looks like, with correlations between each metric.

How can I track and measure lagging metrics?

Many of the tools people conventionally think of as analytics tools are good for tracking lead measures, like user actions. 

The data for lag measures, such as revenue, churn, number of customers, etc., often exists in more business-oriented systems (payment systems, accounting tools) or the company’s own database. 

Many companies are consolidating this data in a data store like Snowflake, BigQuery, or Postgres and using business intelligence presentation layers to visualize metrics.

How can KPIs help marketing and product-led teams push in the same direction when their goals differ? 

Product and marketing clash when they have different intuitions about where to focus their efforts. This is particularly difficult when the product and marketing teams live in separate branches of the organization tree.

When goals cascade from the top of the organization, it’s difficult for teams to collaborate across functions.

Companies can mitigate the problem of conflicting marketing and product goals by allowing front-line contributors to collaborate on their OKRs, across different departments.

However, a better solution to the problem is to create “growth” teams that are not restricted by the traditional boundaries between marketing and product. Growth teams look holistically across the acquisition channels and activation funnel to crack the formula of converting users. 

How should I structure KPIs to improve collaboration between marketing and product teams? 

It’s worth categorizing metrics that are distinctly influenced by product and marketing. For instance, conversion rate can be influenced by improving the product or implementing effective marketing campaigns that attract more qualified visitors. 

Evaluating the conversion rate solely for qualified visitors rather than all visitors reduces the interference of irrelevant factors and highlights the growth constraints.

Without a clean separation, product, and marketing tend to play tug-of-war with each other. When traffic increases, the conversion rate decreases, and vice versa.

Customer retention can also be influenced by both product and marketing. Product improvements can increase customer retention, while an effective marketing campaign communicating the product's benefits can increase stickiness.

To distinguish the effects of product and marketing on customer retention rate, you could segment the data by customers who have interacted with the product only, and those who have been exposed to marketing campaigns.

This way, it becomes clear how much each factor contributes to customer retention, and the business can make more informed decisions about allocating resources to improve this metric.

How can lagging metrics be used to evaluate experiment results that only run for a few weeks?

Your website/app usage will dictate how much signal you can get in a few weeks. If you have enough traffic, you can use experiments to test your theories for how lead measures relate to lag measures.

For example, if you think ‘saving items in a list’ is a lead measure for retention, you could experiment to increase engagement with this feature. Then you can see if retention increases for the user segment with increased engagement while considering other variables.

Given the lag, it will take some time to see the impact.

A team might celebrate after several successful experiments, but the compound impact of those experiments may not lead to the business impact they hoped for.

The team could have reached a local maximum or have been running experiments that cannibalized each other. Or even worse, their experiments could create short-term revenue gains that create user experience problems that kill the business.

In my article, Balancing Short- and Long-term Product Bets, I wrote about how Google uses lag measures to judge experiments. The article explains how a key part of Google’s success was monitoring the long-term user experience impact of short-term revenue optimizations.

Daniel, we heard you are interested in the philosophy of logic. How has this interest influenced your business DoubleLoop? 

At DoubleLoop, we take a "first principles" approach to building our product. In philosophy, "first principles" refer to the fundamental concepts upon which a theory or system is built.

In the context of building a product, this means breaking down a user problem into its essential components and then designing solutions from the ground up rather than imitating how other products solve the same problem.

SpaceX used a first principles approach to disrupt the space industry. Elon Musk's principles for SpaceX were to reduce the cost of space travel, optimize rocket components, and create a vertically integrated company.

For example, the approach led SpaceX to develop reusable rockets, a novel solution to control the cost of space travel.

At DoubleLoop, we believe one of the first principles of effective product development is to create a connection between work and outcomes.

By building up from this foundation, our solution looks different from other product management tools in the market.