Measure the impact of the features you ship with this framework

5 steps to close the test -> measure -> iterate loop

→ If you need help with anything product analytics related, email me at guido@bildungdata.com

It's common for product teams to ship features to production without first defining how they'll measure the impact.

Here's a framework I've found really useful for closing the test → measure → iterate loop

1) Step #1: Hypothesis

In the feature's scope document, start with a Hypothesis section

• Here you lay out the feature's goal and its expected impact.
• The expected impact needs to combine a KPI + a numeric value (the expected improvement)

→ Example:

• Feature: to increase user Conversion Rate, we're adding a Popular Products section to the Home.
• Hypothesis: a lot of users land on the Home and don't know where to find the best products, so they don't end up buying. Conversion Rate will go from 2.5% to 3%.

👉 Pro tip: if you have your metric tree mapped out, identify which branch this feature is acting on. That makes the input and the output you're generating clear. Examples:

• Popular Products → Conversion Rate branch (output)
• Cross-selling feature → Average Ticket branch (output)

2) Step #2: Metrics to track

The hypothesis's KPI is an output metric. It's made up of different input metrics that directly impact it. These are the metrics to measure:

• Conversion Rate (output)
• Conversion Rate New Users (input)
• Conversion Rate Recurring Users (input)
• CTR on Popular Products category (input)

3) Step #3: Events to track

For the output defined in the hypothesis, and its inputs, define which events you need to track to measure them once the feature ships

• Conversion Rate = unique buyers / DAUs
• Conversion Rate New Users = new unique buyers / new users
• Conversion Rate Recurring = recurring unique buyers / recurring users
• CTR on Popular Products category = category clicks / app opens

👉 Pro Tip: event tracking has to ship to production together with the feature.

4) Step #4: Ship to production + measure

• Once the feature is released to prod and you start getting data, check whether the hypothesis is rejected or validated → did Conversion Rate go from 2.5% to 3%?
• Use the input metrics to explain the growth / decline of the metric.

👉 Pro Tip: make sure you have enough data for the results to be statistically significant.

👉 Pro Tip 2: use a tool like Mixpanel to track events and measure feature impact

5) Step #5: Share results and iterate

• Share results with stakeholders: the tech team and whoever requested the feature.
• Use the input metrics' results to propose iterations on the feature to reach the goal you set