Skip to content

Impact Analysis: Relationship Testing

This is part 4 of a 5 part series on Impact Analysis.

Yesterday we covered one method of building a Relationship Model, Historical Testing, but it relied on having enough historical data to explore. For many parts of your business you won’t have enough historical data to truly understand relationships and will need to use other methods. Today we’ll cover one of those methods: Relationship Testing.

Relationship Testing is really simple, and as its name implies you run experiments to isolate and identify relationships in your business. Revisiting our example from yesterday, if we wanted to understand the relationship between metrics for online advertising but had no historical data we could experimentally test the following:

Day 1 Increase Ad Impressions 20%
Day 2 Decrease Ad Impressions 20%
Day 3 Stop all Impressions
… and so on.

As you can see, each day we make a different (and significant) change to one of the metrics involved and watch for any effects it has on the other metrics. In this case, we would see that when Ad Impressions go up, Ad Clicks go up and the click-through rate (CTR) stays the same. Over the course of a few weeks we could identify all the specific relationships necessary to build our Relationship Model.

This is, unsurprisingly, exactly the same approach you would take to A/B testing. A/B testing is a way to explore a single relationship, between a given change and an expected outcome. There are a few differences:

In A/B Testing… In Relationship Testing…
Your Goal is to: Drive better performance for a specific metric. Understand the relationship between two or more metrics.
You will: Run many tests of small changes to a single variable looking for optimal performance. Run a wide variety of tests to understand what responses you get from big changes.

Relationship Testing requires spending a large amount of time doing experiments that will have your business not operating at peak efficiency. This can be a significant cost, so it helps to identify a clear scope and boundaries so that your testing does not adversely affect the business in the long-term. It would be horrible to fundamentally hurt your business by trying to understand it!

Designing good Relationship Tests requires practice, so start small with areas of your business that you understand well and work your way up to more complex areas. Pretty quickly, you’ll find that it is also a great way to double-check the models you build using Historical Testing.

Tomorrow, we’ll apply our Relationship Model to do some Impact Analysis (finally)!

Quote of the Day: “Never give up your right to be wrong, and be sure to give others that right too.” – Tim Fargo

The Impact Analysis series