Impact Analysis: Historical Testing

This is part 3 of our series on Impact Analysis, previous segments are available in our archives.

Our goal in Impact Analysis is to build a Relationship Model, which will make it easy to understand the expected results of any changes we make to our business. While some parts of your business are well-understood and you can likely draw the relationship model from memory, other parts will require exploration and investigation to build the model.

Historical Testing is an exploration technique which relies on historical data about your metrics to understand the relationships that exist. For example, let us assume we did not know the Relationship Model for the online advertising campaign we discussed yesterday. Instead we need to try and work it out ourselves. We might start by assembling all of the metrics related to a few different online ad campaigns, such as the following:


It is clear that all three metrics are related, both visually and by calculating their correlation coefficients*, a measure of how similar two sets of numbers are. Ad Impressions and Ad Clicks have a correlation coefficient of 0.771 (very high) while Ad Clicks and Sessions have a coefficient of 0.956 (almost identical sets) making them interchangeable. This allows us to assemble a rough outline of our Relationship Model:

Rough Relationship Diagram

While we know the metrics and their relationships, we don’t yet know the direction of the relationship. Do Ad Impressions drive Ad Clicks or do Ad Clicks drive Ad Impressions? There are a few ways we can determine the directionality of the relationship:

  • Time. Sometimes metrics that are highly related have a slight offset. If one metric moves one day and the other moves the day after, it’s clear that the first metric is the driver in the relationship (and not vice versa). At least until time travel is perfected.
  • Magnitude. If the metrics use the same units (dollars, users, etc) then you can look at the magnitudes of the metrics to understand the direction. If you consider that both Ad Impressions and Ad Clicks are units of user attention, it’s much more likely that Ad Impressions drive Ad Clicks. Otherwise, you’re somehow generating more attention.

Always beware the spurious correlations where there is no “cause and effect” relationship, sometimes metrics are related for reasons that are unrelated to your business! These are usually easy to spot when you think about your Relationship Model using what you already know about your business.

Tomorrow we’ll cover another way to build a Relationship Model, Relationship Testing!

* Technically I am using the Pearson product-moment correlation coefficient. If you have never calculated a correlation coefficient before, it’s easy to do using the CORREL function available in both Excel and Google Sheets.


Quote of the Day: “History will be kind to me for I intend to write it.” – Winston S. Churchill