Metric Translation: Translations Gone Wrong

This is part 5 of our series on Metric Translation, previous segments are available in our archives page. Have you found this series useful? If so, share the Data Driven Daily with someone else!

Translations Gone Wrong

One of the great dangers in translating your metrics is that the original meaning gets lost along the way. The more you manipulate the data, the more likely you can introduce new meanings or even confuse the original insight by adding bias.

For example, consider the following metric that is growing fairly steadily over time:

wrong_raw

There does not seem to be any clear signal in that data, so we might be tempted to translate it into another form. Here is what happens when I go overboard and make two translations, the first being the relative change and the second plotting it using a logarithmic scale:

wrong_log

I’ve managed to turn a steadily increasing chart into one that implies the metric is shrinking! If you use enough translations, you can make any metric tell any story you want.

Even translating the data with the best of intentions we can make it very misleading. There are many other ways to do this (see Numbers Lie), so you need to be careful and use your judgement when applying metrics translations. While there are no steadfast rules on translations, here are some guidelines:

  • Use the fewest translations possible. As you can see from the example above, the more translations you apply, the easier it is to mislead.
  • Simpler translations are better. This week we’ve gone from simple (log scales) to complex (residual) translations. The simpler the translation, the easier it will be for your audience to understand.
  • Sanity check. Look at the final product and ask yourself how it could be misinterpreted. Will someone seeing this for the first time see the same meaning that you know exists?

Translations are a powerful tool, but like any powerful tool they are dangerous if mishandled!

In Review: While the data should speak for itself, often you need to translate it to make it accessible to the rest of your team. There are many methods to do so, ranging from different scales to calculations and each carries the promise of making insights clear and the danger of hiding the real meaning.

Quote of the Day: Poetry is what is lost in translation” – Robert Frost