What is Metrics Bias?
Numbers can lie to you. If you don’t believe me, keep in mind that some of the most data driven organizations in the world make the biggest mistakes. For example: Canarsie Capital, a hedge fund, lost $60M in only three weeks! Flint, Michigan relied on faulty sampling and data when making disastrous decisions regarding the city’s water.
Numbers lie not when they are clearly wrong (that is obvious), but when they are subtly incorrect and it is hard to know they are wrong. You can use good judgement and make the right decision based on bad data, which in the end will be a bad decision. There are a variety of causes for these subtle data problems and I refer to them all as Metrics Bias, because the metrics are telling you a biased story instead of an objective truth.
Sometimes your metrics are correct, but you read a biased story from them based on your misinterpretation of the data. I also consider this Metrics Bias because the effect is the same! It doesn’t matter whether the speedometer on your car is broken or if you confuse miles with kilometers – in either case you can get a speeding ticket without realizing what you were doing.
It’s hard to be data driven if you can’t trust your data, so this week we are going to focus on how to identify and eliminate bias in your metrics.
Tomorrow we’ll get started by talking about Collection Bias, which happens when you think you have all the data you need and some of it is missing!