Numbers Lie: Hidden Bias

This is part 5 of our series on Metric Bias, previous segments are available in our archives.

Just because you can’t see it, doesn’t mean it’s not there

We have covered a number of different kinds of Metrics Bias this week including collection, interpretation and focus. However, what about all the other bias that is hiding in your data, bias that is specific to your business and the assumptions you have made? How do you find those hidden biases that might color your decisions?

Obviously, it is not healthy to be paranoid about your data and your decisions all the time. However, it’s also impossible to avoid every possible kind of bias in your data. The good news is that to make data driven decisions you do not need certainty, you just need a preponderance of evidence.

For example, let us decide whether a hypothetical company called Tony’s Crabshack should drop any items from their menu to save some money. Below is all their data on purchases from the past week:

Item Total Sales # Sold Total Profit
Crab Roll $500 200 $400
Lobster Roll $750 50 $500
French Fries $200 200 $100
Fried Clams $180 120 $50
Fried Shrimp $50 10 $20


If we assume our data is fully accurate and complete we would immediately drop the Fried Clams and Fried Shrimp! They make the least profit for us.

However, if we assume we might have bias in our data then we might look a little further at the Fried Clams just to be sure it’s worth dropping. We might find that our cashiers were mistakenly recording an order of clams for an order of fries! Or we might find out that 90% of the people who order clams also order a lobster roll, our highest profit item. Or we might find out that customers who buy clams come back every day and are our most loyal customers. Any of those factors might be hiding in the data since the signal is not clear.

Handling hidden bias is as simple as verifying your conclusions, looking for strong signals in the data and double checking your work. By assuming there is bias in your data you’ll be more likely to find it or at least compensate for it when making decisions!

Those shrimp, though, have to go.

Next week we’re going to tackle a topic requested by a reader! How do I determine my customer acquisition cost, the price I pay to gain new customers? It can be a harder question to answer than it seems.


Quote of the Day: “If you want to keep a secret, you must also hide it from yourself.” – George Orwell, 1984