Now that you have your ad performance benchmarks in hand, you can start to identify when performance changes in unexpected ways. This is known as anomaly detection and can help you quickly sift through a pile of data for just the few things that you need to analyze.
The most obvious way to do this, as we discussed yesterday, is to take the global benchmark and flag any campaign that is below that as underperforming. However, our benchmarks relied on aggregating a number of days together which risks losing valuable information. What if a campaign performs very well on Monday and Tuesday but very poorly every other day of the week?
Instead of relying on our benchmarks for detection, we will use them as an ingredient in a more sophisticated approach. The following chart shows the value metric for seven days of the campaign, and the daily benchmark (0.58, shown in green) that we used to summarize it.
As you can see, the benchmark helps to understand which days are above and below normal but it is still not entirely clear which days are truly anomalous.
We can expand the benchmark by adding in an interval above and below which are expected deviation ranges. We expect our metrics to change everyday, so we might expect, based on previous knowledge in working with these campaigns, that 15% above or below the benchmark is still within expected performance. This, in turn, gives us an expected range which we can chart as follows:
As you can see, this new interval approach makes it obvious that Day 2 was well outside expected behavior and classify it as an anomaly that needs to be investigated. Once you have established an expected range of values, you can easily monitor all of your campaigns for anomalies by comparing the daily value metric to the expected range.
Note that simply choosing a range of values above and below the benchmark (in this example 15%) does not capture the typical changes in your campaigns that may happen over the course of a week. You may want to use more advanced techniques, such as ARIMA modeling, to create an expected range that reflects the natural weekly and monthly cycles in your data.
Tomorrow we will revisit the value metric and think about how best to capture the performance of your campaigns in a single metrics we can optimize.