Ad Campaign Optimization: Anomaly Detection

This is part 3 of our series on Ad Campaign Optimization, previous segments are available in our archives.

Now that you have your ad performance benchmarks in hand, you can start to identify when performance changes in unexpected ways. This is known as anomaly detection and can help you quickly sift through a pile of data for just the few things that you need to analyze.

The most obvious way to do this, as we discussed yesterday, is to take the global benchmark and flag any campaign that is below that as underperforming. However, our benchmarks relied on aggregating a number of days together which risks losing valuable information. What if a campaign performs very well on Monday and Tuesday but very poorly every other day of the week?

Instead of relying on our benchmarks for detection, we will use them as an ingredient in a more sophisticated approach. The following chart shows the value metric for seven days of the campaign, and the daily benchmark (0.58, shown in green) that we used to summarize it.

As you can see, the benchmark helps to understand which days are above and below normal but it is still not entirely clear which days are truly anomalous.

We can expand the benchmark by adding in an interval above and below which are expected deviation ranges. We expect our metrics to change everyday, so we might expect, based on previous knowledge in working with these campaigns, that 15% above or below the benchmark is still within expected performance. This, in turn, gives us an expected range which we can chart as follows:

As you can see, this new interval approach makes it obvious that Day 2 was well outside expected behavior and classify it as an anomaly that needs to be investigated. Once you have established an expected range of values, you can easily monitor all of your campaigns for anomalies by comparing the daily value metric to the expected range.

Note that simply choosing a range of values above and below the benchmark (in this example 15%) does not capture the typical changes in your campaigns that may happen over the course of a week. You may want to use more advanced techniques, such as ARIMA modeling, to create an expected range that reflects the natural weekly and monthly cycles in your data.

Tomorrow we will revisit the value metric and think about how best to capture the performance of your campaigns in a single metrics we can optimize.

Is Ad Campaign Optimization a problem you face in your business? Outlier is a product designed to help! Outlier monitors your business data and notifies you when unexpected changes occur, allowing you to know immediately when ad campaigns are underperforming or overperforming. If you’re interested in seeing a demo, schedule a time to talk to us.

Quote of the Day: “You know my method. It is founded upon the observation of trifles.” ― Sherlock Holmes (Arthur Conan Doyle), The Boscombe Valley Mystery