Data Insights: What are New Trends and why do they matter?
New trends develop when the fundamentals of a given metric value change. Unlike anomalies, these changes persist over a period of time. The most interesting form of new trend is called a break, and it happens when the average of the values changes abruptly at a specific point in time.
For example, take the following data:
Clearly something changed! But how do we detect it? We can look at the fundamentals of the data before and after the drop, either by using a linear regression or a moving average. A break is apparent when there is a significant change in that fundamental from one point to the next.
As you can see, the regression lines are both flat but changed their magnitude by almost 1,200. In many cases breaks will not be as obvious as this example and you will need to choose a change threshold that is as sensitive as you need.
Detecting breaks is somewhat more difficult than anomalies because you will need to test if every point is a break point. One way is to create a sliding window that moves across each data point and computes the regressions before and after that point. If those regressions differ by more than a threshold you pick, you will have found a break in the data.
Tomorrow we’ll go even deeper by looking at insights that span more than one metric.
Quote of the Day: “The surest way of concealing from others the boundaries of one’s own knowledge is not to overstep them.” ― Giacomo Leopardi