How to Measure your Demand Forecasting Accuracy

Forecasting is only valuable if it is accurate. But, for complicated forecasts, it can be difficult to determine accuracy. Knowing the error rate for your forecast is also essential for improving future forecasts and building accurate statistical models. 

Commonly Used Demand Forecasting Metrics

The most commonly used methods for calculating forecasting error are:

• Mean Absolute Percent Error (MAPE)
• Weighted Mean Absolute Percent Error (WMAPE)
• Absolute Percentage Error (APE)
• Mean Percentage Error (MPE)
• Mean Absolute Deviation (MAD)

MAPE

MAPE and MPE are the two most widely used methods to calculate forecasting errors. MAPE is easy to understand because it is based in percentages.

MAPE is calculated by taking the absolute value of the actual results minus the forecast and dividing that by the actual results. The resulting number is then multiplied by 100. Because it uses an absolute value, negative numbers will not skew the results like with other techniques. However, MAPE is not an effective choice when dealing with low volumes. The closer to zero the actual volume is, the more worthless the MAPE equation will become.

MAPE can be a problem if you errors tend to overachieve your forecasts. If overachieving is significant issue for your organization, you may need to use another method. MAPE is also not the best choice if you have several different products in one forecast.

WMAPE

Weighted MAPE is the solution for organizations that like the MAPE error calculation, but have scale issues. WMAPE accounts for each product’s contribution to the total error. It weighs the impact of each individual item. For example, if you make an error of 2% on a product that generates $30 million and an error of 10% in a product that generates $50,000 your MAPE will not tell the entire story. The 2% error is actually costlier than the 10% error, something you cannot tell by using MAPE. WMAPE will account for these differences.

APE

Where MAPE gives you an average error, APE gives you the absolute error. APE is rarely used because it is easy to misinterpret the results. If a weekly forecast was accurate, but the daily forecasts were off substantially, the APE will make the forecast seem more accurate than was really the case. APE is useful at looking at long-term trends, but is not helpful when dealing with short-term demand issues.

MPE

MAPE is essentially a hybrid of APE and MPE. It seeks to cure the weaknesses of these other two forecasting techniques. However, MPE is still used by many organizations to determine the error in demand forecasts. It is especially popular in the retail sector.

MPE is calculated by subtracting the forecast from the actual results and diving the resulting number by the actual result. That number is than multiplied by 100 to give you a percentage. Like with MAPE, percentages are easy for all the stakeholders to understand. Saying you were off your forecast by 2% is more meaningful than saying you were 3,000 units off, since not everyone knows what the typical volume of a given product are.

However, MPE is susceptible to misinterpretation if there are large swings in the forecast. Because, unlike MAPE, it is not an absolute value, periods where demand exceeded the forecast and periods where demand fell short of the forecast cancel each other out. This results in an error rate that may not truly reflect the situation on the ground. For many organizations overstating demand is just as costly as understating demand.

MAD

MAD was designed to measure the size of an error. It works best with a single product. If you try and aggregate MAD over several different products, they must all have a similar volume, or the results will be skewed by the heavier volume products.

MAD is especially useful when linked to revenue, as it will make profits and losses due to demand issues clear. MAD is calculated by taking the absolute value of the forecast errors and averaging them over the entire forecast period. This means it is not unduly swayed by swings between overestimating and underestimating demand.

Where to Start?

If your organization is just implementing a demand management process, the choices in how to measure the accuracy of your forecasts can seem overwhelming. For most organizations, the best place to start is with a combination of MAPE and bias. MAPE is simple to calculate, and will calculate errors regardless of whether the forecast overstates or understates demand.

Bias refers to a type of consistent forecasting error. You may have an over-forecasting bias or an under-forecasting bias. MAPE alone will not tell you the bias. You will have to look at the numbers to determine the bias. Armed with these two facts, you can improve your demand management process.

Deciding What Really Matters

Other issues you will face when implementing a demand management system is knowing what data to use and how much data to use for comparison. When selecting data, you need to choose the data points that will be most useful to your organization. The closer you can get to the real numbers the better.

This means that using the numbers actually shipped as opposed to the numbers ordered will give you a more accurate sense of your demand errors. Likewise, you may have a statistical forecast, a demand plan, and a consensus demand plan. How do you know what to measure? You could measure the accuracy of all three plans, but this may not be cost effective in a given organization. The best choice is to select the plan that has the most influence on purchasing decisions. If the supply chain orders are mostly based off of the statistical forecast, measure the accuracy of that with MAPE. If, instead the orders are based on the consensus demand plan, you will want to focus on making that plan more accurate.

When it comes to deciding on the period you should measure for comparison, it will depend on your industry and organizational needs. For electronics manufacturers, quarterly numbers tend to be the most useful. Many retail organizations use weekly numbers. Large durable goods may only be effective measured annually. When deciding how much data to look at, you need to decide what makes the most sense for your type of organization.

Typical Benchmarks

How do you know if you forecast errors are too high? One measure will be the cost of the errors to the organization. If the business is losing money because of the forecast, the error rate is too high. The acceptable error rate will vary considerable from industry to industry.

Here are some typical forecasting error rates for several industries:

Automotive                36%

Technology                12%

Consumer Goods      15%

Food                          18%

Healthcare                   9%

Industrial Products       7%

Pharmaceuticals        24%

Retail                            7%

Make sure to study the typical error rate in your industry or niche before making any decisions based upon your own forecasting error rates.