Archive for the ‘Metrics’ Category

Calculating MAPE – special cases: Obsolete Stock

Friday, February 5th, 2010

When calculating MAPE what is recommended when actuals are positive but forecast is 0 (for example when clearing obsolete stock)?  Can you make the forecast match the actual quantity so as not to penalize the forecaster for something that was not forecastable. Is this the correct way to measure these scenarios?

Our recommendation is to exclude the Obsolete Skus from measurement and in computing the aggregate MAPE as a performance measure for the planner or for the Sales Manager responsible.  Clearing obsolete stock is a supply management activity not a demand forecasting activity.

Demand Plans are externally focused and are a representation of what the market wants.  The obsolescence of stock may be a result of inferior demand forecasting in the past but has nothing to do with the demand for such stock in the current period.  In essence, there is no demand for that stock – either it is sold for scrap or just donated.

Making the Forecasted quantity match the actual demand for such obsolete stock will have a downside as well.  This will artificially inflate the forecast performance of the planner on a weighted basis and hence hide poor forecasting on active open stock items.

For example, you sell 25K units of active SKU for which you forecasted only 5K.  Then you have 100K of obsolete stock you sell.   Under your measure the MAPE will result in a 16% error or an 84% accuracy.  In reality the MAPE on your active SKU is 80% or 20% accuracy.  The 100% accuracy on the obsolescence is a manufactured number but wrongly aggregates to influence the divisional MAPE.

Tags: , , ,
Posted in Metrics | No Comments »

Using Orders to predict POS?!

Wednesday, December 16th, 2009

I saw this linked in comment in a discussion group:

“I hate to get technical”. That expression always bothered me, but I am wondering who IS getting technical….who is using Orders to help predict POS? Who is using a “day of the week” dummy variable?
It is my belief that the current simplistic solutions aren’t able to deal with issues like this. Let me explain why….because they can’t solve the technical modeling issues to solve the problem correctly. I defy someone to tell me TWO solutions that can bring in orders to help forecast POS demand and identify and adjust the model based on the lead/lag relationship between orders and POS demand. The same holds true for “day of the week” dummy variable added in automatically when you have daily data. Also “week of the year”….”day of the month(when appropriate)”….fridays before a WEEKEND holiday…do you see my point?….. I could go on all day here…..How about Interventions?….identifying and adjusting the model for a “level shift”….”local time trend”….the challenge has made…..one more thought….the conclusion from the M3 competition that simpler was better did have a lot to do with the “KISS methodology” conclusion which didn’t necessarily help besides the technical difficulty.

“”I hate to get technical”. That expression always bothered me, but I am wondering who IS getting technical….who is using Orders to help predict POS? Who is using a “day of the week” dummy variable?

It is my belief that the current simplistic solutions aren’t able to deal with issues like this. Let me explain why….because they can’t solve the technical modeling issues to solve the problem correctly. I defy someone to tell me TWO solutions that can bring in orders to help forecast POS demand and identify and adjust the model based on the lead/lag relationship between orders and POS demand. The same holds true for “day of the week” dummy variable added in automatically when you have daily data. Also “week of the year”….”day of the month(when appropriate)”….fridays before a WEEKEND holiday…do you see my point?….. I could go on all day here…..How about Interventions?….identifying and adjusting the model for a “level shift”….”local time trend”….the challenge has made…..one more thought….the conclusion from the M3 competition that simpler was better did have a lot to do with the “KISS methodology” conclusion which didn’t necessarily help besides the technical difficulty.”

I was initially puzzled by the order of the POS versus orders – what is forecasting what?  or even Why?  Is it useful to predict POS using the orders?  Even if there is a significance, would that be a spurious variable just proxying inefficiently for the inventory levels and perhaps promotional activity?

We have always tried to solve client problems that needed the POS data to try to predict the orders.  Now you mention using Orders to predict POS, which seems to be a novel idea.

Orders can predict POS?  If so, how?  If I am the retailer and I can place orders, how will I use my historical order pattern to predict what my consumer is going to buy?

Perhaps just use the old adage “stack ‘em high and see ‘em fly”?!

Just order from the manufacturer and build your shelves and stores with the inventory, so the mere visibility of this inventory creates POS demand?!  If so, this will make the push concept very usable.  Seems very consistent with the supply side argument of the economists on the monetarist side.

It is good to see a refreshing view amidst all the supply chain forecasters subscribing to the pull-based forecasting methodolgy.  And I am one of them too……. I believe that using the POS can better predict Orders, the opposite of what you are proposing.

And this is what we preach and educate to our constituents.  How to better use your POS demand and inventory patterns to better create an order forecast.

You may want to review our web workshop information at

http://www.demandplanning.net/cpg-demand-planning-web-workshop.htm.

Tags: , , , , , , , ,
Posted in Metrics, demand planning training | 1 Comment »

Value Chain Metrics and Score-carding

Sunday, December 6th, 2009

One of the Key steps to improving your Value chain is to measure and score-card the current state.  If you don’t measure where you are, how will you know what you are improving and by how much?  In the six-sigma process, this is a key step using the DMAIC approach – Define, Measure, Assess, Improve and Control.  So Measurement is the key first step in your improvement effort regardless of the project.

If you want to improve forecast accuracy, then you have to ask why that is important.  First of all measure and calculate forecast accuracy, identify the value drivers in the form of inventory and service.  Then connect the dots — connect inventory and service to forecast accuracy.  See if there is a correlation.  This should define your value opportunity.

If you take this approach, then you will understand the importance of developing a balanced metrics process for your entire value chain.  This metrics process should include all metrics necessary to improve the profitability of the business.  These metrics should interest the Chief Executive Officer of the business.

We at Demand Planning LLC, have worked with clients to design a balanced, holistic, and comprehensive  set of metrics for the entire Value Chain.   This should help understand the root-causes of inefficiency – obsolescence, inventory levels and unbalanced cost structures and drags on corporate profitability.  You also need to measure the impact of sub-optimal customer service levels on your topline.

The holistic value chain metrics may incorporate:

  • Profitability Metrics – Return on Investment and Return on Sales, Gross Margins and Net Margins
  • Capacity Utilization – Asset Turnover
  • Demand Metrics – Forecast Error, Forecast Bias etc.
  • Inventory Metrics – Days on Hand (DOH), SLOBs, Inventory Turns
  • Working Capital Metrics  - Receivables
  • Service Measures – First Time Fill Rate or FTFR, Perfect Order, In-Stock levels
  • Manufacturing or Execution Adherence
  • Logistics, Transportation and Distribution Performance

It is not only important to design and develop these metrics, but also to continuously measure and monitor corporate performance.  This should really be the starting point for any value opportunity.  You may also want to set a target as the goal for the improvement efforts.  Then find the people and partners that can commit to achieve those targets.

Tags: , , , , , , , , , , ,
Posted in Metrics | 1 Comment »

Tracking Signal

Thursday, August 6th, 2009

Tracking signal is a measure used to evalue if the actual demand does not reflect the assumptions in the forecast about the level and perhaps trend in the demand profile. In Statistical Process Control, people study when a process is going out of control and needs intervention.

Similarly Tracking signal tries to flag if there is a persistent tendency for actuals to be higher or lower systematically. If Forecast is consistently lower than the actual demand quantity, then there is persistent underforecasting and Tracking Signal will be positive.

Tracking Signal is calculated as the ratio of Cumulative Error divided by the mean absolute deviation. The cumulative error can be positive or negative, so the TS can be positive or negative as well.

TS should pass a threshold test to be significant. If Tracking Signal > 3.75 then there is persistent underforecasting. On the other hand, if this is less than -3.75 then, there is persistent over-forecasting.

So in essence, |TS| > 3.75 implies a forecast bias ==> TS < -3.75 or TS > 3.75 implies a bias.

So what is magical about 3.75. This is an approximation using the relationship between a normally distributed forecast error and the Mean Absolute deviation.

In General, Forecast Error (using RMSE) * 0.8 = MAD.

At 99% promised service level, you will be using a 3 Sigma level. As a measure of MAD, this translates into 3.75 MAD hence the 3.75 as the threshold for TS.

Posted in Metrics | 3 Comments »

Inventory Optimization vs. Days on hand

Monday, July 20th, 2009

Why is it important to learn new techniques than banking on century old DOH measure?

The Days of Supply is a thumb rule calculation used to manage inventory during the era of green accounting spreadsheets. It has many challenges including the demand fluctuations which can affect the metric. The forecast is uncertain and has an error component. So the Days of supply compounds it by basing the safety hedge on a forecast.
Some companies base it on the average historical demand, which can be even worse since the supply chain is presuming that the average demand is a better forecast than the demand plan.

Safety stock calculated in absolute numbers and driven by hard parameters is more optimal than the days of supply method. In my own consulting, I have seen companies cut down on inventories and increase service levels using algorithmic safety stock methods. At DemandPlanning.Net we preach the application of this principle combined with a good demand planning process to deliver an accurate forecast.

There are three major software providers in this area among many others. These include Optiant, Smart ops and Tools group. Ilog is another company that has a comparable tool.

These tools attempt to optimize your inventories and network. The premise is that you need the appropriate inventories to meet customer demand based on the forecast plus the forecast error. This is a short term tactical requirement. In the long-term you may want to optimize the network, by adding or deleting or replacing your inventory locations. This is an inventory placement decision but also a network optimization decision that needs to take into account other logistics costs.

Our web workshop in August may be a good starting point to understand the influence of demand volatility on inventories.
See http://demandplanning.net/inventory_management_metrics.htm .

Tags: , , , ,
Posted in Metrics | 2 Comments »

Is MAPE controversial?

Friday, July 10th, 2009

MAPE = Mean Absolute Percent Error.  This is the common error measure used by the Supply Chain profession.  Demand Planners and Demand Forecasters spread across the manufacturing sector use MAPE to measure the performance of their demand forecasts.  This is used cross-sectionally across several items or SKUs for the previous month using a forecast that was created a few months earlier (1, 2, 3 or even 6 month lags are possible).
However, there is raging debate over what to use in the denominator while calculating this sacro-sanct MAPE measure.
Traditional MAPE measure = (Absolute Error)/Actual demand.
Why dont we divide the error by forecast value instead of dividing by Actual sales?  This also introduces a potential forecasting bias.  To reduce MAPE, the planner can actually over-forecast.
Here is what we need to keep in mind:
1.  Whatever we use as denominator, we need to be consistent across the score-card, division, and company as a whole. So the denominator does NOT depends on the level.  It needs to be the same across everything.

2.  Zero demand may result in an ugly “DivbyZero” error in Excel when you use Actuals as the denominator. The practical solution is to set the error percent to a high number such as 9999%. Note the error is not Zero!! Mathematically the error is infinite (but infinity is not a practical supply chain concept).

3.  Setting the individual item error as a high number when demand is zero, should not affect the overall MAPE, since the overall MAPE is NOT a simple average of individual MAPEs.

4.  The denominator problem disappears if in general forecast errors are lower, that is when Forecast is closer to the Actuals; but note that most supply chain issues and demand side biases also disappear when forecast error is smaller!

5. The point is planners/managers/companies should spend less time worrying/arguing about how large the error is when the error exceeds a certain threshold say 50-60%. Although mathematically it matters between 90% and 900%, from a supply chain perspective a 90% error is disastrous! A 900% error is just plain silly and perhaps shows either there is a metrics process flaw or we are chasing after the wrong rat!

6. In summary, we can definitely perform sensitivity analysis on how the division performance will change based on a denominator change. In some instances we have even used the mid-point between actual and forecast. There is a potential to underforecast when the denominator is Actuals.  So you can use a mid-point to what is called as the Symmetric MAPE.

7. Calculating with reference to the Budget is a good practice to measure the attainment of sales goals with reference to the plan targets – so using Budget as the denominator makes sense for measuring sales force performance.  However, note that this is not absolute error in the sense of MAPE and we are answering a different question. So we should stay away from this confusion.

Tags: , , , ,
Posted in Metrics | No Comments »