Machine Learning Is Changing Demand Forecasting

Abstract

Large retailers and consumer packaged goods (CPG) companies are using machine learning combined with predictive analytics to help them enhance consumer engagement, and create more accurate demand forecasts as they expand into new sales channels like the omnichannel. Now with cloud computing using supercomputers'neural network, algorithms, along with ARIMAX, dynamic regression, and unobserved components models (UCM), are becoming the catalyst for "machine learning-based forecasting." Compared to traditional demand forecasting methods, machine learning-based forecasting helps companies understand and forecast consumer demand that, in many cases, would otherwise be impossible. Companies that have implemented machine learning have found it easy to use, and its ability to learn from existing data takes relatively less time to implement, deliver benefits, and produce high ROI (return on investment).

Headnote

EXECUTIVE SUMMARY | Large retailers and consumer packaged goods (CPG) companies are using machine learning combined with predictive analytics to help them enhance consumer engagement, and create more accurate demand forecasts as they expand into new sales channels like the omnichannel. Now with cloud computing using supercomputers'neural network, algorithms, along with ARIMAX, dynamic regression, and unobserved components models (UCM), are becoming the catalyst for "machine learning-based forecasting." Compared to traditional demand forecasting methods, machine learning-based forecasting helps companies understand and forecast consumer demand that, in many cases, would otherwise be impossible. Companies that have implemented machine learning have found it easy to use, and its ability to learn from existing data takes relatively less time to implement, deliver benefits, and produce high ROI (return on investment).

Machine learning is taking a significant role in many big data initiatives. Large retailers and consumer packaged goods (CPG) companies are using machine learning combined with predictive analytics to help them enhance consumer engagement, and create more accurate demand forecasts as they expand into new sales channels like the omnichannel. With machine learning, supercomputers learn from mining masses of big data without human intervention to provide unprecedented consumer demand insights.

Predictive analytics and advanced algorithms, such as neural networks, have emerged as the hottest (and sometimes controversial) topic among senior management teams. Neural network algorithms are self-correcting and powerful, but are difficult to replicate and explain using traditional predictive analytics methods. For years, neural network models have been discarded due to the lack of storage and processing capabilities required to implement them. Now with cloud computing using supercomputers' neural network algorithms, along with ARIMAX, dynamic regression, and unobserved components models (UCM), are becoming the catalyst for "machine learning-based forecasting."

According to an article in Consumer Goods Technology magazine, through pattern recognition there will be a shift from active engagement to automated engagement. As part of this shift, technology (machine learning) takes over tasks from information gathering to actual execution. Compared to traditional demand forecasting methods, machine learning-based forecasting helps companies understand and forecast consumer demand that, in many cases, would otherwise be impossible. Here are several reasons why.

Traditional demand forecasts are based on time series forecasting methods (Exponential Smoothing, ARIMA, and others) that can only use a handful of demand factors (e.g., trend, seasonality, and cycle). On the other hand, machine learning-based forecasting combines learning algorithms (ARIMAX, dynamic regression, neural networks, and others) with big data and cloud computing to analyze thousands-even millions-of products using unlimited amounts of causal factors simultaneously up and down a company's business hierarchy.

Traditional demand forecasting and planning systems are restricted to only the demand history, while machine learning-based forecasting can take advantage of limitless data, determining what's significant, and then prioritize available consumer insights (demand sensing) to influence future demand using "what if" analysis (demand shaping). Compared to traditional time series forecasting systems, machine learning-based forecasting solutions identify the underlying demand drivers that influence demand, uncovering insights not possible with traditional time series methods. Additionally, the self-learning algorithms get smarter as they consume new data and adapt the algorithms to consumer demand.

Traditional forecasting systems are characterized by a number of singledimension algorithms, each designed to analyze demand based on certain data-limited constraints.As a result, much manual manipulation goes into cleansing data and separating it into baseline and promoted volumes. This limits which algorithms can be used across the product portfolio.

Machine learning-based forecasting takes a more sophisticated approach. It uses pattern recognition with a single, general purpose array of algorithms that adapt to all the data. They fit many different types of demand patterns simultaneously across the product portfolio up/down the company's business hierarchy without data cleansing handling multiple data streams (e.g., price, sales promotions, advertising, in-store merchandising, and many others) in the same model- holistically-without cleansing the data into baseline and promoted volumes. For example, traditional forecasting systems have a specific purpose leading to multiple inconsistent forecasts across the product portfolio. With machine learning-based forecasting, the same algorithm is useful for multiple processes including pricing, sales promotions, in-store merchandising, advertising, temperature, store inventory, and others creating one vision of a realistic integrated forecast.

When creating demand forecasts, traditional demand forecasting and planning systems analyze the demand history for a particular product/SKU, category, channel, and demographic market area. Machine learning-based forecasts leverage history for all items, including sales promotions, to forecast demand for every item at every node in the business hierarchy simultaneously.

Sales promotions drive as much as 80% of annual volume at many companies. Companies run thousands of sales promotions creating thousands of forecasting scenarios for unique SKU's (stock-keeping units) promotions. To make matters worse, the demand lifts (spikes) can amount to as much as 30% to 90% of baseline demand. Marketing investment for advertisingand sales promotions can accumulate to more than 40% of sales for those consumer products. According to Michael Kantor, founder ofthe Promotion Optimization Institute, only about 1 in 50 of the promotion lifts can be forecasted accurately, as well as evaluating the return on investment (ROI). Without investment in more advanced predictive analytics supported by scalable technology, will companies be able to forecast sales promotion lifts effectively in such a highly promoted environment?

Machine learning-based forecasting can automatically calculate the promotional lifts above the trend and seasonality, as many sales promotions are utilized to enhance holidays (seasonal periods) with much more precision across the entire product portfolio. In addition, it can calculate the revenue and profit impact on those brands. As discussed, machine learning-based forecasting models require no data cleansing, which has been proven to be more accurate in predicting promotional lifts.

Top ranked CPG companies continue to launch thousands of new products annually, proliferating retailers' shelves. Trying to predict demand for such a vast assortment of new products is more than a demand planner can reasonably handle in a time efficient manner. Furthermore, new products by definition are the most difficult to forecast. This re- quires automation, advanced analytics, technology, and marketing intelligence to accurately predict the initial demand using structured data. Then, more advanced text mining (sentiment analysis) using unstructured data to determine effectively and efficiently how to manage the launch to assure success. Unfortunately, most new products end up in what is referred to as the "long-tail" of demand. These slow-moving "long-tail" products are those that consumers purchase infrequently in small quantities. As we all know, outliers are also hard to predict, making it difficult for inventory planning purposes. In most cases, although companies can predict average demand using moving averaging methods with some degree of accuracy, they are not able to predict demand lifts associated with sales promotions. This makes it impossible to effectively manage inventory safety stock (buffer inventory) when sudden demand spikes occur without accumulating additional inventory costs.

Machine learning-based forecasting can automatically correct for outliers, and now there are machine learning new product forecasting algorithms that can forecast thousands of new products across a product portfolio automatically using new product machine-learning based methods. (These new product machine learning methods will be discussed in more detail in my JBF 2017 Spring column.)

Demand forecasting and planning wasn't so complicated during the 1970s and 1980s, when demand was stable and supply chains were much simpler. During the early 1990s, when companies began to globalize, consolidated supply chain complexity intensified as companies became multinational comprising thousands of products that are sold across multiple channels. This has intensified further as consumers have been connected through digital devices creating new channels (omnichannel). Today, Amazon.com is a separate channel in itself. Traditional demand forecasting and planning technology was not designed to handle this new growth and complexity, let alone the digital world created by the Internet of Things (loT). What has resulted is progressively inaccurate demand forecasts that require labor intensive manual manipulation (fine tuning, as many say) of the demand forecasts, resulting in even less accuracy.

There is a common pattern that companies are experiencing-proliferation of new data and information. This is primarily consumer and market data that can help companies predict demand more accurately. As a result, demand planners spend over 80% of their time managing data and information, rather than on analytics. Furthermore, traditional demand management systems were not designed to handle the huge volumes of diverse and escalating data streams. Trying to include all this information into the demand forecast using Excel spreadsheets or legacy demand management systems is frustrating and, in many cases, futile, not to mention very costly.

Companies all share these inherent challenges due to complexity and scale, making it almost impossible for demand planners to create accurate reliable forecasts. Businesses are no longer simple with stable demand patterns driven by only trend and seasonality. This makes it difficult to use base demand projections solely created using demand history. This has created havoc for demand planners. As result, people who participate in the demand forecasting and planning process do not start contributing to the forecast until late in the process (or at the very end of the process). Instead of providing input to generate the demand forecast, they are collaborating to manually adjust the forecast. This almost always introduces personal bias, whether intentional or nonintentional. A typical reason is when senior management artificially adjusts the demand forecast to match the final plan to meet revenue targets.

As a result, demand planners are spending too much time managing information, cleansing data, and manually adjusting forecasts, and then often delivering less than stellar results. This leads to demand planner burnout, poor productivity, and low morale. Machine learning-based forecasting considers more demand factors (variables) and weights them according to their significance, resulting in more accurate forecasts. This frees up demand planners' time allowing them to refine forecasts using analytical driven insights and domain knowledge. Companies that have implemented machine learning have found it is easy to use, and its ability to learn from existing data takes relatively less time to implement, deliver benefits, and produce high ROI (return on investment).

Many feel the next generation of machine learning will also include cognitive computing where the supply chain becomes self-healing. This would improve upon machine learning by going beyond predictions to making decisions to automatically correct for anomalies in the supply chain.

Do you see machine learning-based forecasting supporting the nextgeneration demand management? Will it eventually lead to cognitive learning creating an autonomic self-healing supply chain? Or are you still relying on cognitive dissonance to justify and maintain judgmental harmony within your current demand forecasting and planning process?

-Send Comments to: [email protected]