Demand forecasting in Python made easy with futureEXPERT
January 8, 2025
Do you have loads of data waiting to look into the future? Then futureEXPERT is just the tool you need! Simply start your Python environment and you’re ready to go. Let’s dive into the world of forecasting together.
What to expect
In this blog post, we’ll show you a use-case that we frequently encounter: Demand forecasting as a key to efficient, data-driven planning. We will create monthly forecasts, which are often required for production planning, with a sufficiently long forecast horizon of at least one year, focusing on medium-term planning.
The raw data typically comes from ERP systems and may be available as individual orders or on a daily basis. Therefore, we start by aggregating this data into monthly time series. We also need to consider aspects such as non-negative integers and possible package sizes. Thats quite a few requirements. Let’s find out if futureEXPERT can meet them all.
We’ll now show you how to use the futureEXPERT’s Python library to load data, configure a suitable model, and create meaningful visualizations. With the accompanying prepared Jupyter notebook, you can recalculate the results step by step, carry out experiments and use this as a template for your own data and application. Sounds interesting? Then let’s get started!
Content:
- CHECK-IN: From raw data to time series, the starting point for data-based forecasts
- FORECAST: From past data to future demand forecasts
- Examining demand forecast results: What’s inside?
- Conclusion: Demand forecasting with Python is really easy with the right toolbox
CHECK-IN: From raw data to time series, the starting point for data-based forecasts
The easiest way to prepare data via CHECK-IN is through the frontend:
Figure 1: Uploading ERP data extracts to the futureNOW frontend.
The graphical interface helps us define the correct data format, avoiding the typical problems right from the start that can arise when importing new data. We’ve all been there: after a long computation time, we realize the date format or decimal separator was incorrect, meaning the entire process has to be started again. The plausibility checks identify such inconsistencies early on, resolving issues at this initial step.
Figure 2: Defining data columns
We start the CHECK-IN process by selecting the file from the “Data” menu. Since our dataset does not contain any unnecessary entries, we can proceed directly to the next step, where we specify the type of information contained in each column. To do this, we assign each column to one of three categories: date, value, or grouping. We need a date column to assign demands to specific time periods, while a value column (in this case, we only have “DEMAND”) contains the numerical demand data. Optional grouping variables allow us to consider hierarchical structures. For our demand forecasting use-case, the column containing material numbers is particularly relevant, as we want to forecast demand at the item level.
Figure 3: Reviewing raw data preparation
Before moving to the final step of data preparation, we review a summary of the imported data to ensure it meets our expectations. This includes checking the number of entries on each date and confirming that all materials are correctly captured. In our file, we find historical demand data from 17 materials, one of which we’ll exclude later. The plausibility checks allow us to define how to handle the identified anomalies, such as values in the wrong format or those outside predefined bounds. While our aggregated time series aim to exclude negative monthly demands, individual negative values may still appear in the raw data due to cancellations. Since we want these values to be included in the monthly aggregation, we refrain from setting boundaries at this stage and will address them in a later step.
Figure 4: Preparing the time series dataset
Our data is extracted from an ERP system that provides daily customer and regional demands. To forecast monthly demand at the material level, we set the time granularity and the data level accordingly. We also exclude the material C011414576, a discontinued product that is no longer relevant for planning. By selecting “Demand” as the target variable and using summation as the aggregation method, we calculate the total monthly demand per material – exactly what we need for our demand planning. For months with no demand at all, we set the value to 0. Now we’re ready to create the time series. Since we ultimately want to work with Python, we seamlessly load the prepared data into our Python code by copying the “Version ID”, a unique identifier for our data. We can then use this ID in futureEXPERT to generate demand forecasts based on the aggregated data.
from futureexpert import *
client = ExpertClient(user='my_username', password='my_password')
ts_check_in_results = client.check_in_time_series(
raw_data_source='../example-data/demand_planning_data.csv',
file_specification=FileSpecification(delimiter=';', decimal='.'),
data_definition=DataDefinition(
date_columns=checkin.DateColumn(name='DATE', format='%d.%m.%Y', name_new = 'Date'),
value_columns=[checkin.ValueColumn(name='DEMAND', name_new='Demand')],
group_columns=[checkin.GroupColumn(name="CUSTOMER", name_new='Customer'),
checkin.GroupColumn(name="MATERIAL", name_new="Material"),
checkin.GroupColumn(name="REGION", name_new="Region")]),
config_ts_creation=TsCreationConfig(
time_granularity='monthly',
start_date="2007-10-01",
end_date="2024-06-01",
value_columns_to_save=['Demand'],
grouping_level=["Material"],
missing_value_handler="setToZero",
filter=[checkin.FilterSettings(type='exclusion', variable='Material', items=['C011414576'])]))
Alternatively, we could also call CHECK-IN directly from the Python client. We can either set the configuration directly in the code or load the prepared config file we downloaded from the frontend.
FORECAST: From past data to future demand forecasts
Let’s dive into the forecast calculation! Before starting FORECAST, we’ll fine-tune some specific settings. The various config classes help us set suitable parameters for preprocessing, forecasting, and method selection. While we could just stick with the default settings, we’ll customize a few options to tailor the results to our demand forecasting needs.
fc_report_config = ReportConfig(
title='Monthly Demand Forecast on Material Level',
preprocessing=PreprocessingConfig(remove_leading_zeros=True,
detect_outliers=True,
replace_outliers=True,
detect_changepoints=True,
detect_quantization=True),
forecasting=ForecastingConfig(fc_horizon=18,
lower_bound=0,
use_ensemble=True,
confidence_level=0.75,
round_forecast_to_integer=True),
method_selection=MethodSelectionConfig(number_iterations=18,
refit=True,
step_weights={3:1, 4:1, 5:0.5, 6:0.5},
additional_accuracy_measures=['me', 'mae']))
forecast_identifier = client.start_forecast(version=ts_check_in_results.version_id, config=fc_report_config)
Preprocessing
For the preprocessing step, we specify that leading zeros are removed. In our case, these zeros correspond to missing values that could distort the FORECAST results. Such effects often appear in demand forecasting, for example, when a product is added in the ERP system before its market launch, leading in initial periods without demand.
Additionally, we enable outlier detection and replacement to ensure clean input for model training, and the identification of structural changes, like level shifts.
Quantized value recognition shows whether historical demand values follow certain patterns, i.e., occur almost exclusively as multiples of a certain value. This is incredibly useful because it often indicates logistical requirements, such as package sizes in which the product is available, or characteristic ordering behavior of individual customers. These findings feed directly into the forecast creation, providing an optimal foundation for realistic and actionable planning.
Forecasting
We want to predict demand for the next 18 months. Knowing that our values can’t drop below 0 and must be integers (since materials are ordered in whole quantities), we choose 0 as the lower bound and let forecasts round to integers. Additionally, we set the confidence level for the prediction intervals to 75%.
Method selection
When it comes to method selection, we have several configuration options. The FORECAST module selects the forecasting method that delivers the best results for each time series – whether it’s a classical statistical approach or a modern machine learning algorithm. However, not all forecasting steps are equally important for our use-case, so we want to consider this in the model evaluation.
For example, we can’t adjust production for the next two months due to lead time, making the accuracy of the demand forecasts for this period less relevant. In contrast, the third and fourth months are critical, while steps five and six are slightly less important. All other months are however excluded from the model selection process, even though we still generate forecasts for them.
To account for these priorities, we use the step_weights parameter to assign weights to specific forecasting steps. Unspecified steps automatically receive a weight of 0, excluding them from the model evaluation. By default, all steps would be equally weighted.
For accuracy measures used in model selection, we rely on the default settings, which adapt to the time series type (MSE and PIS). Since we are also interested in additional accuracy measures, we calculate ME and MAE by setting the additional_accuracy_measures parameter.
Examining demand forecast results: What’s inside?
After just a few minutes, all results are ready, with FORECAST doing all the heavy lifting in the background. During this time, the system first carries out extensive preprocessing of the time series: detecting seasonality, any trends, missing values, and outliers, and replacing them with appropriate values. Suitable methods are then pre-selected based on the time characteristics of the time series, followed by training, optimization, and testing of up to 30 methods against each other. In our case, the process doesn’t just generate future forecasts: It also creates 18 historical forecasts for backtesting. Then a thorough plausibility check with about ten individual checks filters out forecasting models with inappropriate results. Finally, the system ranks the methods based on accuracy measures, while fallback methods ensure reliable results even in tricky scenarios.
We are now interested in the top three models and backtesting results for each time series.
results = client.get_fc_results(
id=forecast_identifier, include_backtesting=True, include_k_best_models=3)
The output is a list of ForecastResult objects. Each item in the list contains various results for each of our 16 time series: In addition to the forecasts themselves, we find other valuable outputs such as historical forecasts for backtesting, an overview of preprocessing changes (e.g., outliers), and much more.
Overview of time series characteristics and model results
To get a first overview, we call the Python function export_result_overview_to_pandas, which summarises important time series characteristics and model results in a table:
from futureexpert.forecast import export_result_overview_to_pandas
export_result_overview_to_pandas(results)
Figure 5: Model results and time series characteristics
This table allows a quick look at essential details about our materials and reveals one thing above all: we are dealing with a diverse set of sporadic, lumpy, and smooth time series, some with seasonal patterns, with and without trends, and so forth. One time series has an outlier. For others we have identified quantized values from the historical data. A quick look confirms the latter observation, as the corresponding materials are only available in specific package sizes. Similarly, the forecasting methods identified as the best are as varied as the time series themselves:
- Machine learning methods (SVM and Glmnet) were ranked first three times.
- Classic statistical forecasting method (TBATS and Exponential Smoothing) topped the list five times.
- Specialized methods for sporadic time series (Croston, InterpolID, MedianPattern) were voted best for six time series.
- An ensemble method took first place in one instance.
Many of our time series exhibit seasonality, usually annual seasonality (length of 12), while only three show no seasonal patterns. Few time series have a global trend over the entire historical period, although for a few time series a “recent trend” was identified.
Demand predictions with smooth demand patterns in the past
Now comes the exciting part: What will the demand look like in the future?
For a clear impression, we can visualize the forecasts. Let’s focus on the last eight years (96 data points) and the best model in each case.
from futureexpert import plot
for ts in results:
plot.plot_forecast(ts,
plot_last_x_data_points_only=96,
ranks=[1],
plot_prediction_intervals=False,
plot_outliers=True,
plot_change_points=True)
Let’s examine the demand forecasts for selected materials and see how our settings affect the results.
Example 1: Time series with annual seasonality and global trend
Figure 6: Forecast with TBATS and global trend
Here, futureEXPERT recognizes a global upward trend in the data. This example represents a smooth time series with a seasonal demand pattern, which is reflected clearly in the demand forecast.
Example 2: Detecting recent trends
Figure 7: Recent Trend
Not all trends are global. For this material, our FORECAST module detected a recent downward trend after a slight increase in the overall level in the last two years, even though there is no global trend over the entire period. The forecast starts at the new level and the identified slight negative trend is reflected in the forecasts.
Example 3: Detecting level shifts
Figure 8: Level shift
Changepoint Detection has identified a level shift in January 2022 for this material. FORECAST accounts for this increase and generates a forecast at this level.
Example 4: Ensemble forecast
Figure 9: Ensemble forecast
The best demand forecast for this material combines multiple forecasting models into an ensemble. In this case, it consists of the results from 17 methods, thus a mixture of statistical forecasting methods like ARIMA and machine learning algorithms such as SVM and Random Forest. The forecasts of these 17 methods are weighted equally, as this generally leads to the most stable results. Poor or implausible results are automatically filtered out to stop them detrimentally effecting the ensemble. We find the exact list of included models in the method_specific_details field of the (ensemble) model.
There’s another special feature here: The quantization detection found a package size of 78 in the historical data. As a result, all forecast values are multiples of 78.
Demand predictions with sporadic demand patterns in the past
Example 5: Croston forecast considering detected package size
Figure 10: Croston forecast
For this sporadically ordered material, we recognize that it is also only sold in specific (but much smaller) package sizes, namely in packs of 6, which is clearly visible in the plot. At the start of 2025, we predict a slightly higher demand than in other months. This brings us to Croston’s forecasting method: it either produces a constant forecast or a sporadic demand pattern with an identical demand level. If this is not the case, as seen here, an adjustment in the form of quantization must have taken place before returning the forecasts.
Example 6: Short time series
Figure 11: Forecast of a short demand time series
For this newly introduced material, FORECAST automatically removed leading zeros from the time series before training the model, as configured. It ensures that the training process is not distorted; otherwise, for example, a supposed upward trend might erroneously be carried into the demand forecasts. However, this adjustment in turn (correctly) produces a very short time series.
But short time series present us with another challenge: Backtesting cannot be performed to assess forecast accuracy. Fortunately, futureEXPERT is designed to handle this by using suitable fallback forecasting methods for the corresponding time series type. These methods provide reliable forecasts that we can trust, even without backtesting.
Example 7: Demand variations and low-demand phases
Figure 12: InterpolID forecast
In addition to structural changes such as sudden level shifts (seen in example 2), the data history can also reveal varying demand distributions and noticeable phases of unusually low demands. For example, this time series shows significantly reduced demand between October 2020 and April 2022 compared to the rest of the history. Unfortunately, this period falls in the middle of the historical data, limiting its usefulness for forecasting. Insights like this are more valuable when such patterns occur near the end of the historical period.
Example 8: Outliers and phased-out products
Figure 13: Forecast with an outlier in the time series
This material in turn shows such a transition into a “phased-out product” after experiencing increased demand for about a year from mid-2021. The change point detection identified a structural break at the end of 2023, classifying it as “few observations”: Demand has been near zero since then. The chosen forecasting method, InterpolID, also reflects this phasing out of the material by predicting only minimal demand values for the coming months.
Additionally, the historical data included an outlier in January 2020. The automated detection replaced this value with a data-driven estimate, shown as a green dot in the plot. This replacement ensures the outlier didn’t distort model training or results.
Example 9: InterpolID forecast considering trend in demand level
Figure 14: InterpolID forecast with trend
The InterpolID method, developed by prognostica, extends Croston’s principles by incorporating seasonality and trends. In this example, a trend observed since 2021 is projected forward in the demand forecasts.
Conclusion: Demand forecasting with Python really is child’s play with the right toolbox
This blog post has demonstrated how easy it is to create demand forecasts with Python using futureEXPERT. With automated trend detection, seasonal pattern analysis, and robust plausibility checks, we can quickly and accurately forecast materials with minimal effort: Just define your data and set the forecast horizon, and let FORECAST take care of the rest.
Want more control? You can customize many options to tailor things to your needs, such as method selection and accuracy measures. The results speak for themselves: from factoring in package sizes to identifying structural changes and phased-out products, futureEXPERT empowers you to optimize your demand planning effortlessly.
Ready to optimize your demand planning? Then start using futureEXPERT today and unlock the full potential of your data with future! Get started right away with the prepared Jupyter Notebook to gain hands-on experience and test it in action.