Agricultural Demand Forecasting

Forecasting the Harvest

Predicting demand for agricultural products isn't like forecasting sales for socks or software. The entire system is governed by natural cycles. Crops have distinct growing seasons, leading to extreme seasonality. A product might be abundant for a few weeks and then disappear for the rest of the year. On top of this, you have unpredictable shocks: a sudden pest infestation can wipe out supply, while an unexpected bumper crop can flood the market. Traditional forecasting models often struggle with these sharp peaks and volatile patterns.

To build a robust forecasting system, we need a hybrid approach. We'll start with a model that excels at understanding time-based patterns and then enhance it with another model that can capture complex, non-linear relationships. This allows us to model both the predictable rhythm of the seasons and the chaotic nature of the market.

Decomposition with Prophet

Our first tool is , a time-series forecasting library developed by Facebook. Prophet's main strength is its ability to decompose a time series into several distinct components. It sees the data not as a single line, but as the sum of a few underlying signals:

Trend: The general, long-term direction of the data. Is demand for avocados slowly increasing year over year?
Seasonality: Predictable, repeating patterns. This includes weekly cycles (e.g., more sales on weekends) and yearly cycles (e.g., huge demand for pumpkins in October).
Holidays: The effect of specific, irregular events. These are like one-off seasonal spikes, such as a surge in demand for cranberries around Christmas.

For agricultural data, this decomposition is incredibly useful. Prophet can automatically identify the strong yearly seasonality of a specific crop's harvest cycle. We can also feed it a custom list of holidays and important events, like the dates of a major food festival, to help it model specific demand spikes. By breaking the problem down, Prophet gives us a solid baseline forecast.

However, Prophet has its limits. It's essentially an additive model, meaning it assumes the final output is just the sum of its components. It can't easily capture interactions between variables. For example, a heatwave might boost demand for salad greens, but only if it doesn't happen during a major holiday weekend when everyone is grilling meat. Prophet struggles with these kinds of conditional, non-linear relationships. That's where our second model comes in.

Modeling Complexity with XGBoost

(Extreme Gradient Boosting) is a powerful, tree-based machine learning algorithm. Unlike Prophet, which is designed specifically for time-series, XGBoost is a general-purpose model that can find complex patterns in any kind of tabular data. Its strength lies in identifying non-linear relationships and high-dimensional interactions between a large number of input variables, or 'features'.

To use XGBoost for forecasting, we first need to perform on our time-series data. This involves creating new input columns that give the model context about each data point. For an agricultural use case, we can create features like:

day_of_year, week_of_year, month
days_since_harvest_start
days_until_next_holiday
rolling_avg_price_7_days
is_promotion_active (0 or 1)

These features transform the time-series problem into a standard regression problem, where we try to predict the demand target based on the features for that day.

import pandas as pd

# Assume 'df' is a DataFrame with a 'date' column
# and 'harvest_start_date' is a known date
df['date'] = pd.to_datetime(df['date'])
harvest_start_date = pd.to_datetime('2023-08-15')

# Calculate 'days_since_harvest_start'
df['days_since_harvest_start'] = (df['date'] - harvest_start_date).dt.days

# Set negative values (before harvest) to 0
df.loc[df['days_since_harvest_start'] < 0, 'days_since_harvest_start'] = 0

print(df[['date', 'days_since_harvest_start']].head())

The Hybrid Approach

We can get the best of both worlds by combining Prophet and XGBoost. The strategy is to use Prophet to do what it does best: model the primary trend and seasonalities. Then, we use XGBoost to model what's left over—the complex patterns that Prophet couldn't capture.

The process works like this:

Train Prophet: Fit a Prophet model to the historical sales data.
Extract Components: Use the trained Prophet model to generate its forecast, including the individual trend and seasonality components. These components now become features for our next model.
Calculate Residuals: Find the error, or residual, of the Prophet forecast ( $error = actual\_demand - prophet\_forecast$ ). This residual represents the part of the demand that Prophet failed to explain.
Train XGBoost: Train an XGBoost model where the target is the residual from the previous step. The features for this model will include our engineered features (like days_since_harvest) and the Prophet components (trend, seasonality).

The final prediction is the sum of the Prophet forecast and the XGBoost prediction: $Final\_Forecast = Prophet\_Forecast + XGBoost\_Forecast(residuals)$

This hybrid architecture allows Prophet to anchor the forecast with a strong, interpretable baseline, while XGBoost adds a powerful, non-linear correction on top.

Validation and Evaluation

With time-series data, we can't use a standard random train-test split for validation. Doing so would let the model 'see' future data to predict the past, leading to an unrealistically optimistic performance estimate. Instead, we must use a method that respects the arrow of time, like walk-forward validation.

In walk-forward validation, you train the model on a historical chunk of data, then validate it on the next chunk of data immediately following it. Then, you expand the training window to include the previous validation data, and test on a new, future chunk. This process mimics how the model would actually be used in production: periodically retraining on new data to predict the immediate future.

To evaluate the performance across these folds, we use standard regression metrics. The two most common are Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). MAE tells you the average error in your predictions in the original units (e.g., "we were off by 50kg on average"). RMSE is similar, but it penalises larger errors more heavily, making it sensitive to occasional wild mispredictions.

MAE = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i| \\ RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}

By applying this hybrid model architecture and validating it correctly, you can create forecasts that are not only accurate but also robust enough to handle the unique challenges of the agricultural market. These predictions provide the data-driven foundation for making smarter inventory, pricing, and sales decisions.

AI for Agricultural Sales Engineering