FFB Yield Forecasting

Forecasting palm oil yield 18 months ahead for 121 estates across 6 regions, with an honest uncertainty band, and numbers that add up from estate to region to country.

Direct multi-horizon forecastingGradient boostingBottom-up reconciliationPython

To protect enterprise data, only high-level techniques from the real project are shared. The worked example uses a public dataset (London smart-meter energy use) with the same shape of problem, so every number there can be reproduced.

121

palm oil estates across 6 regions

18 months

forecast horizon

3 levels

estate → region → country

MAPE at country level

The problem

The company runs 121 palm oil estates across 6 regions. Budgets, labour, logistics and sales commitments are all planned well before the crop comes in, so planners need to know roughly how much fresh fruit bunch (FFB) each estate will produce, long before harvest.

121 PALM OIL ESTATES ACROSS MALAYSIA

Each dot represents one estate · illustrative placement

121 estates6 regions

Schematic footprint of 121 estates. Dots do not indicate actual estate locations or regional allocation.

Three things were being asked for at once:

See far ahead. A forecast for each of the next 18 months, not just next month.
Show the uncertainty. A single number hides risk. Planners need a likely range.
Make the numbers add up. If estate forecasts do not sum to the regional figure, nobody trusts any of them.

PALM HARVESTING / FIELD CYCLE00 / 06

Fronds / palmRipe FFBLoose fruits (brondolan)Truck / transport route

Round every 7–10 days · FFB 15–25 kg · Mill within 24 h

0. Full-colour overview

The complete field cycle: palm and fronds in purple, ripe bunches in red, loose fruits in pink, and transport in blue.

Why it is harder than it looks

Think of it like forecasting exam results for 121 schools, 18 months early, and the district total must equal the sum of the schools.

Requirement	What usually goes wrong	What this project does
Long horizon (18 months)	A one-step model is fed its own predictions again and again, so small errors snowball.	Train a separate model for each month ahead (direct forecasting).
Uncertainty	A single line looks precise but says nothing about risk.	Report a p10 to p90 band around the central forecast.
121 estates, 3 levels	Separate models per level disagree with each other.	Forecast at the bottom, then sum upwards (bottom-up reconciliation).

The approach

Step 1: Describe each month with three kinds of signal

Feature group	What it captures	Examples
Time-dependent (T)	Where the series is heading	Trend, recent past values (lags)
Seasonal (S)	Patterns that repeat on a calendar	Fourier terms (sine and cosine waves)
Exogenous (X)	Outside drivers of the crop	Rainfall

Step 2: One model per horizon (direct forecasting)

Instead of one model that predicts next month and then feeds on its own output, a separate model is trained for each step ahead. No prediction is ever used as an input, so errors cannot compound.

Observed valuePredicted valuePredictors (lags)

Direct forecasting: the same inputs, a different model for each step ahead. Adapted from skforecast.

To predict 1 month ahead (t+1), we use data up to time t:ŷ_t+1 = f₁(X_t, T_t, S_t)To predict 2 months ahead (t+2), we also use data up to time t:ŷ_t+2 = f₂(X_t, T_t, S_t)⋮In general, for h months ahead:ŷ_t+h = f_h(X_t, T_t, S_t)

y is FFB yield, h is months ahead (1 to 18), f is a gradient-boosted tree model trained for that horizon.

Step 3: Put a range around every forecast

Each forecast is published as a central line (p50) with a p10 to p90 band: "we expect the result to land inside this band about 8 times out of 10."

10 futures → 1 band

Many plausible futures are summarised into a shaded band, which is what planners see on the dashboard. Redrawn from the supplied Australia arrivals illustration.

LightGBM quantile regression learns a separate lower and upper forecast for each horizon, using the same data available up to time t. Set the quantile objective’s alpha to 0.10 for p10 and 0.90 for p90.

P10 — lower forecast (α = 0.10)

ŷ_{t+h, 0.10} = f_{h, 0.10}(X_t, T_t, S_t)

P90 — upper forecast (α = 0.90)

ŷ_{t+h, 0.90} = f_{h, 0.90}(X_t, T_t, S_t)

Each model minimises the sum of the quantile (pinball) loss over its training examples. With residual u = actual − predicted, the two losses are:

ρ_0.10(u) = max(0.10u, −0.90u)

ρ_0.90(u) = max(0.90u, −0.10u)

The p10 loss penalises overprediction more strongly; the p90 loss penalises underprediction more strongly. Together, the forecasts define a nominal 80% prediction interval — actual coverage should be checked on held-out data.

Pinball loss / How α works

Illustrative seeded data · one straight-line model

α 0.50Show OLS point

Same data, different line

OLS (squared)P50 (pinball)

Cost of getting it wrong

ρ_α(u) = max(αu, (α − 1)u)

Step 4: Add it up from the bottom

Forecasts are made only at estate level. Regional and country figures are the sum of the estates beneath them, so every level agrees by construction.

Yield(region) = Σ Yield(estates in that region)

Yield(country) = Σ Yield(regions)

Worked example on open data

Because the estate data is confidential, the same recipe is demonstrated on daily smart-meter readings from London homes, joined with London weather. It has the same three ingredients: a hierarchy, strong seasonality and a weather driver.

	Real project	Open-data proxy
Target	Estate FFB yield	Household energy use (kWh per day)
Hierarchy	Estate → Region → Country	Household → Borough → London
Outside driver	Rainfall	Temperature, cloud cover, solar radiation
Seasonality	Yearly palm oil crop cycle	Weekly cycle (Fourier terms, period 7)
Horizon	18 months ahead	7 days ahead
Models	One per horizon	One per borough per horizon: 12 × 7 = 84

Daily energy consumption in London

Average household energy use

Average daily household energy use across London, Nov 2011 to Feb 2014 (827 days, about 3.5 million rows). Redrawn from the supplied chart.

Which weather signals move energy demand?

London: 7-day forecast vs actual

London: forecast with P10–P90 prediction interval

ActualForecastP10–P90

Mean of household quantiles. Band boundaries are redrawn from the supplied chart; averaging household quantiles does not guarantee a calibrated 80% interval for London.

Forecast error at each level of the hierarchy

Noise cancels as forecasts roll up. Error drops about 5× from household to London.

Household-level error by forecast horizon

Error on day +7 is close to day +1. Direct forecasting keeps far horizons from drifting.

Features (17): last 7 days of usage, today's usage, weekly Fourier terms, calendar fields, mean temperature, cloud cover, solar radiation, household ID.
Model: XGBoost (300 trees, depth 6, learning rate 0.03), one per borough per horizon.
Test: the final 7 days of every household were hidden; forecasts made from 20 Feb 2014.
Reconciliation: household forecasts averaged up to borough, then London.

Level	Series	MAE (kWh/day)	MAPE
Household	4,984	1.94	n/a
Borough	12	0.39	3.7%
London	1	0.37	3.6%

Reality check

One test window. Scored on a single 7-day window; a rolling backtest would be sturdier.
A consistent lean. The London forecast was above actual on 6 of 7 days, a small bias worth correcting.
No baseline yet. Not yet compared against a simple "same day last week" forecast.
Boroughs are stand-ins. Households were split into 12 equal groups to create the middle level.
Not every home could be forecast. 582 households had no reading on the start date.

Key takeaways

For long horizons, do not let a model eat its own predictions.

Forecast at the bottom and sum upwards, so every level agrees automatically.

Always publish the range. A band turns a forecast from a promise into a planning tool.

Judge the forecast at the level where the decision is made.

Tech stack

PythonpandasNumPyXGBoostLightGBMstatsmodelsFourier featuresPlotlyBottom-up reconciliation