Evaluation of short-term multi-target respiratory forecasts over winter 2024-25 in England using sub-ensemble contribution analyses

Researchers looked back at how well UK health forecasts predicted COVID-19 and Flu hospital admissions during the 2024-25 winter. They used special math to see which individual models helped or hurt the overall group prediction. They found that different models work better at different times of an outbreak.

The limits of simple consensus

In epidemic forecasting, the standard procedure is to build an ensemble. An ensemble is a composite prediction created by combining several different mathematical models. The goal is to reach a consensus. This consensus should be more robust and less biased than any single model acting alone. However, the authors note that adding more models does not guarantee better performance.

Current approaches struggle to quantify how a specific model affects collective accuracy. This is difficult when satisfying different stakeholders. For instance, a hospital manager needs to know absolute patient numbers to allocate beds. This is measured by the per capita weighted interval score (pcWIS). Conversely, a public health official needs to know if the epidemic is trending up or down. This is measured by the ranked probability score (RPS). As shown in, real-world forecasting is messy.

Models are often added or dropped mid-season due to technical bugs or data shifts. This makes it hard to tell if the resulting ensemble was truly optimal.

Dissecting ensemble contributions

To move beyond simple consensus, the researchers ran a retrospective simulation. They tested "sub-ensembles," which are groups formed from subsets of available models. Their methodology follows a structured pipeline:

1. Retrospective Simulation: The authors re-ran every available model from the start of the season. This created a library of potential model combinations.
2. Contribution Estimation via GAMs: The authors used Generalised Additive Models (GAMs) to understand the "value add" of a model. A GAM is a flexible regression tool that captures non-linear relationships. It was used to smooth sub-ensemble scores. This helped estimate how much a specific model changed the total error rate.
3. Pareto Optimization: Improving absolute count accuracy often hurts trend direction accuracy. To solve this, the authors applied Pareto analysis. A Pareto front identifies "optimal" solutions. At these points, you cannot improve one metric without making another worse.

This allowed researchers to treat model selection as a search for trade-offs. They balanced competing mathematical objectives.

Evidence of shifting utility

Results show that model utility depends on the pathogen and the epidemic phase. For Influenza, operational ensembles were generally effective. They performed better than individual models for both counts and trends .

Specifically, the Influenza operational ensemble was, on average, 58% better in pcWIS and 41% better in RPS than individual models.

Findings for COVID-19 were more striking. Retrospective sub-ensembles could outperform the operational COVID-19 ensemble by 43% in pcWIS. They also outperformed it by 280% in RPS. The authors' GAM-based analysis explains these gaps .

For example, they report the Gaussian process (GP) growth rate model was often detrimental. This was particularly true for COVID-19 around the December epidemic peak.

The Pareto analysis showed there is no single "perfect" ensemble.

For COVID-19, the best sub-ensembles included a random walk component. A random walk is a model assuming the next value is a slight variation of the current one. This worked because COVID-19 dynamics were relatively flat that season.

Constraints and technical caveats

Several limitations remain. First, the researchers used only pure statistical approaches. They did not include semi-mechanistic models. These models use biological parameters like infection rates to drive predictions.

Second, the analysis is strictly retrospective. The GAM-based method relies on finalized, revised data. In real-time settings, data is often noisy and subject to frequent revisions .

Finally, the authors acknowledge some issues with their GAMs. They noted deviations from normality and homoscedasticity (the assumption that error variance stays constant). These issues may limit the precision of their statistical inferences.

The verdict on ensemble design

Is "more is better" a valid strategy for ensembling? This study suggests the answer is no. Larger ensembles do not guarantee better performance. Instead, diversity and complementary information drive accuracy.

Forecasting teams should move toward dynamic, phase-aware compositions. They can use Pareto fronts to manage stakeholder trade-offs. They can also use GAMs to identify "dead weight" models. Code for this evaluation framework is reportedly available; see the paper for the canonical link at https://github.com/jcken95/sub-ensemble-evaluation.

Figures from the paper