Context-Aware Hospitalization Forecasting Evaluations for Decision Support using LLMs

Abstract: Medical and public health experts must make real-time resource decisions, such as expanding hospital bed capacity, based on projected hospitalization trends during large-scale healthcare disruptions (e.g., operational failures or pandemics). Forecasting models can assist in this task by analyzing large volumes of resource-related data at the facility level, but they must be reliable for decision-making under real-world data conditions. Recent work shows that LLMs can incorporate richer forms of context into numerical forecasting. Whereas traditional models rely primarily on temporal context (i.e., past observations), LLMs can also leverage non-temporal public health context such as demographic, geographic, and population-level features. However, it remains unclear how these models should be used to produce stable or decision-relevant predictions in real-world healthcare settings. To evaluate how LLMs can be effectively used in this setting, we evaluate three approaches across 60 counties with low-,mid-, and high-hospitalization intensities in the United States: direct LLM-based forecasting, classical time-series models, and a context-augmented hybrid pipeline (HybridARX) that incorporates LLM-derived signals into structured models. Because the goal is operational decision-making rather than error minimization alone, we evaluate performance with bias and lead-lag alignment in addition to standard forecasting metrics. Our results show that HybridARX improves over classical ARX by yielding more stable and better-calibrated forecasts, particularly when incorporating noisy contextual signals into structured time-series models. These findings suggest that, in non-stationary healthcare resource forecasting, LLMs are most useful when embedded within structured hybrid models.

• The paper introduces a hybrid forecasting model that integrates LLM-generated contextual predictors with classical ARX techniques to yield improved calibration and stability.
• It applies a rolling-origin evaluation protocol on granular county-level data, incorporating hospital capacity and behavioral indicators to address non-stationary trends.
• Results demonstrate that while direct LLM forecasting lags real-time changes, the HybridARX approach mitigates error and bias, enhancing decision support in healthcare resource allocation.

Context-Aware Hospitalization Forecasting with LLMs: An Expert Evaluation

Motivation and Background

Effective hospital resource allocation demands robust forecasting of hospitalization at granular levels, particularly during healthcare disruptions such as pandemics. Classical time-series models often fail to anticipate non-stationary regime shifts, are sensitive to reporting lags and regional heterogeneity, and frequently underperform in context-rich decision settings. LLMs, which encode heterogeneous context and non-temporal features, offer potential as transformative components in operational pipelines. However, their utility as stable forecasters in real-world healthcare remains unestablished due to risks of hallucination, instability, and low interpretability.

The paper "Context-Aware Hospitalization Forecasting Evaluations for Decision Support using LLMs" (2604.23949) targets this gap by evaluating multiple forecasting paradigms—including direct LLM prompting, classical ARX-style statistical models, and a context-augmented hybrid pipeline (HybridARX)—at the county level using COVID-19 hospitalization and capacity data. Distinct from conventional error minimization, the study emphasizes metrics relevant for decision reliability: bias, lead-lag alignment, and temporal forecast stability.

Methodological Design

Multi-Modal Data Integration and Model Design

Data sources include granular county-level hospitalization counts, ICU capacity features, ventilator utilization, behavioral measures, and population statistics. Leading indicators (e.g., anosmia/ageusia search trends) are extracted and aggregated to standardize inputs for all models.

Three forecasting approaches are assessed:

• Direct LLM Prompting: The LLM receives structured historical series as text and returns numerical forecasts for the next period.
• Classical Baselines (AR, Lag-1, Exponential Smoothing, ARX): Built on univariate or multivariate regimes; ARX integrates contextual signals.
• HybridARX: An LLM forecasts next-period values of relevant contextual predictors (capacity and behavioral indicators), which are then used as exogenous regressors in classical ARX or regression models for hospitalization forecasts.

A rolling-origin, one-week-ahead protocol is used to minimize data leakage and maximize relevance to real-time decision-making.

Figure 1: Evaluation pipeline operationalizes standardized county-level inputs, leading indicators, and forecasting algorithms for comparative performance assessment.

Indicator Selection and Temporal Correlation

Pearson correlation analysis is performed to rank auxiliary indicators by predictive strength. Hospital capacity measures and search activity for sensory symptoms (anosmia/ageusia) display highest positive correlation with hospitalization trends.

Figure 2: Hospital capacity and anosmia/ageusia search volume exhibit strongest associations with near-term hospitalization counts.

Results

Forecast Accuracy, Stability, and Bias

Aggregate performance (MAPE) varies across county types; models perform best in high-intensity counties due to higher signal-to-noise ratios. Lag-1 and exponential smoothing baselines consistently exhibit robust accuracy.

• LLM (Prompt-Only): Delivers competitive MAPE and low systematic bias, capturing trend direction but predominantly lagging real-time changes by approximately one week. Run-to-run variance is modest.
• Classical ARX: Contextual variables, when directly incorporated, degrade forecast stability and induce bias, especially in regions with sparse data.
• HybridARX: Integrating LLM-generated contextual predictors into classical ARX mitigates large errors and bias seen in ARX, yielding more stable, calibrated forecasts. These effects are pronounced when contextual signals are inherently noisy or missing.

The lead-lag correlation metric $\rho^\star$ is maximized for exponential smoothing and prompt-only LLMs, indicating strong temporal trend tracking. Rare zero-lag alignments are observed in select high-intensity counties when structured exogenous signals anticipate shifts.

Implications for Decision Support

The evaluation demonstrates that simply appending contextual signals to forecasting pipelines is insufficient. HybridARX modality, with explicit contextual encoding and smoothing via LLMs, enhances reliability, calibration, and stability. Importantly, this paradigm preserves interpretability and reduces susceptibility to forecast volatility, critical for operational resource allocation where small errors have outsized effects.

Practical and Theoretical Impacts

Healthcare Operations

In practice, deployment of LLMs as standalone forecasters remains risky due to instability and limited alignment to real-world clinical needs. Hybrid architectures can leverage LLMs as flexible encoders of heterogeneous, region-specific signals, improving forecast stability and quality. This is particularly salient in resource-constrained facilities where errors amplify disparities.

Forecasting Science

The results reinforce that decision-aware evaluation must transcend pointwise accuracy and include timing, calibration, and context integration. The research further establishes LLMs' value as context extractors and regularizers rather than sole forecasters. The findings are extensible to other domains with irregular, non-stationary time series and heterogeneous exogenous information.

Future Directions

Identifying causal, actionable leading indicators remains open. Integrating probabilistic forecasting and uncertainty quantification with hybrid LLM pipelines is an avenue for advancing operational robustness. Extending hybrid frameworks to fragmented, facility-level data environments (characterized by privacy constraints and reporting heterogeneity) will increase generalizability and real-world impact.

Conclusion

HybridARX, a context-augmented forecasting pipeline utilizing LLMs to smooth and encode exogenous signals within structured statistical models, improves stability and calibration of hospitalization forecasts under realistic, noisy healthcare conditions. The work establishes the principle that LLMs are optimally leveraged as components in hybrid pipelines rather than direct forecasters. Future developments should prioritize identification of actionable indicators, integration of probabilistic outputs, and adaptation to fragmented operational environments, thus closing the gap between forecast science and clinical decision support.