Once-for-All: Scalable Simultaneous Forecasting via Equilibrium State Estimation

Abstract: We introduce Equilibrium State Estimation (ESE), a novel paradigm for simultaneous prediction, where multiple interacting systems require separate yet coordinated forecasts. Such scenarios often arise in real-world settings such as economics and healthcare modeling. Unlike existing approaches that predict one system at a time, ESE forecasts all systems in a single pass. It first estimates the equilibrium state across systems, then generates holistic forecasts based on the difference between the current state and the estimated equilibrium. Extensive experiments on synthetic and real-world datasets, including currency exchange and COVID-19 spread modeling, demonstrate that ESE is at least as accurate as state-of-the-art (SOTA) methods while being significantly faster. In addition, ESE integrates seamlessly with conventional predictors, combining their accuracy with its exceptional efficiency and delivering a 10-70x speedup. With linear-time complexity, ESE scales far better than SOTA methods as the number of systems increases. Moreover, it remains accurate under diverse perturbations, establishing ESE as a fast, generalizable, robust, and scalable multi-prediction method.

• The paper introduces an Equilibrium State Estimation (ESE) framework that enables one-pass joint forecasting across multiple interacting systems with enforced statistical equilibrium.
• The methodology decomposes forecasting into latent equilibrium state estimation and trend-based prediction, achieving up to 70x speedup and improved accuracy versus state-of-the-art models.
• The approach is robust to noise and missing attributes, and its linear scaling makes it ideal for large-scale applications in finance, epidemiology, and cyber-physical systems.

Simultaneous Multi-System Forecasting via Equilibrium State Estimation

Problem Statement and Motivation

The paper proposes Equilibrium State Estimation (ESE) as a paradigm for scalable simultaneous forecasting across multiple, interacting systems. Classical approaches such as univariate time-series models or multivariate regressors (e.g., ARIMA, VAR, LSTM, SCINet) are insufficient for high-dimensional, coupled forecasting scenarios encountered in econometrics and epidemics. Existing models either ignore cross-system influence or suffer from superlinear computational scaling. In contrast, ESE exploits the concept of equilibrium—specifically statistical equilibrium based on inter-system attribute compatibility—for one-pass joint prediction across all systems, with computational complexity that is linear in system count and attribute dimension.

Methodological Framework

ESE decomposes simultaneous forecasting into two explicit stages:

1. Equilibrium State Estimator: Estimates the latent equilibrium state (vector of targets) across all systems, given system attributes. The estimator normalizes attributes, weighs them using MLE (maximum likelihood estimation), and iteratively updates equilibrium estimates via a cointegration-based convergence criterion. The modeling guarantees the proportional relationships among systems (sum-to-one constraint, change-conservation) are enforced.
2. Predictor: Forecasts the future state for all systems based on their current deviation from equilibrium plus an aggregate trend forecast for the super-system. The procedure is modular—ESE can act as a standalone regressor or a plug-in for existing sequence models.

Explicitly, the prediction for all systems at horizon $h$ is:

$$\mathbf{S}_{t+h} = \theta_{t+h} \mathbf{MS}_t \cdot \mathbf{ES}_t + \varepsilon_{t+h}$$

where $\mathbf{MS}_t$ is the super-system total and $\mathbf{ES}_t$ provides the equilibrium proportions, while $\varepsilon$ models residual error. The equilibrium estimation halts when cointegration between past system state and equilibrium state achieves a $p<0.05$ in statistical testing.

Empirical Evaluation

Datasets and Benchmarks

Experiments span both synthetic and real-world datasets:

• Synthetic: Multiple systems with known attribute-target coupling.
• Currency Exchange: 16 G20 non-USD exchange rates, five years of daily data, with macro-financial attributes.
• COVID-19 Spread: Victoria (Australia) municipal-level new-case forecasting (20, 79, and 320 region granularities).

Comparisons involve 13 SOTA models—univariate and multivariate regressors (ARIMA, VAR, LSTM, DLinear, Informer, PatchTST, TimeLLM, etc.) and their ESE-augmented variants.

Results

Prediction Accuracy

• ESE (standalone) consistently matches or surpasses the best SOTA model for most settings, particularly as system count ($n$) or input step/window increases.
• Augmenting SOTA predictors with ESE ("With ESE") always maintains, often improves, point and normalized error metrics (RMSE, MAE, RMSE*, MAE*) compared to their vanilla form.
• In high-dimensional or longer-horizon settings (e.g., 320 COVID-19 regions, currency aggregates with $n=16$), pure ESE or ESE-augmented SOTA methods provide the lowest errors.

Computational Efficiency and Scalability

• ESE provides a 10x–70x computational speedup over deep or transformer-based SOTA models, especially as system count or input dimensionality grows.
• Computational cost for ESE scales linearly in system count and input length; SOTA comparators scale superlinearly or suffer from combinatorial explosion.

Robustness and Generality

• ESE's predictions degrade minimally under localized attribute or observation perturbations. Randomized shocks/noise in a subset of systems have sublinear impact on overall ensemble forecast error.
• ESE tolerates missing or noisy attributes with little change in performance, indicating effective attribute-weight adaptation.
• Completeness of modeled ensemble enhances but does not limit accuracy (G7 vs. G20 experiments).
• Recursive multi-step rollout and phase shift analysis reveal that ESE structure is preserved under realistic temporal misalignment and composition.

Contrasts With Naive Schemes

• Naive weighting (e.g., proportional allocation based on current sizes) collapses in long-horizon or high-noise settings; equilibrium-informed disaggregation outperforms any fixed or direct averaging strategy for allocating forecasts to subsystems.
• Direct multivariate approaches, even with attributes (e.g., SCINet, VAR), degrade rapidly on accuracy or scalability compared to ESE.

Theoretical and Practical Implications

Theoretical Contribution

ESE formalizes joint multi-system forecasting not as multivariate time series inference but as equilibrium-based allocation, introducing proportional relationships and statistical cointegration as separation mechanisms for disentangling system structure from aggregate dynamics. This aligns forecasting closer with economic/game-theoretic equilibrium analysis, but operationalizes it in attribute-induced, dynamically estimated statistical states—distinct from classic Nash or regulatory equilibria.

ESE’s separability between aggregate forecast and internal distribution allows modular integration: any size-agnostic predictor can supply the trend, while ESE allocates this trend across the system ensemble. This flexibility distinguishes ESE from both graph/hierarchical methods and neural equilibrium models, which rely on explicit architecture-contingent or hierarchical dependencies.

Practical Implications

For large-scale, high-frequency domains (macro-finance, public health surveillance, industrial process control), where cross-entity coupling is strong and computational resources are finite, ESE enables deployment of efficient, robust, and accurate forecasting pipelines. The capacity to integrate with legacy or deep neural predictors means practitioners can immediately realize significant gains, bypassing the need for explicit inter-system graphical structure estimation or costly, repeated single-target modeling.

ESE’s linear scaling, robustness to local perturbations, and insensitivity to missing features make it well-adapted for real-time, operational environments where delays, shocks, and incomplete sensing are the norm.

Limitations and Future Directions

• ESE presupposes attribute similarity and cross-system coupling as formalized in its system definitions and summability constraints. Arbitrary, disconnected system ensembles are not suitable.
• Short input windows undermine equilibrium identifiability, limiting ESE’s performance in ultra-small-data regimes.
• True equilibrium (as in dynamical or game-theoretic senses) is not necessary for ESE’s statistical procedure, but further investigation into equilibrium-based dynamics with adaptive attribute selection (possibly via GNNs or learned graph structure) could generalize applicability.
• Future extensions include automated attribute selection, adaptation to time-varying and evolving system networks, and uncertainty quantification for regulatory environments.

Conclusion

ESE provides a formal, scalable, and robust paradigm for simultaneous forecasting in sets of interacting systems, outperforming classic time series and modern deep models in accuracy, efficiency, and resilience. By leveraging statistical equilibrium estimation and attribute-conditioned allocation, ESE solves multi-system forecasting as a canonical one-shot linear-time operation, directly enabling large-scale deployment for economic, epidemiological, and complex cyber-physical domains.

Citation: "Once-for-All: Scalable Simultaneous Forecasting via Equilibrium State Estimation" (2606.13285)