Falcon-X: A Time Series Foundation Model for Heterogeneous Multivariate Modeling

Abstract: Time series foundation models (TSFMs) are transforming the forecasting paradigm through large-scale cross-domain pretraining. However, most existing TSFMs remain univariate, and recent efforts to enable cross-variate modeling still operate directly within the raw variate space. This design introduces fundamental limitations in semantic alignment and relational expressivity. Specifically, raw-space group mixing lacks a dedicated mechanism to align heterogeneous physical quantities, while standard non-negative attention fails to capture the complex synergistic and antagonistic interactions ubiquitous in real-world systems. To address these challenges, we propose Falcon-X, decouples variates from the raw space and maps them into a unified latent prototype space. Falcon-X employs a Unified Prototype Diff-Attention mechanism that explicitly evaluates both positive and negative semantic affinities to explicitly align heterogeneous variates. Cross-variate interactions are then efficiently performed within this shared space via Latent Entity Attention, naturally facilitating zero-shot structural transfer. Finally, a Variate Reassembly Router robustly reconstructs variate-specific trajectories via a request-and-dispatch mechanism. Extensive evaluations on the GIFT-Eval and fev-bench benchmarks demonstrate that Falcon-X achieves state-of-the-art forecasting performance, offering a principled and scalable paradigm for complex multivariate environments. Falcon-X is publicly released to support future research.

• The paper introduces a unified latent prototype paradigm using UPDA to decouple semantic alignment and capture both synergistic and antagonistic dependencies.
• It details the innovative architecture with latent entity attention and a variate reassembly router, achieving state-of-the-art MASE and CRPS improvements.
• Empirical evaluations demonstrate robust scalability, forecast stability, and effective multivariate modeling in diverse real-world applications.

Falcon-X: A Unified Latent Prototype Paradigm for Heterogeneous Multivariate Time Series Forecasting

Motivation and Theoretical Innovations

Falcon-X addresses two central constraints in multivariate TSFMs: the lack of semantic alignment across heterogeneous variates and limited relational expressivity in attention mechanisms. Prior approaches, including group attention and flattening-based paradigms, operate in the raw variate space, confining their modeling capacity to aggregative dependencies while exhibiting severe semantic collapse when faced with fundamentally distinct temporal patterns (Figure 1).

Figure 1: Comparison of modeling paradigms—latent prototype projection induces highly discriminative attention maps for heterogeneous variates, avoiding over-smoothing.

Falcon-X establishes a principled decoupling by projecting variates into a high-capacity, fixed-dimensional latent prototype space via Unified Prototype Diff-Attention (UPDA). This architecture explicitly models both synergistic (positive affinity) and antagonistic (negative affinity) interactions using learnable prototype keys, leveraging differential attention as a core mechanism. The latent entity attention (LEA) module then operates entirely within this unified semantic space, facilitating robust cross-variate dependencies and zero-shot structural transfer. To reconstruct physical variate trajectories, Falcon-X introduces the Variate Reassembly Router (VRR), which executes gated soft-routing from prototypes back to the entity space, dynamically fusing global context with temporal features.

Architectural Design, Components, and Training Protocol

The Falcon-X pipeline consists of four major stages: normalization and tokenization, time attention, variate attention, and probabilistic quantile forecasting (Figure 2). Inputs are normalized via the arcsine transformation and augmented with timestamp and mask tokenization to address scale and missingness heterogeneity. The time attention module applies $n$ independent Transformer encoder layers per variate, fully decoupling temporal modeling from cross-variate mixing and enabling stable evolutionary pattern extraction.

Figure 2: Falcon-X architecture: variate decoupling, prototype alignment, global latent entity interaction, and dynamic reassembly.

UPDA projects all entity representations into a compact prototype space, where both positive and negative correlations are modeled. Post-projection, LEA applies $l$ global MHA layers across prototype-aligned entities, enabling holistic cross-dataset dependency capture. VRR then executes variate-specific soft-routing, ensuring robust reassembly and context-aware gated fusion.

Falcon-X incorporates extensive pretraining on a diverse corpus combining real-world and synthetic datasets (GIFT-Eval, Chronos, QuitoBench, TSMixup/KernalSynth, synthetic multivariate injectors), enabling structural adaptability across domains and variate dimensionalities. The training design leverages Megatron-LM infrastructure with flexible batching, variate-wise shuffling, context and horizon sampling, and orthogonality-constrained optimization. It achieves convergence with highly stable loss trajectories across one million iterations (Figure 3).

Figure 3: Training dynamics and scaling behavior—loss, MASE, and CRPS converge stably across $10^6$ steps.

Empirical Performance and Quantitative Analysis

Falcon-X demonstrates state-of-the-art accuracy on GIFT-Eval and fev-bench, achieving substantial improvements in both MASE and CRPS over prior TSFMs and scaling benchmarks. On GIFT-Eval, it delivers 0.666 MASE and 0.453 CRPS, outperforming STRIDE, Toto-2.0-FT, and Chronos-2 (performance increases up to $6.6\%$ CRPS reduction) (Figure 4). Falcon-X maintains remarkable forecast stability across all horizons, especially in medium- and long-term settings where alternative models suffer horizon-wise error accumulation.

Figure 4: Falcon-X maintains consistent accuracy across forecasting horizons in GIFT-Eval.

On fev-bench, Falcon-X rivals Chronos-2 with 0.652 MASE and 0.490 CRPS, relying solely on endogenous targets without covariate augmentation. The model demonstrates strict neural scaling: increasing parameters from 59M to 591M consistently improves metrics without saturation or instability (Figure 5).

Figure 5: Performance scaling with layer allocation, prototype dimension, and model size.

Case studies across representative datasets validate both positive and negative dependency modeling. Falcon-X accurately synchronizes forecasts in positively correlated channels (bitbrains/fast storage), while capturing antagonistic trajectories in negatively correlated signals (bizitobs/application). Multivariate inference robustly calibrates predictions compared to univariate settings, especially under strong inter-variable coupling (Figure 6).

Figure 6: ETT1/15T channel-wise forecasts demonstrate deviation in univariate mode and calibrated trajectories via multivariate prototype routing.

Additional visualization further confirms quantile calibration and predictive robustness across frequencies and horizons (Figures 10–16).

Figure 7: Loop Seattle/H—medium-horizon quantile forecast.

Figure 8: KDD Cup 2018/H—medium-horizon quantile forecast.

Figure 9: Electricity/H—long-horizon quantile forecast.

Ablation, Sensitivity, and Inference Paradigm Analysis

Comprehensive ablation studies (Figure 10) rigorously isolate the contribution of UPDA negative prototype keys, gated fusion, timestamp/mask components, and training strategies. Removing negative keys ($K_\text{neg}$) leads to the most severe performance loss, confirming Falcon-X's ability to capture antagonistic, dual-dependency dynamics is non-negotiable for heterogeneous multivariate modeling.

Figure 10: (a) Ablation validates necessity of key architectural components; (b) inference paradigm analysis highlights superiority over Chronos-2 group attention.

Inference setting analysis reveals that group attention in Chronos-2 does not improve performance, evidencing semantic collapse and over-smoothing in raw variate space (Figure 1b). In contrast, Falcon-X's multivariate inference mode consistently enhances predictive accuracy without degrading univariate results, confirming transferable cross-variate modeling.

Sensitivity analysis shows that balanced allocation between temporal modeling ($n$) and cross-variate routing ($l$) is critical. An optimal trade-off is achieved at $16/16$ layers, and increasing prototype dimension ($C$) to 6–8 yields peak expressivity without redundancy.

Implications and Future Directions

Falcon-X's latent prototype paradigm introduces a scalable, expressively aligned framework for universal multivariate time series modeling. Practically, this architecture enables deployment across domains with extreme heterogeneity—energy, economics, healthcare, cloud observability—without retraining or manual feature engineering. Theoretical implications extend to foundation modeling of complex dynamical systems and the principled abstraction of physical variates into reusable semantic anchors. Its dual-dependency expressivity and strict scaling behavior will inform both future TSFM design and broader AI foundation model research.

Anticipated future developments include hierarchical semantic prototype spaces, integration of exogenous covariates via prototype augmentation, lifelong adaptation of prototypes under online learning, and further generalization to irregular and asynchronous time series modalities.

Conclusion

Falcon-X operationalizes a unified latent prototype space for heterogeneous multivariate forecasting, resolving semantic alignment and relational expressivity concerns inherent in prior TSFMs. Empirical evaluations across benchmarks confirm its superiority, generalization, and scalable modeling capacity. The architecture's innovations and results decisively establish the latent prototype routing paradigm as a prevailing solution for universal multivariate time series foundation modeling (2605.27286).