Distribution-Aware Diffusion-LLM for Robust Ultra-Long-Term Time Series Forecasting

Abstract: Time series forecasting is a fundamental machine learning task. Recent work has explored LLMs for this purpose due to their strong generalization, pattern recognition, and zero-shot or few-shot capabilities. Despite their suitability for long-context learning, LLMs face challenges in multimodal settings: they lack calibrated probabilistic modeling for non-text data and struggle to align heterogeneous representations. To address these issues, we propose a new framework Diffusion-LLM that integrates a conditional diffusion model into an LLM-based forecasting pipeline. This joint design enables learning the conditional distribution of future data while improving semantic alignment in a shared latent space. We evaluate Diffusion-LLM on six long-term forecasting benchmarks, including ETT, Weather, and ECL. Our method consistently outperforms existing LLM-based baseline, achieving notable gains in ultra-long-term and few-shot forecasting and demonstrating the value of distribution-aware regularization for enhancing robustness and generalization in time series LLMs.

• The paper introduces a novel Diffusion-LLM framework that integrates a conditional DDPM with LLM embeddings to enable robust ultra-long-term forecasting.
• The model achieves significant error reductions, including up to 25.79% improvement in few-shot settings, and ensures enhanced probabilistic calibration.
• The approach offers efficient multimodal alignment and diffusion-based regularization while maintaining minimal computational overhead.

Distribution-Aware Diffusion-LLM for Ultra-Long-Term Time Series Forecasting

Framework Overview and Motivation

Recent advances in time series forecasting have leveraged LLMs due to their generalization, pattern recognition, and zero-/few-shot abilities. However, pretrained LLMs exhibit modality misalignment for non-textual data, and deterministic objectives (e.g., MSE) fail to capture the full uncertainty and multimodal structure present in long-horizon forecasting. This framework introduces Diffusion-LLM, which integrates a conditional denoising diffusion probabilistic model (DDPM) as an auxiliary regularizer within an LLM-based semantic embedding space. This joint design enables learning the conditional distribution of forecast embeddings given past context, enhancing probabilistic calibration and multimodal alignment.

Figure 1: Diffusion-LLM training architecture incorporates patch encoding and a conditional DDPM regularizer in a shared token embedding space.

The architecture consists of three components: (A) reprogramming and patch embedding of time series into the LLM prototype space, (B) forecasting via a frozen LLM body with trainable output projection, and (C) diffusion-based regularization via joint embedding space denoising. This approach avoids full LLM fine-tuning, maximizing efficiency while leveraging distribution-aware regularization.

Time Series Reprogramming and Multimodal Alignment

The patch encoder maps normalized time series segments to text-like prototypes derived from LLM vocabulary using an attention mechanism, enabling semantic processing for non-textual patterns.

Figure 2: Example of prompt encoding for dataset-specific task specification.

Prototype composition analysis reveals that the learned embedding space forms higher affinity for time series-specific vocabulary after attention layers, which enhances multimodal alignment and robustness against hallucinations.

Forecasting Pipeline and Diffusion Regularization

After encoding both lookback and forecast windows via the shared LLM encoder, the input representation is used for direct forecasting $\hat{y}$ through the frozen LLM body and output projection. The forecast loss $\mathcal{L}_{\text{forecast}}$ is combined with the diffusion loss $\mathcal{L}_{\text{ddpm}}$, which regularizes the embedding space by predicting added noise in a conditional denoising process. The total loss $$\mathcal{L}_{\text{joint}} = \mathcal{L}_{\text{forecast}} + \lambda \mathcal{L}_{\text{ddpm}}$$ balances deterministic prediction and probabilistic refinement.

Ablation studies demonstrate that joint encoders and simple concatenation for conditioning are most effective, with no benefit from overparameterized architectures (e.g., U-Net) or explicit feature-ID conditioning.

Empirical Evaluation and Results

Diffusion-LLM is evaluated on six established benchmarks, including ETTh1, ETTh2, ETTm1, ETTm2, Weather, and ECL, across long-term and ultra-long-term horizons (up to 2048 steps), and in low-data (few-shot) regimes.

Figure 3: Ultra-long-term forecast (2048 steps) for ETTh1; Diffusion-LLM remains closer to ground truth than TimeLLM in late regions and shows slower performance degradation under data scarcity.

Diffusion-LLM achieves superior robustness and lower forecast error compared to TimeLLM, especially in ultra-long-term and few-shot settings. On ETTh1 and ETTh2, the MSE reductions are 19.26% and 11.38% respectively for ultra-long-term, and up to 25.79% in extreme few-shot (5%) scenarios. Diffusion-LLM outperforms the baseline on 4/6 datasets in the challenging regime, demonstrating stronger distributional modeling and generalization.

Figure 4: Optimal regularization at $\lambda=1$ achieves lowest MSE (0.729) for ETTh1-2048, balancing deterministic and diffusion objectives. Both under- and over-regularization degrade performance.

Moreover, Diffusion-LLM exhibits minimal overhead (+1.82% GPU memory, +11.54% parameters, +0.39% slower training) compared to TimeLLM, while inference speed is unchanged due to exclusive use of LLM modules.

Multimodal Alignment Analysis

Prototype attention visualization suggests Diffusion-LLM places significantly stronger focus on time series-specific prototypes in ultra-long forecasting, which stabilizes semantic alignment and mitigates uncertainty amplification and hallucinations frequently observed in LLM-based forecasters.

Practical and Theoretical Implications

Integrating diffusion models as regularizers within LLM-based semantic spaces enables the joint modeling of temporal uncertainty and modality alignment, overcoming deterministic regression bias and modality mismatch. This strategy is readily extendable to other multimodal forecasting domains and enables uncertainty-aware prediction without loss of LLM efficiency. The approach is alignment-agnostic, compatible with existing LLM frameworks, and particularly advantageous in high-uncertainty and low-data regimes.

Practically, Diffusion-LLM expands the applicability of LLMs to robust forecasting in energy, climate, healthcare, and supply chain applications requiring ultra-long-term prediction from limited data. Theoretically, this framework suggests a new paradigm for auxiliary probabilistic modeling in deep learning systems, supporting future developments in multimodal generative forecasting, uncertainty quantification, and adaptive embedding reprogramming.

Conclusion

Diffusion-LLM introduces distribution-aware regularization for LLM-based time series forecasting through conditional diffusion modeling, demonstrably enhancing performance for ultra-long-horizon and low-data scenarios. The framework achieves improved robustness, probabilistic calibration, and multimodal alignment without sacrificing efficiency. Future directions include adaptive reprogramming, direct generative forecasting, uncertainty-aware multi-prediction, and extension to additional modalities and embedding spaces. Diffusion-LLM sets a principled foundation for robust, uncertainty-aware long-term forecasting in LLMs.