Towards a Unified Generative Model for Scarce Time Series with Domain Experts

Abstract: Synthesizing realistic time series with generative models has wide-ranging applications in real-world scenarios. Despite recent progress, most existing methods are trained under the assumption of abundant training data, which substantially limits their effectiveness in data-scarce settings. In this paper, we propose TimeMoDE, a novel framework that integrates Diffusion Transformers with Mixture-of-Experts to exploit both domain adaptability and diffusion-stage awareness for time series generation under data scarcity. It is pre-trained on a large-scale collection of multi-domain datasets to extract domain-agnostic temporal representations and domain-specific information benefiting generalization during fine-tuning. We propose Domain Prompts to condition expert assignment for indistinguishable noised tokens, mitigating the limitations of capturing inter-dataset relationships. Moreover, we incorporate diffusion timestep signals to equip the experts with awareness of time series degradation variations, facilitating adaptive calibrate to stage-dependent denoising requirements. Extensive experiments demonstrate that TimeMoDE outperforms existing methods under diverse low-data settings. It establishes an innovative paradigm for advanced time series few-shot generation.

• The paper introduces TimeMoDE, a diffusion-based generative model that integrates a transformer backbone with a Mixture-of-Experts layer to address extreme data scarcity.
• It employs domain prompts, prototype-guided expert routing, and diffusion timestep conditioning to achieve notable improvements, including a 25% reduction in c-FID and over 30% boost in discriminative score in few-shot settings.
• Empirical evaluations demonstrate TimeMoDE’s capacity to generalize across domains with minimal data, offering high-fidelity data augmentation for applications in healthcare, finance, and science.

Unified Generation of Scarce Time Series: Analysis of TimeMoDE

Overview

The paper "Towards a Unified Generative Model for Scarce Time Series with Domain Experts" (2606.15172) introduces TimeMoDE, a diffusion-based generative framework designed to address the challenge of synthesizing realistic, domain-adapted time series data under extreme data scarcity. TimeMoDE unifies domain adaptability and diffusion-stage awareness by leveraging a transformer backbone integrated with a Mixture-of-Experts (MoDE) layer, enabling generalization across multiple domains. This work specifically targets scenarios where the availability of high-quality time series data is restricted due to privacy, cost, or acquisition constraints.

Motivation and Problem Formulation

Contemporary time series generative models, particularly those based on denoising diffusion probabilistic models (DDPMs), presuppose abundant single-domain data during training, a condition rarely met in practical applications. Under realistic, low-data settings, models trained in this paradigm demonstrate degraded performance due to inability to capture inter-domain relationships and domain-specific noise structures. The central technical impediment arises from (i) the indistinguishability of noised tokens obscuring domain semantics during diffusion, and (ii) heterogeneous degradation profiles across domains, which confound the diffusion-stage embeddings and impair accurate denoising. Previous approaches employing class labels or language-conditioned prompts are either overly rigid or lack the granularity to capture cross-domain temporal nuances.

Architectural Innovations

Diffusion Transformer Backbone with MoDE

TimeMoDE is built upon a Diffusion Transformer (DiT) architecture equipped with a multi-layer MoDE module. Traditional MLPs in the transformer blocks are replaced with sparsely-activated, domain-specialized experts. Unlike conventional dense parameter activation, MoDE selectively activates a subset of experts per token, promoting modeling efficiency and domain specificity. Experts are further augmented with lightweight embeddings of the diffusion timestep, introducing stage awareness directly into expert processing.

Expert Routing via Domain Prompts and Prototypes

To resolve expert assignment under indistinguishable noise, a two-stage routing mechanism is proposed:

• Domain Prompts (DP): For each dataset, representative exemplars are encoded into global embeddings using learnable combinations of convolution and pooling layers. These embeddings act as compressed ‘prompts’ summarizing domain characteristics.
• Prototypes and Basis Vectors: Each expert is associated with a prototype—a learnable subspace composed of orthogonal basis vectors. The DP associated with each input is matched via cosine similarity to these prototypes, determining activation weights for each expert.

This design encourages intra-prototype diversity and inter-prototype separability, which is enforced by dedicated orthogonality and discriminability losses during pre-training.

Diffusion Timestep Conditioning

The model incorporates diffusion step information both in the global conditioning and directly into expert layers through adaptive layer normalization. This enables experts to distinctly process different degradation regimes, addressing the issue of variable noise attenuation and non-stationarity intrinsic to multi-domain time series under diffusion.

Training Protocol

TimeMoDE employs a two-stage training protocol:

1. Pre-training: Conducted on a comprehensive multi-domain corpus, jointly optimizing diffusion, prototype-based, and load-balancing auxiliary losses. Prototypes and expert weights are refined in this phase.
2. Fine-tuning: Targeted at a previously unseen domain under data scarcity (as little as 10 samples), only domain-relevant experts are activated and prototypes are frozen, mitigating overfitting and spurious transfer from unrelated domains.

Empirical Evaluation

TimeMoDE is extensively benchmarked against recent SOTA time series generative models including ImagenFew, ImagenTime, DiffusionTS, and KoVAE under both few-shot and full-shot regimes on multiple real-world datasets. Evaluation metrics capture fidelity (c-FID), discriminability, predictive accuracy, population-level distributional alignment, and preservation of temporal dependencies.

Key Results

• Few-shot superiority: On average, TimeMoDE achieves a 25% reduction in c-FID and over 30% improvement in discriminative score relative to the strongest baselines at 5–15% training data or ≤50-shot settings.
• Generalization: When fine-tuned with only 5% of target domain data, TimeMoDE surpasses models trained from scratch on the full dataset, demonstrating strong transfer from pre-trained representations.
• Ablation Evidence: Dense expert variants (removing sparsity) and removal of prompt/prototype routing yield consistently inferior results, underscoring the necessity of the proposed MoDE routing and domain conditioning.
• Expert Activation Dynamics: Analysis shows shallow transformer layers activate similar experts for related domains, while deeper layers diversify, capturing fine-grained distinctions emerging during iterative denoising.

Theoretical and Practical Implications

The introduction of routed, diffusion-aware experts represents a methodological advancement for generative modeling under severe data scarcity. TimeMoDE enables foundation models to generalize not only across tasks, but also across temporal and distributional heterogeneity of time series from varied application domains. The efficacy of prompt- and prototype-guided expert selection highlights a direction for controllable generative models integrating prior knowledge and structure, rather than solely relying on end-to-end learning.

This architecture is poised to (i) facilitate high-fidelity data augmentation where collection is prohibitive, (ii) support reliable downstream forecasting/classification in medical, financial, or scientific settings with privacy restrictions, and (iii) serve as a blueprint for future AI systems that must operate in continually-evolving, multi-domain, and data-constrained environments.

Future Outlook

Potential research directions include leveraging language-based or multimodal prompts to further enhance domain transferability, dynamic expert allocation conditioned on streaming data, and scaling to billions of parameters for universal time series foundation models. Integration with causal discovery mechanisms and uncertainty quantification could further empower decision-critical time series applications.

Conclusion

TimeMoDE establishes a framework for unified cross-domain time series generation under acute data scarcity by synergistically combining domain-expert routing, diffusion-stage awareness, and large-scale pre-training. Empirical and analytic results affirm significant gains over existing generative paradigms, providing a platform for future advancement in data-scarce sequence modeling.