---
title: 'DualSG: Dual-Stream Semantic Forecasting'
url: https://www.emergentmind.com/topics/dualsg
type: topic
---

# DualSG: Dual-Stream Semantic Forecasting

DualSG is a dual-stream explicit semantic-guided multivariate time series forecasting framework designed to systematically address the limitations of both end-to-end large language model (LLM)-based forecasters and latent space modality-alignment approaches in multivariate time series forecasting (MTSF). Unlike end-to-end LLM approaches that suffer from numerical imprecision and modality misalignment, DualSG positions the LLM as a semantic guide within a structured dual-stream architecture, providing interpretable natural language trend summaries and explicit, lightweight fusion mechanisms to refine, rather than replace, traditional numerical forecasts. This explicit separation enables DualSG to robustly exploit both fine-grained numeric dynamics and trend-level semantic context for state-of-the-art forecasting accuracy across diverse real-world datasets [2507.21830].

## 1. DualSG Architecture and Functional Components

DualSG operates on an input $X \in \mathbb{R}^{N \times T}$ (where $N$ is the number of channels/variables and $T$ the history length), producing a fused forecast $\hat{Y} \in \mathbb{R}^{N \times F}$. The architecture comprises two parallel streams:

**Numerical Forecasting Stream**:
- Utilizes Multi-scale Adaptive Patching (MAP) to partition the input sequence into three learnable temporal regions (near, mid, far) via quantile thresholds, then generates region-specific overlapping patches processed by independent MLPs, yielding region-specific embeddings.
- Adaptive Importance Masking computes a gradient-based patch relevance score, selects the top-k informative patches (via differentiable SoftTopK), and masks out less relevant patches, reducing both noise and computation.
- The selected patch embeddings are concatenated and passed through a lightweight temporal encoder (e.g., shallow Transformer blocks) to produce a base numeric forecast $\hat{Y}_{\text{base}}$.

**Textual Reasoning Stream**:
- Employs a Time Series Caption Generator (TSCG), a frozen small encoder-decoder, to summarize each univariate channel $X_{n,:}$ into a one-sentence natural-language caption $S_n$, e.g., “The time series exhibits a gradual increase, then plateaus.”
- The encoder processes numerical patches with attention and positional encodings; the decoder is a GPT-style autoregressive model utilizing cross-attention to time features.
- Captioning is supervised by a cross-entropy loss, and the TSCG remains frozen during the main DualSG training.

**Fusion Modules**:
- **SemFuse (Caption-Guided Channel Fusion):** For each channel pair $(n,i)$, compute semantic affinity using LLM-encoded caption embeddings. The top-K related channels are selected and features fused via GeLU-activated MLPs, reducing complexity to $O(N)$.
- **Cross-Stream Fusion with STAM:** Caption embeddings for all channels are aggregated and projected via a frozen LLM into semantic correction terms $\Delta \hat{Y}$; a learned spatio-temporal attention mask $\mathcal{W}_{\text{STAM}}$ adaptively balances numerical and semantic corrections per channel and forecast step:
  $$
  \hat{Y} = \mathcal{W}_{\text{STAM}} \odot \hat{Y}_{\text{base}} + (1 - \mathcal{W}_{\text{STAM}}) \odot \Delta \hat{Y}
  $$

This joint system enables explicit trend- and event-level semantics to refine numeric predictions without direct end-to-end mapping.

## 2. Time Series Caption Module: Semantic Summarization and Context

The TSCG provides explicit, interpretable, and standardized input to the LLM by generating one-sentence, channel-wise captions summarizing three trend axes: direction (e.g., “rising”, “falling”), intensity (e.g., “sharp”, “gradual”), and transitions (“then”, “followed by”). Empirically, “Caption”-style prompts outperform domain-, timestamp-, statistics-, or instruction-based prompts in forecasting benchmarks, both in terms of mean squared error (MSE) and number of best-performing runs (Table 6; Fig. 5 in [2507.21830]).

No additional bespoke trend feature formulas are introduced; feature extraction relies on MLP projection and attention mechanisms within the TSCG and subsequent LLM encoding. TSCG is pre-trained with a cross-entropy objective and is not updated during DualSG training, preventing label leakage and ensuring reproducible semantic context.

## 3. Role of the LLM as Explicit Semantic Guide

The frozen LLM (e.g., GPT2) is used exclusively for embedding the generated captions. For each channel, the caption $S_n$ is passed through the LLM to obtain a context vector $\mathrm{LLM}(S_n)\in\mathbb{R}^d$. These vectors are:
- Used in SemFuse to compute semantic relevance scores and guide sparse neighbor selection for channel fusion.
- Aggregated in STAM to provide semantic correction terms for the fused forecast.

Prompt engineering is minimal and restricted to the choice and formatting of the caption sentence. During main training, no explicit loss is imposed to align the numerical and textual streams beyond the end-to-end L1 loss for final forecast accuracy:
$$
\mathcal{L}_{\text{forecast}} = \|Y - \hat{Y}\|_1
$$

Ablation studies demonstrate that direct LLM involvement via caption embeddings is critical for semantic guidance, but freezing LLM and TSCG parameters supports training efficiency and modularity.

## 4. Explicit Caption-Guided Fusion: Mathematical Formulation and Computational Efficiency

SemFuse computes a semantic affinity $g_{n,i}$ via a learned projection of caption embeddings, followed by sparsification to select top-K relevant channels:
$$
g_{n,i} = \sigma(W_{\mathrm{proj}}\, e_n)[i]
$$
$$
F_n^{\mathrm{fused}} = \mathrm{GeLU}\big(W_{\mathrm{fuse}}[F_n; \sum_i \mathcal{M}_{n,i} g_{n,i} F_i]\big)
$$
where $\mathcal{M}_{n,i}$ is a binary mask indicating the top-K channels per $n$.

Cross-stream fusion via STAM adaptively integrates semantic corrections from caption-driven LLM encodings with the numerical forecast, with per-channel/per-horizon weighting:
$$
\hat{Y} = \mathcal{W}_{\mathrm{STAM}} \odot \hat{Y}_{\mathrm{base}} + (1-\mathcal{W}_{\mathrm{STAM}}) \odot \Delta\hat{Y}
$$

By focusing on trend-level, interpretable embeddings and using sparse, semantics-driven fusion, DualSG avoids the $O(N^2)$ cost of full channel-wise attention while circumventing overfitting to local noise and improving interpretability in inter-variable relationships.

## 5. Training Procedures and Experimental Setting

The forecast stream is trained end-to-end with an L1 loss, with the caption generator and LLM both frozen. Training uses Adam optimizer, batch size 32, and cosine annealing schedule on 4× NVIDIA A800 GPUs. Gradient accumulation and memory-efficient attention mechanisms are used for scalability. Forecasting performance is evaluated on MSE and MAE over hold-out sets.

No joint fine-tuning of caption generator or LLM is performed. Only the numerical encoder, projection, fusion, and STAM weighting parameters are learned during DualSG training. Caching pre-computed LLM caption embeddings reduces training time by approximately 72% (Table 7 on ETTh2), with no measurable degradation in accuracy.

## 6. Empirical Performance and Component Analysis

DualSG is evaluated on 12 real-world MTS datasets spanning domains such as weather, traffic, electricity, and solar energy. Baselines include MLP, CNN, Transformer, and LLM-based methods. Across all tasks and forecast horizons (H = {96, 192, 336, 720}), DualSG achieves best-in-class results (lowest MSE/MAE in 65 out of 80 settings). Notable gains include:
- Solar-Energy: MSE 0.192 (vs. CALF 0.324; 40.7% reduction).
- Weakly correlated series (ETTm2): MSE 0.254 (Crossformer: 1.216; PatchTST: 0.285).
- Non-stationary series (Traffic, Weather): MSE reductions of 35.1% and 26.2% vs. FEDformer [2507.21830].

Ablation studies confirm that disabling TSCG, MAP, or SemFuse significantly degrades performance, while the advantage of using the LLM (as opposed to raw caption text) is quantitatively positive but smaller. Prompt experiments show trend-focused captions (versus other prompt types) account for marked improvements, and trend-MAE reductions (up to 20.9%) on long-horizon forecasts validate the value of semantic guidance for slow-changing components.

## 7. Limitations and Prospective Developments

The introduction of a dual-stream pipeline, with caption generation and stream fusion, entails additional computation and can slow inference relative to singlestream baselines. The fixed lookback window in the numerical encoder can hinder modeling of strong seasonal components or unusual periodicities. As future work, the authors outline directions in efficiency optimization, richer multimodal data integration, and the extension of explicit semantic guidance to broader classes of time series tasks or more granular temporal structures. The observed limitation that LLM-based semantic correction dominates mainly for trend- or shift-driven dynamics suggests further investigation into how explicit semantics can be leveraged for other temporal phenomena.

In summary, DualSG provides a paradigm wherein numerical forecasting and semantic reasoning are explicitly decoupled yet synergistically fused, yielding interpretable, efficient, and accurate forecasts in complex, high-dimensional multivariate time series [2507.21830].

Source: https://www.emergentmind.com/topics/dualsg