Multi-layer Steerable Embedding Fusion (MSEF)

• The paper introduces a dual-foundation architecture that fuses semantically rich TS embeddings into every LLM layer via learnable steering vectors.
• MSEF is a parameter-efficient fusion framework that addresses TS signal dilution by repeatedly injecting TS information throughout the transformer layers.
• The method demonstrates significant forecasting gains, achieving up to 31% MSE reduction over state-of-the-art models on multivariate, few-shot long-term forecasting benchmarks.

Multi-layer Steerable Embedding Fusion (MSEF) is a parameter-efficient fusion framework introduced for integrating time series (TS) data into LLMs for few-shot multivariate long-term forecasting. In its original formulation, MSEF addresses a recurrent limitation of TS–LLM adaptation methods: TS information is usually injected only at the input layer, after which its influence progressively fades in deeper transformer blocks. MSEF instead combines a frozen LLM with a frozen time series foundation model (TSFM), uses the TSFM to produce semantically rich TS embeddings, and injects those embeddings into every intermediate LLM layer through layer-specific steering vectors. The result is a dual-foundation-model architecture intended to preserve TS signal throughout depth, improve TS–text alignment, and support efficient few-shot adaptation while training only the steering vectors and a lightweight output projection layer (Chen et al., 22 Aug 2025).

1. Origin, motivation, and forecasting setting

MSEF was introduced in "Integrating Time Series into LLMs via Multi-layer Steerable Embedding Fusion" (Chen et al., 22 Aug 2025) in response to four difficulties in TS–LLM integration. First, there is a modality mismatch: LLMs are pretrained on discrete textual tokens and hierarchical linguistic representations, whereas TS data are continuous, high-dimensional, and weakly “semantic” in the natural-language sense. Second, prior TS–LLM approaches are typically shallow: textualization methods convert numeric values into long, semantically sparse prompts, while vectorization methods project TS into continuous vectors compatible with the LLM input embedding, but in both cases TS information is injected only at the input layer. Third, because the TS signal is present only at the input, it is progressively diluted as hidden states propagate through many transformer layers and become dominated by the LLM’s pretraining biases toward text. Fourth, TS embeddings are not explicitly aligned with the hierarchical representations formed inside the LLM, leaving no mechanism to ensure that lower layers emphasize local temporal structure while higher layers capture broader patterns and semantic interpretations.

The forecasting problem studied by MSEF is few-shot multivariate long-term forecasting. Given a historical sequence

$X_T \in \mathbb{R}^{N \times T},$

with $N$ variables and history length $T=512$, the task is to predict

$$\widehat{X}_H \in \mathbb{R}^{N \times H}, \qquad H \in \{96, 192, 336, 720\}.$$

The optimization target is mean squared error,

$$\mathcal{L}_\text{forecast} = \frac{1}{N H} \sum_{i=1}^{N} \sum_{t=1}^{H} \left(\widehat{X}_{H, i, t} - X_{H, i, t}\right)^2,$$

while mean absolute error is also reported as an evaluation metric. Experiments are conducted on seven multivariate benchmarks: ETTh1, ETTh2, ETTm1, ETTm2, ECL, Weather, and Traffic. The regime is explicitly data-scarce: only 10% of time steps are used for training (Chen et al., 22 Aug 2025).

2. Architectural organization

MSEF is described as a frozen-LLM, frozen-TSFM architecture in which Pythia-1B provides a 32-layer textual representation stack and MOMENT-1-large provides semantically rich TS embeddings (Chen et al., 22 Aug 2025). The core architectural departure from input-only fusion is that TS representations are attached to every intermediate LLM layer, not just to the prompt embedding.

The data flow begins by encoding the historical TS $X_T$ with MOMENT-1-large to obtain a TS representation $e_\text{TS}$. In parallel, a textual prompt describing the forecasting task is tokenized and embedded by the LLM, producing

$h^{(0)}_\text{text} \in \mathbb{R}^{L \times d},$

where $L$ is token length and $d$ is the LLM hidden size. For each transformer layer $N$0, MSEF introduces a learnable steering vector $N$1. The TS embedding and steering vector are concatenated to form a per-layer prefix:

$N$2

That prefix is then fused with the current textual hidden states and passed into the next transformer block:

$N$3

After the final transformer layer, the terminal textual representation is mapped to the forecast by a lightweight output projection:

$N$4

where $N$5 denotes pooling or position selection. The paper treats $N$6 as a fixed embedding over layers for fusion, even though MOMENT itself is hierarchical. Operationally, the method follows a prefix-tuning-style interface: each transformer block processes both the usual textual tokens and a dedicated TS-plus-steering prefix (Chen et al., 22 Aug 2025).

3. Steerability and cross-modal alignment

The two defining properties of MSEF are encoded directly in its name. “Multi-layer” means that TS embeddings are injected into every transformer layer, so the LLM can repeatedly access TS patterns as it builds deeper representations. “Steerable” means that each layer has its own learnable steering vector $N$7, which modulates how the TS embedding interacts with the layer’s textual hidden states (Chen et al., 22 Aug 2025).

These steering vectors are not simple scalar weights. They are trainable, layer-specific parameters that live in a compatible embedding space and are concatenated with TS embeddings to form per-layer prefixes. The paper’s operational description is explicit: “the trainable dynamic steering vectors and the representations of TS data are first concatenated as prefixes of the output of each LLM layer. Then, they are passed to the next LLM layer as input.” This places MSEF close to the prefix-tuning paradigm rather than to explicit cross-attention or LoRA-based adaptation. The paper does not specify a more detailed matrix-form fusion such as gating or cross-attention; alignment is instead mediated by repeated prefix injection and by optimizing forecasting loss.

The intended effect is hierarchical TS–text alignment. Lower layers are described as focusing on “original feature alignment,” middle layers on pattern recognition, and upper layers on semantic interpretation and decision-ready representations. This layered division is enabled by the fact that each depth receives the same TS embedding but a different steering vector. A central claim of the method is therefore not merely that TS features are present throughout the model, but that their interaction with the textual stream is differentiated by layer. The paper also emphasizes what MSEF is not: there is no explicit extra regularization term or contrastive alignment loss, and there is no LoRA or low-rank adaptation; the trainable components are the steering vectors and the output projection (Chen et al., 22 Aug 2025).

4. Training strategy and parameter efficiency

MSEF is designed for few-shot adaptation under a strongly constrained training budget. In the reported setup, both backbones are frozen: the LLM is Pythia-1B and the TS foundation model is MOMENT-1-large. Only the set of layer-specific steering vectors and a light output projection layer are optimized, using Adam and early stopping on validation loss (Chen et al., 22 Aug 2025).

This training regime makes the method explicitly parameter-efficient. The adaptation capacity is concentrated in $N$8 steering vectors—one per transformer layer—and a small output head, while the pretrained capabilities of both foundation models are preserved. The paper draws a conceptual connection to prefix-tuning and P-tuning in NLP: MSEF behaves like a TS-specialized soft-prompt mechanism, except that the prefixes are formed jointly by TSFM representations and steering vectors rather than by arbitrary learned prompt embeddings.

Prompting remains part of the pipeline, but the prompt is not the primary carrier of numerical information. The textual prompt provides task context such as the forecasting instruction, dataset type, and possibly horizon $N$9 and input length $T=512$0, while the numerical pattern information is supplied by the TSFM embedding through multi-layer fusion. The paper does not detail exact prompt templates, which leaves the linguistic interface underspecified relative to the fusion mechanism. It does, however, provide an implementation recipe: select pretrained base models, extract TS embeddings, define one steering vector per LLM layer, modify each transformer block to accept the TS-plus-steering prefix, add a small output layer, and train only the steering vectors and output projection. The code is reported as publicly available at https://github.com/One1sAll/MSEF (Chen et al., 22 Aug 2025).

5. Empirical performance and ablation evidence

Across the seven multivariate benchmarks and four forecast horizons, MSEF is reported to outperform TimesNet, FEDformer, Autoformer, Stationary, ETSformer, LightTS, and Informer in the 10%-data few-shot regime (Chen et al., 22 Aug 2025). The summary result is that MSEF achieves the best MSE and MAE on every dataset in 69 out of 70 comparisons. The paper reports an average MSE reduction of 31.8% versus TimesNet and 31.1% versus FEDformer, and a performance improvement of up to 59.5% versus LightTS.

The ETTh1 example illustrates the scale of the reported gains. Averaged over the four horizons, MSEF obtains MSE $T=512$1 and MAE $T=512$2, compared with TimesNet at MSE $T=512$3, MAE $T=512$4; FEDformer at MSE $T=512$5, MAE $T=512$6; and Informer at MSE $T=512$7, MAE $T=512$8. The paper further states that MSEF remains superior across electricity, traffic, weather, and industrial domains, which it interprets as strong cross-domain generalization.

The ablation study isolates the role of both multi-layer injection and steering. A plain LLM-based variant that converts TS to text and feeds it directly to the LLM is much worse on ETTh1, with average MSE $T=512$9 and MAE $$\widehat{X}_H \in \mathbb{R}^{N \times H}, \qquad H \in \{96, 192, 336, 720\}.$$0. A variant without steering vectors, but still with layer-wise TS injection, improves sharply to MSE $$\widehat{X}_H \in \mathbb{R}^{N \times H}, \qquad H \in \{96, 192, 336, 720\}.$$1 and MAE $$\widehat{X}_H \in \mathbb{R}^{N \times H}, \qquad H \in \{96, 192, 336, 720\}.$$2. Full MSEF further improves ETTh1 to MSE $$\widehat{X}_H \in \mathbb{R}^{N \times H}, \qquad H \in \{96, 192, 336, 720\}.$$3 and MAE $$\widehat{X}_H \in \mathbb{R}^{N \times H}, \qquad H \in \{96, 192, 336, 720\}.$$4, and yields Weather average MSE $$\widehat{X}_H \in \mathbb{R}^{N \times H}, \qquad H \in \{96, 192, 336, 720\}.$$5 and MAE $$\widehat{X}_H \in \mathbb{R}^{N \times H}, \qquad H \in \{96, 192, 336, 720\}.$$6. These results are used to support the claim that both repeated TS injection and layer-specific steering are necessary. An additional depth ablation shows that steering applied to deeper and broader subsets of layers tends to improve performance, reinforcing the idea that the benefit is genuinely multi-layer rather than shallow (Chen et al., 22 Aug 2025).

6. Relation to prior methods, broader interpretations, and scope

Within time-series forecasting, MSEF is positioned against three main families of LLM-based methods. Textualization approaches convert TS values into digits or textual descriptions and rely on the LLM to interpret a long prompt. Vectorization approaches attach a projected TS embedding to the LLM input. Knowledge-distillation approaches such as TimeKD emphasize teacher–student transfer rather than deep TS–LLM integration. MSEF differs from all three by using a TS foundation model explicitly, injecting TS information at every LLM layer, and adapting the fusion behavior by layer-specific steering vectors rather than by input-only conditioning (Chen et al., 22 Aug 2025).

The term “MSEF” has also acquired a broader interpretive role in subsequent literature, although not always as the name of a single standardized architecture. "Layer Embedding Deep Fusion Graph Neural Network" states that Multi-layer Steerable Embedding Fusion can be naturally understood through its Layer Embedding Deep Fusion operator and Dual-Topology Parallel Strategy, extending the idea of steerable multi-layer fusion to heterophilic graph learning (Xu et al., 25 Apr 2026). "Layer-Aware Embedding Fusion for LLMs in Text Classifications" presents its layer-selection and multi-model fusion procedures as ingredients that can inform a design for MSEF in text classification (Gwak et al., 8 Apr 2025). "Beyond the Last Layer: Multi-Layer Representation Fusion for Visual Tokenization" describes DRoRAE as very close to MSEF in practice, emphasizing controllable multi-layer fusion through routing and incremental correction (Zhu et al., 11 May 2026). "Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices" frames its findings on stage-diverse layer selection and external direct fusion as evidence for designing a hypothetical MSEF module in multimodal systems (Lin et al., 8 Mar 2025).

This pattern suggests that MSEF is functioning in two related senses. In the strict sense, it denotes the TS–LLM forecasting framework introduced in (Chen et al., 22 Aug 2025). In a broader methodological sense, it is being used as a label for architectures that fuse embeddings across depth with learnable control over layer contributions. The original paper itself points toward extension beyond forecasting, proposing time-series explanation, anomaly detection and narrative reporting, and decision support and planning as possible downstream directions. A plausible implication is that the essential design principle of MSEF is not the forecasting head, but the repeated, layer-specific reintroduction of non-textual structure into a frozen deep LLM or related backbone (Chen et al., 22 Aug 2025).