FTS-Text-MoE: Sparse Multimodal Stock Forecasting

• The paper introduces a decoder-only Transformer that replaces dense feed-forward blocks with a sparse Mixture-of-Experts to directly forecast multi-horizon stock prices.
• FTS-Text-MoE is defined by its day-level alignment of numerical prices and textual summaries via point embeddings, coupled with multi-resolution forecasting heads.
• Empirical results show improved portfolio returns and Sharpe ratios when text is integrated, though challenges remain in temporal alignment and expert routing.

FTS-Text-MoE is a decoder-only Transformer for stock forecasting that combines financial time-series inputs with aligned textual evidence from news and social media, and replaces dense feed-forward blocks with a sparse Mixture-of-Experts mechanism. It is designed to forecast future stock prices directly rather than only short/fixed horizons or direction labels, while incorporating richer textual semantics than sentiment-only pipelines. Its defining design choices are day-level alignment of numerical and textual representations through “point embeddings,” a decoder-only MoE backbone with top-$K$ sparse routing and a shared expert, and multi-resolution forecasting heads that support flexible prediction horizons during inference (Xu et al., 28 Jul 2025).

1. Problem formulation and research motivation

FTS-Text-MoE is framed as a response to three limitations in prior financial forecasting systems, especially prompt-based LLM approaches: input length restrictions, a tendency to predict only short/fixed horizons or direction labels rather than actual future values, and reliance on dense computation that is expensive at inference time. The model therefore targets direct value forecasting with multimodal conditioning, using historical prices together with public news and X/Twitter comments (Xu et al., 28 Jul 2025).

The forecasting problem is defined by historical numerical observations

$$\mathbf{X}_{1:M} = (x_1, x_2, \ldots, x_M) \in \mathbb{R}^M$$

and corresponding textual data

$$\mathbf{T}_{1:M} = (t_1, t_2, \ldots, t_M) \in \mathbb{R}^M,$$

with the objective of predicting the next $N$ days: $$\hat{\mathbf{X}}_{M+1:M+N} = f_\theta(\mathbf{X}_{1:M}, \mathbf{T}_{1:M}) \in \mathbb{R}^N.$$ The model is described as supporting flexible context length $M$ and horizon $N$ during inference. It also follows the channel-independence principle from prior time-series work, under which multivariate series are decomposed into univariate series for better generalization. In the reported experiments, however, the primary target is adjusted closing price, although open, high, low, close, and volume are present in the raw stock dataset description.

The underlying motivation is not merely that text may correlate with price movement, but that sentiment-only compression discards too much information. Historical prices encode trend and volatility, while news and social posts can reflect company fundamentals, macro events, product launches, earnings, scandals, and public sentiment. FTS-Text-MoE therefore attempts to preserve richer semantic information by selecting and summarizing key daily textual evidence and aligning it with the corresponding day in the numerical series.

2. Multimodal input construction and “point embeddings”

On the numerical side, each financial value at time $t$ is embedded through a SwiGLU-based projection: $$\mathbf{h}_t^0 = \operatorname{SwiGLU}(x_t) = \operatorname{Swish}(W x_t) \otimes (V x_t),$$ where $W \in \mathbb{R}^{D \times 1}$ and $$\mathbf{X}_{1:M} = (x_1, x_2, \ldots, x_M) \in \mathbb{R}^M$$0 are learnable parameters, $$\mathbf{X}_{1:M} = (x_1, x_2, \ldots, x_M) \in \mathbb{R}^M$$1 is the hidden dimension, and $$\mathbf{X}_{1:M} = (x_1, x_2, \ldots, x_M) \in \mathbb{R}^M$$2 denotes elementwise multiplication. In the implementation appendix, $$\mathbf{X}_{1:M} = (x_1, x_2, \ldots, x_M) \in \mathbb{R}^M$$3, so the resulting hidden representation lives in a 384-dimensional space (Xu et al., 28 Jul 2025).

On the text side, the preprocessing pipeline is deliberately selective. Each day may contain many news articles and many tweets/comments, so the model first ranks daily texts and keeps only the top example(s). The main text states that, on a daily basis, the top $$\mathbf{X}_{1:M} = (x_1, x_2, \ldots, x_M) \in \mathbb{R}^M$$4 news articles and X comments are selected, while the introduction says that the system extracts “1–2 key factual summaries at each time point.” For news, records are grouped by date, a relevance score $$\mathbf{X}_{1:M} = (x_1, x_2, \ldots, x_M) \in \mathbb{R}^M$$5 is computed for each article $$\mathbf{X}_{1:M} = (x_1, x_2, \ldots, x_M) \in \mathbb{R}^M$$6 for company $$\mathbf{X}_{1:M} = (x_1, x_2, \ldots, x_M) \in \mathbb{R}^M$$7, and the top item

$$\mathbf{X}_{1:M} = (x_1, x_2, \ldots, x_M) \in \mathbb{R}^M$$8

is selected, truncated to $$\mathbf{X}_{1:M} = (x_1, x_2, \ldots, x_M) \in \mathbb{R}^M$$9, and summarized with a pretrained DistilBART-CNN summarization model. Tweets/X comments are also ranked by relevance score per date and the top item is selected, but tweets are not summarized because they are already short. After ranking and summarization, the text is tokenized with the MiniLM L3 v2 tokenizer and encoded into semantic embeddings using an SBERT encoder.

The paper uses “point embeddings” informally to denote timestamp-level multimodal alignment rather than a standalone new embedding equation. For each day $$\mathbf{T}_{1:M} = (t_1, t_2, \ldots, t_M) \in \mathbb{R}^M,$$0, a numerical embedding derived from the stock price and a semantic embedding derived from the selected daily news or tweet text are attached to the same timestamp. The main text then states that “Textual and numerical embeddings undergo mean pooling for aggregation before entering the subsequent MoE Transformer modules.” This makes the fusion mechanism a temporally aligned pooled aggregation rather than a prompt-style concatenation of raw text.

This design has two immediate implications. First, the model preserves chronological linkage between external text and numerical observations. Second, the temporal alignment is intentionally simple: the paper later notes that market reactions to news and tweets often lag or diffuse over multiple days, so same-day matching is a limitation rather than a solved issue.

3. Decoder architecture and sparse expert routing

The sequence model is a decoder-only Transformer inspired by Time-MoE and LLM design choices such as RMSNorm, Rotary Positional Embeddings, and bias removal outside QKV projections. At each layer, masked self-attention handles autoregressive forecasting, and the conventional feed-forward sublayer is replaced by a Mixture-of-Experts block (Xu et al., 28 Jul 2025).

The per-layer computation is given as

$$\mathbf{T}_{1:M} = (t_1, t_2, \ldots, t_M) \in \mathbb{R}^M,$$1

$$\mathbf{T}_{1:M} = (t_1, t_2, \ldots, t_M) \in \mathbb{R}^M,$$2

$$\mathbf{T}_{1:M} = (t_1, t_2, \ldots, t_M) \in \mathbb{R}^M,$$3

Here $$\mathbf{T}_{1:M} = (t_1, t_2, \ldots, t_M) \in \mathbb{R}^M,$$4 denotes masked self-attention, and $$\mathbf{T}_{1:M} = (t_1, t_2, \ldots, t_M) \in \mathbb{R}^M,$$5 is the sparse MoE replacement for the dense FFN.

The mixture layer contains multiple FFN-style experts and one shared expert. Its output is defined as

$$\mathbf{T}_{1:M} = (t_1, t_2, \ldots, t_M) \in \mathbb{R}^M,$$6

where experts $$\mathbf{T}_{1:M} = (t_1, t_2, \ldots, t_M) \in \mathbb{R}^M,$$7 are non-shared and expert $$\mathbf{T}_{1:M} = (t_1, t_2, \ldots, t_M) \in \mathbb{R}^M,$$8 is shared. The non-shared routing scores are

$$\mathbf{T}_{1:M} = (t_1, t_2, \ldots, t_M) \in \mathbb{R}^M,$$9

and sparse top-$N$0 activation is applied: $N$1 The shared expert gate is

$N$2

with trainable routing weights satisfying $N$3.

The implementation appendix specifies 12 decoder layers, 12 attention heads per layer, and 8 experts per layer. For each token, top-2 experts are selected, so $N$4. The hidden size is $N$5, the feed-forward dimension is $N$6, and each expert has dimension $N$7. Total parameter count is 113M, while only about 50M parameters are active at inference because only a subset of experts fire for each token. This is the paper’s central efficiency claim: total capacity is increased relative to a similarly budgeted dense model, but sparse routing limits activated computation.

The model also introduces an auxiliary load-balancing loss to mitigate expert collapse or uneven expert usage: $N$8 with

$N$9

Here $$\hat{\mathbf{X}}_{M+1:M+N} = f_\theta(\mathbf{X}_{1:M}, \mathbf{T}_{1:M}) \in \mathbb{R}^N.$$0 is the average routing probability assigned to expert $$\hat{\mathbf{X}}_{M+1:M+N} = f_\theta(\mathbf{X}_{1:M}, \mathbf{T}_{1:M}) \in \mathbb{R}^N.$$1, $$\hat{\mathbf{X}}_{M+1:M+N} = f_\theta(\mathbf{X}_{1:M}, \mathbf{T}_{1:M}) \in \mathbb{R}^N.$$2 is the actual fraction of selected tokens routed to expert $$\hat{\mathbf{X}}_{M+1:M+N} = f_\theta(\mathbf{X}_{1:M}, \mathbf{T}_{1:M}) \in \mathbb{R}^N.$$3, and $$\hat{\mathbf{X}}_{M+1:M+N} = f_\theta(\mathbf{X}_{1:M}, \mathbf{T}_{1:M}) \in \mathbb{R}^N.$$4 is the number of time steps/tokens considered. The stated purpose is to align expected routing mass and realized token assignments so that training is spread more evenly over experts.

4. Forecasting heads, objectives, and optimization

FTS-Text-MoE is a regression model rather than a classifier. The appendix’s input-output figure is explicit that the model takes current-day text and stock price as input and outputs the next-day predicted stock price; more generally, the formulation allows forecasting the next $$\hat{\mathbf{X}}_{M+1:M+N} = f_\theta(\mathbf{X}_{1:M}, \mathbf{T}_{1:M}) \in \mathbb{R}^N.$$5 days (Xu et al., 28 Jul 2025).

A distinctive architectural component is the multi-resolution prediction head. Instead of producing only a single fixed horizon, the decoder output feeds $$\hat{\mathbf{X}}_{M+1:M+N} = f_\theta(\mathbf{X}_{1:M}, \mathbf{T}_{1:M}) \in \mathbb{R}^N.$$6 projection heads, each a single-layer FFN tuned to a different horizon: $$\hat{\mathbf{X}}_{M+1:M+N} = f_\theta(\mathbf{X}_{1:M}, \mathbf{T}_{1:M}) \in \mathbb{R}^N.$$7 Each head predicts $$\hat{\mathbf{X}}_{M+1:M+N} = f_\theta(\mathbf{X}_{1:M}, \mathbf{T}_{1:M}) \in \mathbb{R}^N.$$8 future steps. During inference, a greedy scheduling algorithm concatenates these projections to produce arbitrary-length forecasts. The paper does not provide explicit scheduling pseudocode, but the conceptual mechanism is to mix short and long heads so that a requested prediction window can be tiled from the available horizons.

Training combines the forecasting loss across all four heads with the MoE balancing penalty. The auxiliary coefficient is $$\hat{\mathbf{X}}_{M+1:M+N} = f_\theta(\mathbf{X}_{1:M}, \mathbf{T}_{1:M}) \in \mathbb{R}^N.$$9, and the autoregressive error is the Huber loss: $M$0 The use of Huber loss is justified as improving robustness against outliers and stabilizing training when textual information introduces additional noise.

The training setup is comparatively detailed. The model is trained for 10,000 steps with batch size 64 and maximum sequence length 1,024. Sequence packing is used for efficiency on variable-length examples. Optimization uses AdamW with

$M$1

The learning-rate schedule is linear warm-up for the first 10,000 steps followed by cosine annealing. Training runs in FP16 on two NVIDIA A6000 48GB GPUs. The text also says that “Each iteration processes 64 time steps,” which is somewhat ambiguous relative to the maximum sequence length; this is one of several implementation details that remain underreported.

5. Data resources, evaluation, and empirical profile

The stock time-series dataset contains 55 stocks from 11 sectors, with daily Yahoo Finance data including open, high, low, close, and volume; the adjusted closing price is the primary target. The paper reports 191,512 sample points over up to roughly 20 years, depending on ticker. The sectors are Basic Materials, Financial Services, Consumer Defensive, Utilities, Energy, Technology, Consumer Cyclical, Real Estate, Healthcare, Communication Services, and Industrials, with 5 stocks each. The tweet/X dataset contains 637,395 tweets for the same stock universe, collected through the Twitter API from 2020 to 2022. The news dataset comes from FNSPID and was extended by the authors to January 19, 2025, yielding 216,308 high-quality news records containing URLs, headlines, and full article texts (Xu et al., 28 Jul 2025).

Evaluation uses both predictive-error metrics and investment metrics. For forecast accuracy, the paper uses

$M$2

For portfolio evaluation it reports cumulative return (“Overall”), return standard deviation, and annualized Sharpe ratio. Baselines include Chronos small/base/large, Moirai small/base, an equal-weight $M$3 portfolio, and the S&P 500 index.

The sector-level forecasting table indicates that adding text often improves over the time-series-only variant, but the improvement is not uniform across sectors or text sources. Basic Materials improves from Ours-ts MSE/MAE $M$4 to $M$5 with news and to $M$6 with X comments. Financial Services improves from $M$7 to $M$8 with news and $M$9 with X comments. Utilities changes from $N$0 to $N$1 with X comments, while Technology changes from $N$2 to $N$3 with X comments but worsens to $N$4 with news. The narrative says summarized X comments “result in consistently lower MSE and MAE compared to models using only news or other inputs,” but the table does not fully support “consistently” in every sector. The broader claim that text improves over the time-series-only variant is mostly supported.

The paper is explicit that FTS-Text-MoE does not uniformly beat Chronos and Moirai on MSE/MAE in every industry or scenario. The authors argue that Chronos and Moirai are more conservative and smoother, which can lower pointwise forecasting errors in volatile settings, whereas FTS-Text-MoE is more aggressive and responsive to short-term changes. Its strongest reported advantage is therefore in downstream investment performance rather than universal pointwise error minimization.

Portfolio	Overall	Std. Dev.	Sharpe
$N$5 portfolio	0.0570	$N$6	0.1725
S&P 500 Index	-0.0082	$N$7	-0.3002
Moirai positive-prediction portfolio	0.0079	$N$8	0.3320
Chronos positive-prediction portfolio	-0.0077	$N$9	0.1223
FTS-Text-MoE	0.1347	$t$0	1.0818

These results indicate substantially stronger cumulative return and Sharpe ratio for FTS-Text-MoE, albeit with higher volatility than Moirai or Chronos. The paper interprets this as evidence that text integration yields a more concentrated and aggressive strategy.

The explainability claim is practical rather than formal. Predictions can be associated with the selected daily news summary or tweet/comment aligned to each day, and MoE routing is explicit, so one can in principle inspect which experts were selected for which timesteps. However, the paper does not include explanation metrics, attention visualization, expert specialization analysis, or case studies tracing individual predictions back to specific summaries. A common overreading is therefore to treat explainability as experimentally established; the evidence is more limited and mostly input-level.

6. Position within the MoE literature and documented limitations

Within the broader MoE literature, FTS-Text-MoE is most naturally situated as an application-specific sparse Transformer rather than a systems paper. FastMoE, for example, addresses distributed MoE execution on PyTorch/GPU stacks, emphasizing scatter/gather, expert sharding across GPUs and nodes, batched per-expert execution, and heterogeneous gradient synchronization for Transformer-style expert layers (He et al., 2021). This suggests that large-scale deployment of architectures like FTS-Text-MoE would likely depend on the same systems concerns—routing, communication, batching, and expert-parallel execution—even though the FTS-Text-MoE paper does not provide explicit throughput or latency benchmarks.

It also differs from modality-based MoE designs such as MoE-TTS, where the “experts” are fixed text and speech pathways and routing is determined by token modality rather than a learned sparse top-$t$1 gate (Xue et al., 15 Aug 2025). FTS-Text-MoE uses learned routing over FFN-style experts inside a financial forecasting decoder, with one shared expert and top-2 non-shared experts per token. This places it closer to conventional sparse MoE Transformers than to modality-partitioned adaptation schemes.

A further comparison point is RA-MoE, which treats multilingual fine-tuning of text MoEs as a routing-alignment problem and argues that MoE adaptation should respect heterogeneous expert structure learned during pretraining (Deng et al., 27 May 2026). This suggests, as a plausible implication, that future extensions of FTS-Text-MoE could analyze whether its experts specialize by market regime, sector, or text source rather than functioning as undifferentiated sparse capacity. The present paper does not perform such an analysis.

The paper’s limitations are explicit and substantial. Temporal alignment is simplistic, because text and price are matched on the same day even though market reactions may lag. The relevance score $t$2 used for ranking daily texts is never fully specified, which weakens reproducibility and interpretability. Train/validation/test split reporting is unclear, even though forecasting evaluation should respect temporal ordering. The experiments mostly compare one text source at a time rather than offering a clean joint-modality ablation with both news and tweets simultaneously. The paper does not isolate the contribution of MoE versus a dense decoder, nor the independent contribution of the multi-resolution head versus a single-horizon head. It claims inference efficiency from sparse activation but does not provide explicit throughput or latency comparisons against dense baselines.

Taken together, these features define FTS-Text-MoE less as a new MoE algorithm than as a multimodal financial forecasting template. Its core innovation lies in combining day-aligned textual summaries with numerical price embeddings, processing them through a sparse decoder-only Transformer with top-$t$3 expert routing and load balancing, and emitting forecasts through multi-resolution heads that can be greedily composed into arbitrary horizons. The reported evidence indicates that text often improves forecasting relative to a time-series-only variant and that the resulting strategy delivers markedly better portfolio return and Sharpe ratio than the reported baselines, while leaving key methodological details and several causal attributions unresolved (Xu et al., 28 Jul 2025).