MTP-S2UT Loss in S2UT Models

• MTP-S2UT Loss is a training objective that applies multi-token prediction at an intermediate CTC-supervised decoder layer to enrich semantic content early in the model.
• This approach significantly improves translation quality, as evidenced by notable ASR-BLEU score gains across different tokenizers and decoding schemes.
• The method demonstrates potential for generalizing multi-token supervision to enhance semantic fusion in various cross-modal sequence generation tasks.

MTP-S2UT loss is a training objective for speech-to-unit translation (S2UT) models that addresses the sparse semantic density of individual speech tokens. Standard S2UT models convert target speech into discrete tokens but rely on next-token prediction (NTP) objectives, which supervise the decoder to predict only the next token per step. In contrast, multi-token prediction (MTP) losses require the prediction of multiple subsequent tokens at each position. MTP-S2UT loss innovates by applying the multi-token prediction loss not just at the final decoder layer but specifically at an intermediate decoder layer concurrently supervised by a Connectionist Temporal Classification (CTC) loss. This design infuses the hidden representation with enhanced semantic content early in the model pipeline, ultimately reducing uncertainty and improving translation quality.

1. Rationale and Motivation

Speech tokens derived from quantized target speech lack the semantic density seen in text representations; representing a complete semantic unit typically requires a sequence of such tokens. This sparsity limits the capacity of models trained with only NTP losses to encode rich contextual information per position. MTP loss addresses this by supervising the model to predict multiple future tokens concurrently, thus increasing per-step information content.

The central motivation behind MTP-S2UT loss is to foster earlier and more effective integration of contextual and cross-modal (speech and text) signals. Enforcing multi-token prediction at an intermediate stage encourages the model to "pull" critical future content forward in the hidden state sequence, reducing ambiguity in token prediction and facilitating earlier semantic fusion. This mechanism is hypothesized—and demonstrated—to result in lower prediction entropy and improved translation performance.

2. Architectural Integration and Loss Formulation

In conventional S2UT architectures, the decoder receives a right-shifted version of the token sequence and outputs a prediction supervised by the NTP objective at the final layer. Initial MTP variants extended this setup, applying the MTP loss at the final output, but this delays semantic information enrichment.

MTP-S2UT diverges by applying the MTP loss at an intermediate decoder hidden state ($H_{\text{dec}}^m$) where CTC loss is also computed. This intermediate representation is already supervised to align well with both text and speech modalities via CTC, making it a natural locus for concurrent multi-token supervision.

The losses are formally defined as: $$\mathcal{L}_{\text{MTP-S2UT}} = -\sum_{k=0}^{N-1} \log P(U^{(-k)} | H_{\text{dec}}^m)$$ where $U^{(-k)}$ is the target sequence shifted by $k$ positions. The CTC loss at this layer is simultaneously optimized: $$\mathcal{L}_{\text{CTC}} = -\log P(Y | H_{\text{dec}}^m)$$ By combining these objectives at $H_{\text{dec}}^m$, the model is encouraged to advance semantic information, as evidenced by quantitative metrics showing earlier availability of key token content.

3. Implementation Specifics

Within the S2UT framework, the workflow involves the following stages:

• The encoder produces a representation $H_\text{enc}$ from the input speech.
• The decoder processes this and at the intermediate layer $H_{\text{dec}}^m$, losses are applied as follows:
  • CTC Loss is computed at $H_{\text{dec}}^m$ for alignment to the target transcription.
  • MTP Loss is also computed at $H_{\text{dec}}^m$, with a lightweight prediction head (either a linear head or a shallow decoder block) used for multi-token output.

For each $k = 0, \ldots, N-1$: $$H^{k}_{\text{out}} = \text{Decoder}^k(H_{\text{enc}}, H_{\text{dec}}^m)$$

$$P(U^{(-k)} | H_{\text{dec}}^m) = \text{softmax}(W_{\text{out}} \cdot H^{k}_{\text{out}})$$

The final loss is the negative log-likelihood across all $N$ future token predictions. This simultaneous supervision at $H_{\text{dec}}^m$ leads to a "forward shift" in semantic alignment, as observed in detailed analyses of the CTC-aligned token positions.

4. Comparative Experimental Analysis

Empirical validation is conducted on the CVSS-C dataset for French→English and Spanish→English speech-to-speech translation tasks, with various speech tokenizers and decoding strategies. Key findings include:

• The baseline S2UT model using the $\mathcal{S}^3$ tokenizer with greedy search yields an ASR-BLEU score of 17.79; incorporating the MTP-S2UT loss raises this to 24.36.
• Consistent improvements are recorded across beam search and different tokenization methods (HuBERT with K-means, GLM-4-Voice-Tokenizer).
• MTP-S2UT outperforms other MTP variants—such as MTP-Parallel-Linear, MTP-DeepSeek-V3, and MTP-VocalNet—on all tested configurations.
• Analysis of CTC alignments indicates a forward shift, with average first-occurrence positions of text tokens dropping well below 50% of the sequence length, signifying earlier semantic availability.
• Entropy analysis over 1.2 million token predictions demonstrates that MTP-S2UT consistently reduces predictive uncertainty relative to the NTP baseline.

Model Variant	Tokenizer	Decoding	ASR-BLEU Baseline	ASR-BLEU w/ MTP-S2UT
S2UT	$\mathcal{S}^3$	Greedy	17.79	24.36
S2UT	HuBERT + K-means	Greedy	—	↑ (consistent gain)
S2UT	GLM-4-Voice	Beam	—	↑ (consistent gain)

Consistent performance advantages for MTP-S2UT are observed across languages, tokenizers, and decoding methods.

5. Functional Consequences and Semantic Analysis

The early application of MTP loss at the CTC-supervised layer results in several measurable functional consequences:

• Hidden representations show increased semantic integration at earlier positions, facilitating a shift in the relative location of informative textual content.
• Token prediction entropy is reduced across a large prediction set, reflecting heightened certainty and context integration.
• These effects are robust across a variety of model configurations and do not appear confined to any particular tokenization or language pair, underscoring the generality of the approach.

This suggests that the approach may generalize as a technique for reordering and densifying semantic information in other cross-modal sequence generation tasks.

6. Broader Impact and Future Directions

MTP-S2UT loss demonstrates that introducing multi-token supervision at an intermediate, CTC-supervised decoder state in S2UT models reliably delivers earlier and denser semantic fusion. Observed benefits include:

• More robust and certain token predictions.
• Early manifestation of informative content in hidden states, signaled by forward-shifted CTC alignments.
• Consistency of improvements across languages and tokenizers.

A plausible implication is that early intervention in the hidden representation, by means of multi-token prediction objectives, can serve as a blueprint for improving multi-modal and sequence-to-sequence models beyond speech-to-speech translation. Future work may explore alternative strategies and modalities for early semantic enrichment, with research directions including adaptation to non-speech input/output regimes and refinement of prediction head architectures.

The demonstrated success of MTP-S2UT loss advocates for deeper exploration of early fusion and multi-token objectives in a range of next-generation end-to-end communication and translation systems.