---
title: Token-Level Serialized Output Training (t-SOT)
url: https://www.emergentmind.com/topics/token-level-serialized-output-training-t-sot
type: topic
---

# Token-Level Serialized Output Training (t-SOT)

Token-Level Serialized Output Training (t-SOT) is a paradigm for sequence modeling in multi-talker, multilingual, and multi-task speech processing. t-SOT redefines training targets and inference outputs by serializing multiple streams of conceptually distinct content (speaker turns, language pairs, or modality outputs) into a single, fully interleaved token sequence, using special tokens to encode stream or role transitions. This architectural simplification enables streaming, low-latency inference with a single decoder, effectively handling overlapping speech, speaker-attribution, and joint tasks such as ASR and ST—without the computational burden or architectural complexity of multiple output branches or explicit permutation solutions [2202.00842, 2309.08131, 2403.02010].

## 1. Serialization Principle and Label Construction

In t-SOT, each unit (token, word, or subword) from multiple sources—such as speaker streams in ASR or source/target outputs in joint ASR-ST—is assigned an emission time (from forced alignment, timestamp inference, or aligner). The reference label sequence is constructed by merging all tokens across sources in increasing time order. Whenever the current token originates from a different source than the previous, a stream-identity or change-channel token (e.g., ⟨cc⟩, ⟨sc⟩, ⟨asr⟩, ⟨st⟩) is inserted. The resulting sequence takes the form:

\[
Y = [w_1,\dotsc,\langle\mathrm{cc}\rangle,w_{i},\dotsc]
\]

where each ⟨cc⟩ denotes a source or speaker change [2202.00842, 2305.13716, 2403.02010]. This principle generalizes to joint tasks by inserting tokens marking modality switches (e.g., ⟨asr⟩, ⟨st⟩ for joint recognition and translation [2307.03354, 2310.14806]), or role tokens for speaker-attribution [2506.10349].

Token-level interleaving, as opposed to utterance-level SOT or PIT-based streams, ensures strict chronological ordering and stream-switch information at token granularity, which is critical for handling rapid overlaps, cross-lingual switching, and hybrid content.

## 2. Model Architecture and Training Objectives

The dominant t-SOT instantiation is the streaming Transformer Transducer (TT) or RNN-Transducer (RNN-T), with the following components:

- **Encoder:** Causal transformer or conformer stack operating on log-Mel features (with chunk-wise lookahead for streaming).
- **Prediction network:** LSTM stack that consumes previously emitted non-blank (and non-⟨cc⟩) tokens.
- **Joint network:** Fuses encoder and predictor states and outputs logits over the expanded vocabulary with the special change tokens [2202.00842, 2203.16685].

Training proceeds with the standard RNN-T loss over the serialized target $\mathcal{L} = -\log P(Y|X)$, where $Y$ is the t-SOT sequence. The vocabulary is augmented by special change tokens, which are treated as standard prediction targets.

For multi-task scenarios (e.g., joint ASR/ST), the label construction involves interleaving the ASR and ST tokens using explicit textual aligners (e.g., awesome-align), with separate tokens for output mode transitions. The training loss remains a single sequence-level RNN-T or CTC loss [2307.03354, 2310.14806].

Recent variants further decouple the language model (LM) component from the predictor using a factorized neural transducer (FNT), allowing text-only domain adaptation by isolating the multi-stream disruption introduced by change tokens [2309.08131].

## 3. Decoding and Inference Strategies

Streaming inference with t-SOT produces a single token stream. As each token is emitted, inference logic monitors change-channel tokens, partitioning the token stream into separate “virtual output channels” corresponding to speakers, roles, or modalities:

- **Speaker switching:** On ⟨cc⟩ emission, toggle the target buffer to a different speaker [2202.00842, 2203.16685].
- **Joint ASR-ST:** On ⟨asr⟩ or ⟨st⟩ emission, toggle the hypothesis buffer to transcript or translation [2307.03354].
- **Role tagging:** On ⟨spk0⟩/⟨spk1⟩ tokens, switch transcript attribution [2506.10349].

This enables real-time demultiplexing with compute and latency comparable to single-talker ASR, even under heavy overlap or rapid context switching. For more structured architectures (e.g., FNT), the LM maintains multiple hidden states—one per channel—switching context only on change token emission [2309.08131].

## 4. Extensions: Auxiliary Supervision and Multi-Task Losses

To address frequent t-SOT challenges—including degraded context modeling at rapid stream switches and unreliable speaker-change prediction—recent work introduces auxiliary objectives and architectural blocks:

- **Speaker-aware auxiliary losses:** Masked t-SOT loss reinforces intra-speaker context learning by feeding the model a per-speaker masked target, where only tokens from one speaker are visible (others are masked), thereby reducing semantic confusion in autoregressive decoding [2403.02010].
- **Speaker similarity attention:** Decoder self-attention is regularized using a similarity matrix over learned speaker embeddings, suppressing cross-speaker attention and prioritizing intra-speaker dependencies [2403.02010].
- **Boundary-aware losses:** Penalties on decoder cross-attention, guided by oracle or estimated timestamps, constrain the attention to remain block-diagonal at segment boundaries [2305.13716].
- **Speaker change detection heads:** Auxiliary prediction heads jointly estimate speaker-change or utterance boundaries, with associated CE or binary cross-entropy losses [2305.13716].

A generalized multi-task loss aggregates main RNN-T or attention loss with CTC (sometimes two-stage, for token-level monotonicity followed by FIFO re-ordering), auxiliary speaker- or boundary losses, and structural penalties [2305.13716].

## 5. Application Domains and Evaluation

t-SOT underpins major advances in a variety of tasks:

- **Multi-talker streaming ASR:** State-of-the-art WER on LibriSpeechMix, LibriCSS, AMI, and AliMeeting, with single-pass models matching or improving on two-stream or PIT-based systems [2202.00842, 2209.04974, 2305.13716].
- **Speaker-attributed ASR:** Joint ASR and speaker identification/diarization with low latency using token-aligned speaker embeddings or role tokens [2203.16685, 2506.10349].
- **Joint ASR/ST and Multi-Task Streaming:** Token-level interleaving unifies translation and transcription—enabling simultaneous recognition with single decoder and minimal latency, and outperforming separate models in quality–latency tradeoff [2307.03354, 2310.14806].
- **Domain adaptation:** Text-only adaptation becomes feasible via FNT, yielding 8–11% relative WER gains after cross-domain LM fine-tuning [2309.08131].

Metrics such as concatenated minimum-permutation WER (cpWER), utterance-dependent CER (UD-CER), and attribution error rate (AER) are used to capture not only lexical but also speaker change and segmentation accuracy [2305.13716, 2403.02010, 2506.10349].

| Task                         | Typical Change Token      | Key Metric       |
|------------------------------|--------------------------|------------------|
| Multi-talker ASR             | ⟨cc⟩, ⟨sc⟩               | WER, UD-CER      |
| Joint ASR/ST                 | ⟨asr⟩, ⟨st⟩              | WER, BLEU, LAAL  |
| Speaker Attribution/Role     | ⟨spk0⟩, ⟨spk1⟩           | mtWER, AER       |

## 6. Limitations, Ablations, and Future Directions

Observed limitations include the need for accurate, timestamped supervised data to construct the interleaved label sequences and for explicit utterance or speaker boundary annotation [2506.10349]. Semantic confusion during frequent interleaving can degrade recognition accuracy, motivating auxiliary masking and attention methods [2403.02010]. For higher numbers of concurrent speakers, scalability is constrained by fixed channel token assignment, and change-point modeling becomes more complex [2203.16685, 2305.13716].

Proposed directions include:

- End-to-end segmentation and change-detection within the ASR/ST model, moving beyond forced alignment [2506.10349, 2305.13716].
- Hierarchical and multimodal t-SOT (e.g., combining audio, video, gaze for speaker-role tagging) [2506.10349].
- Continuous pre-training and large-scale adaptation with text-only corpora or robust two-channel simulated mixtures [2309.08131, 2209.04974].
- Prompt-based serialized output as semantic control for LLM-based ASR and multi-modal tasks [2509.04488].

## 7. Impact and Summary of Empirical Results

t-SOT models have established new standards in streaming conversational ASR, with empirical results consistently demonstrating lower WER and latency than multi-branch and offline separation-based baselines: e.g., LibriSpeechMix WER of 4.4% (2-speaker, TT-36, 2.56 s), AliMeeting UD-CER reduction of 19.9% with BA-SOT, and cpWER of 3.41% on LibrispeechMix with SA-SOT after extensive training [2202.00842, 2305.13716, 2403.02010]. t-SOT unifies single and multi-talker recognition, allows text-only domain adaptation, and supports joint ASR/ST/role-aware generation with no increase in inference cost.

The serialized output training paradigm, and its token-level variant t-SOT, have demonstrated both theoretical elegance and practical efficacy in the deployment of unified, flexible, and scalable sequence models for complex streaming speech tasks [2202.00842, 2305.13716, 2309.08131, 2403.02010, 2506.10349, 2509.04488].

Source: https://www.emergentmind.com/topics/token-level-serialized-output-training-t-sot