Token Interleaving: A Multipurpose Sequence Technique

• Token interleaving is a family of structured techniques that alternate or rearrange token units to preserve semantic meaning across various modalities.
• It is applied in domains ranging from semantic packet streaming and LLM token transmission to document understanding and distributed control, offering enhanced efficiency and alignment.
• Research demonstrates that token interleaving can lower computational complexity and improve performance metrics, such as ATS and LPIPS, in challenging network and multimodal environments.

Token interleaving denotes a family of operations in which tokens, token-like units, or token-indexed control objects are alternated, mixed, repeated, or redistributed across a sequence, packet stream, modality boundary, or scheduling process. In recent work, the term appears in substantially different technical settings: semantic packetization for token communications, retransmission-aware LLM token streaming, layout-text fusion for document understanding, latent visual-textual reasoning, speech-text co-training, and cross-lingual spoken language modeling. This suggests a broad methodological pattern rather than a single canonical procedure: interleaving is used to preserve semantics under loss, couple heterogeneous context sources, reduce modality gaps, or expose models to sequential relations that would otherwise be omitted (Lee et al., 24 Jun 2025, Li et al., 2024, Lu et al., 2024, Fan et al., 2 Jul 2026, Moumen et al., 1 Dec 2025).

1. Conceptual scope and major variants

A common misconception is that token interleaving is simply token shuffling. The literature instead uses the term for structured alternation under constraints: packet non-overlap in outage channels, autoregressive ordering in LLMs, alignment-preserving speech-text alternation, sentence-level language switching in speech token streams, or repeated inclusion of unacknowledged tokens in network packets. In one line of work, interleaving is an optimization target; in another, it is an input-format design; in a third, it is a transmission or middleware behavior; and in security research it can be an attack surface rather than a capability.

Setting	Interleaved units	Stated role
Semantic packet aggregation (Lee et al., 24 Jun 2025)	Tokens grouped into packets	Maximize average token similarity under outage channels
LLM token streaming (Li et al., 2024)	Newly generated and unacknowledged tokens	Avoid stalls caused by missing packets
Document understanding (Lu et al., 2024)	Bounding box token and text tokens	Efficiently leverage autoregressive traits of LLMs
Speech-LLM ASR (Fan et al., 2 Jul 2026)	Aligned speech and text	Reduce the speech-text modality gap
Cross-lingual SLMs (Moumen et al., 1 Dec 2025)	Speech tokens across languages	Enable robust cross-lingual continuation

This diversity matters terminologically. In some papers, the token is a semantic unit produced by a tokenizer; in others, it is a control right in a MAC protocol or an abstract circulating object in distributed control. Accordingly, “interleaving” may refer to sequence construction, packet composition, scheduling, or command mixing.

2. Semantic communication and transport robustness

In token communication over outage channels, interleaving is driven by the fact that token semantics are context-dependent rather than independent bitwise payloads. The objective in SemPA-Look is to maximize average token similarity (ATS) between original and received token messages under packet erasures. Because token grouping is combinatorial, the framework introduces the residual semantic score (RSS),

$$\psi(\mathcal{C}_i, \mathcal{W}) = \phi(\mathcal{W} \setminus \mathcal{C}_i, \mathcal{W}),$$

which evaluates how much meaning remains if packet $\mathcal{C}_i$ is lost, and a lookahead score,

$\Psi(\tilde{\mathcal{C}_i, \mathcal{W}) = \frac{1}{k+1} \left( \psi(\tilde{\mathcal{C}_i, \mathcal{W}) + \sum_{j=1}^k \psi(\hat{\mathcal{C}_j^{(i)}, \mathcal{W}) \right),$

used to choose packet candidates while maintaining linear complexity, reported as $(N-1)\cdot P \cdot (k+1)$. Experiments on a remote AIGC task with MS-COCO show ATS and LPIPS scores comparable to exhaustive search, up to $40\times$ lower complexity than exhaustive search, and $10\times$ lower complexity than the genetic algorithm for similar performance; the paper also reports better robustness than random and non-semantic interleaving, particularly for packet erasure probability $p < 0.6$ (Lee et al., 24 Jun 2025).

A different transport-level form of interleaving appears in Eloquent for LLM token streaming. There, each outgoing packet contains newly generated tokens together with previously sent but unacknowledged tokens. This repeated inclusion lets a later successful packet repair earlier gaps without waiting for retransmission-triggered recovery. The sender maintains an unacked buffer, includes as many earliest unacknowledged tokens as fit, and removes acknowledged tokens upon ACK reception. Under simulation, Eloquent reduces stall ratio by 71.0% relative to the retransmission method commonly used by real chatbot applications and by 31.6% relative to a packet duplication baseline. The paper also gives the condition

$G \times (T - 1) \leq 2 \times RTT + L$

for the regime in which all unacknowledged tokens can be recovered in a single packet, where $G$ is the token generation gap, $T$ the maximum number of tokens per packet, $\mathcal{C}_i$0 the round-trip time, and $\mathcal{C}_i$1 the duration of packet loss (Li et al., 2024).

The multi-user wireless TokenCom literature uses a related but more implicit notion. The RL-based tokenizer agreement framework does not use the phrase “token interleaving” explicitly, but the summary associates it with joint optimization of tokenizer agreement, codebook selection, and sub-channel assignment, where a user’s token stream is mapped over multiple resource blocks according to the RL policy. The formulation couples tokenizer choice $\mathcal{C}_i$2, rate constraints such as $\mathcal{C}_i$3, DQN-based tokenizer and RB assignment, and DDPG-based beamforming, with simulation reporting a 68% reduction in video freezing events relative to a conventional H.265-based scheme (Zeinali et al., 12 Feb 2026).

3. Multimodal sequence construction and latent interleaving

In document understanding, interleaving can be an explicit sequence design that preserves autoregressive structure. LayTextLLM projects each bounding box $\mathcal{C}_i$4 to a single embedding

$\mathcal{C}_i$5

then interleaves bounding box and text as

$\mathcal{C}_i$6

This design uses a Spatial Layout Projector and Partial Low-Rank Adaptation so that the LLM can process bounding box tokens and text tokens in one flat sequence. Training is posed as Layout-aware Next Token Prediction, with loss computed only on text tokens. The reported efficiency advantage is substantial: on DocVQA, average input length is 4085.7 tokens for coordinate-as-tokens, 827.5 for DocLLM, and 664.3 for LayTextLLM. Benchmark evaluations report a 15.2% increase on KIE tasks and 10.7% on VQA tasks compared to previous SOTA OCR-based LLMs (Lu et al., 2024).

In DMLR, interleaving is moved from the explicit token stream into latent space. The framework introduces $\mathcal{C}_i$7 latent think tokens,

$\mathcal{C}_i$8

adds multiplicative Gaussian noise,

$\mathcal{C}_i$9

defines a confidence-based reward from top-$\Psi(\tilde{\mathcal{C}_i, \mathcal{W}) = \frac{1}{k+1} \left( \psi(\tilde{\mathcal{C}_i, \mathcal{W}) + \sum_{j=1}^k \psi(\hat{\mathcal{C}_j^{(i)}, \mathcal{W}) \right),$0 entropy,

$\Psi(\tilde{\mathcal{C}_i, \mathcal{W}) = \frac{1}{k+1} \left( \psi(\tilde{\mathcal{C}_i, \mathcal{W}) + \sum_{j=1}^k \psi(\hat{\mathcal{C}_j^{(i)}, \mathcal{W}) \right),$1

and updates latent tokens by a REINFORCE-style gradient step. Dynamic visual injection retrieves candidate patches at each latent think token, retains only those that improve reward, and injects them to realize dynamic visual-textual interleaving. Across seven multimodal reasoning benchmarks and various model architectures, DMLR is reported to outperform baselines on over 95% of tasks, with average gains up to +4.5% on mathematical reasoning and +3.45% on visual reasoning for reasoning-centric models, while maintaining high inference efficiency (Liu et al., 14 Dec 2025).

Taken together, these two lines separate explicit and latent interleaving. LayTextLLM interleaves observable sequence elements to exploit next-token prediction directly, whereas DMLR performs confidence-guided visual-textual interleaving at test time inside latent reasoning states.

4. Speech-text and cross-lingual spoken interleaving

For ASR-oriented Speech-LLM training, JSTIP constructs aligned interleaved speech-text sequences at word level and segment level. At word level, speech and text pieces alternate without additional special tokens between word-aligned units; at segment level, switching is marked by $\Psi(\tilde{\mathcal{C}_i, \mathcal{W}) = \frac{1}{k+1} \left( \psi(\tilde{\mathcal{C}_i, \mathcal{W}) + \sum_{j=1}^k \psi(\hat{\mathcal{C}_j^{(i)}, \mathcal{W}) \right),$2 and bounded by $\Psi(\tilde{\mathcal{C}_i, \mathcal{W}) = \frac{1}{k+1} \left( \psi(\tilde{\mathcal{C}_i, \mathcal{W}) + \sum_{j=1}^k \psi(\hat{\mathcal{C}_j^{(i)}, \mathcal{W}) \right),$3 and $\Psi(\tilde{\mathcal{C}_i, \mathcal{W}) = \frac{1}{k+1} \left( \psi(\tilde{\mathcal{C}_i, \mathcal{W}) + \sum_{j=1}^k \psi(\hat{\mathcal{C}_j^{(i)}, \mathcal{W}) \right),$4. The speech encoder outputs are projected into the LLM embedding space, and loss is computed only on text positions:

$\Psi(\tilde{\mathcal{C}_i, \mathcal{W}) = \frac{1}{k+1} \left( \psi(\tilde{\mathcal{C}_i, \mathcal{W}) + \sum_{j=1}^k \psi(\hat{\mathcal{C}_j^{(i)}, \mathcal{W}) \right),$5

An efficient implementation concatenates all speech pieces before the encoder and re-splits them after adapter projection. Experiments on 38k hours of ASR data report consistent entity accuracy improvement relative to ASR-only and joint speech-text baselines. The summary gives Med-AVG EER improvements from 7.97% to 7.32% for ASR-only versus +Interleave, and from 7.49% to 6.87% when domain text is added; zero-shot speech QA rises from 0.05% to 41.92%, and S2T MMLU accuracy rises from 35.68% to 51.77%. Mixed word + segment interleaving gives the best EER and modality-gap performance, suggesting that fine-grained interleaving helps entities while coarser interleaving improves general modality alignment (Fan et al., 2 Jul 2026).

Cross-lingual interleaving for Spoken LLMs removes text supervision entirely. Sentence-aligned EN-FR speech token sequences are concatenated as

$\Psi(\tilde{\mathcal{C}_i, \mathcal{W}) = \frac{1}{k+1} \left( \psi(\tilde{\mathcal{C}_i, \mathcal{W}) + \sum_{j=1}^k \psi(\hat{\mathcal{C}_j^{(i)}, \mathcal{W}) \right),$6

and the model is trained with a standard autoregressive objective over speech tokens only. Training uses a mixture of monolingual and interleaved sequences, with a reported 50/50 ratio in stage 2. The work releases an EN-FR TinyStories training set of approximately 42k hours and spoken StoryCloze and TopicCloze benchmarks. Under matched token budgets for 360M and 1B SLMs, interleaving improves monolingual semantic accuracy, enables robust cross-lingual continuation, and strengthens cross-lingual hidden-state alignment. The summary reports cross-lingual StoryCloze moving from about random performance without interleaving to 56.44% for EN→FR and 55.37% for FR→EN with interleaving, and hidden-state cosine similarity for aligned EN-FR pairs increasing from 0.73 to 0.75 and then 0.76 after fine-tuning in the 1B model (Moumen et al., 1 Dec 2025).

These speech results sharpen an important distinction. In JSTIP, interleaving is cross-modal and alignment-driven; in cross-lingual SLMs, it is multilingual and textless. Both are presented as mechanisms for reducing representational gaps that ordinary batch mixing does not close.

5. Sequence information preservation through overlap

The sequence modeling literature provides a precursor-like formulation in which the problem is not packet loss or multimodality but discretization. The TOI paper defines Token Order Imbalance as unequal exposure of token pair orderings when a long sequence is split into fixed-length data points. Boundary pairs can disappear entirely from training windows, producing partial sequence information loss. The proposed remedy, Alleviated TOI, constructs several overlapping versions of the sequence using different starting offsets. With data point length $\Psi(\tilde{\mathcal{C}_i, \mathcal{W}) = \frac{1}{k+1} \left( \psi(\tilde{\mathcal{C}_i, \mathcal{W}) + \sum_{j=1}^k \psi(\hat{\mathcal{C}_j^{(i)}, \mathcal{W}) \right),$7, number of overlapped sequences $\Psi(\tilde{\mathcal{C}_i, \mathcal{W}) = \frac{1}{k+1} \left( \psi(\tilde{\mathcal{C}_i, \mathcal{W}) + \sum_{j=1}^k \psi(\hat{\mathcal{C}_j^{(i)}, \mathcal{W}) \right),$8, and $\Psi(\tilde{\mathcal{C}_i, \mathcal{W}) = \frac{1}{k+1} \left( \psi(\tilde{\mathcal{C}_i, \mathcal{W}) + \sum_{j=1}^k \psi(\hat{\mathcal{C}_j^{(i)}, \mathcal{W}) \right),$9, the overlap scheme is generated for offsets $(N-1)\cdot P \cdot (k+1)$0, and the paper defines the imbalance ratio as

$(N-1)\cdot P \cdot (k+1)$1

As $(N-1)\cdot P \cdot (k+1)$2 increases, representation of split and non-split pairs becomes more uniform. For recurrent networks, prime batch sizes are recommended so that $(N-1)\cdot P \cdot (k+1)$3, reducing redundancy when batching overlapped data points (Kocher et al., 2019).

The empirical results show that this form of overlap can materially affect both text and speech tasks. On Penn Treebank test perplexity, Standard TOI is reported at 58.94, Alleviated TOI with $(N-1)\cdot P \cdot (k+1)$4 at 57.97, and Alleviated TOI with $(N-1)\cdot P \cdot (k+1)$5 at 56.46; with a non-prime batch size $(N-1)\cdot P \cdot (k+1)$6, the same $(N-1)\cdot P \cdot (k+1)$7 setting worsens to 65.88. AWD-LSTM-MoS plus Alleviated TOI reaches 54.58 versus 55.97 for the base SOTA. In speech emotion recognition, weighted accuracy is reported as 0.486 for Standard TOI and 0.553 for Alleviated TOI under the same steps, with further gains as $(N-1)\cdot P \cdot (k+1)$8 increases (Kocher et al., 2019).

This suggests a broader interpretation of interleaving in sequence learning: alternating offsets across data points can function as a coverage mechanism for local order relations that fixed segmentation would omit.

6. Control, scheduling, and adverse interleavings

Outside LLM and multimodal modeling, “token” often denotes a control object rather than a linguistic symbol. In WiLD point-to-multipoint links, the token-based MAC protocol uses explicit signalling messages from an Access Point to grant exclusive access to the medium and assign credits. The AP manages token circulation in round-robin fashion, and the schedule can be simple round-robin or adaptive/weighted. The paper explicitly describes token interleaving as centrally programmable through token grants, credit assignment, and scheduling of transmission opportunities, with simulations reporting superior fairness and efficiency under long range and faulty links relative to IEEE 802.11 DCF (Leocadio et al., 2019).

In self-stabilizing distributed control on rings, circulating tokens are separated rather than semantically interleaved. The protocol enforces minimum distance $(N-1)\cdot P \cdot (k+1)$9 under the feasibility constraint

$40\times$0

and a second protocol computes maximal separation

$40\times$1

when $40\times$2 is known and $40\times$3 is unknown. Uniform and non-uniform variants are given, including a single corrective process that estimates ring size from token round-trip timing. Here the technical focus is token spacing and stabilization, not content ordering, but the term illustrates the older distributed-systems meaning of a token as a privilege-bearing abstraction (0908.1797).

A sharply different usage appears in smartcard interoperability, where command interleaving is a threat. In that setting, APDU sequences intended for one smartcard can be routed to another and become interleaved with the certified command flow. The paper distinguishes “globally legal” commands, which occur in the correct certified context, from “locally legal” commands, which are accepted syntactically by the card even when they are not part of the certified process. Experiments on Common Criteria certified digital signature processes show that some interleaved external commands are accepted without error or warning; an injected “MSE Erase” can even delete the Security Environment Object and render digital signatures impossible, requiring physical replacement of the smartcard (Talamo et al., 2012).

Across these control-oriented works, interleaving is not intrinsically beneficial. It can be a scheduling primitive, a stabilization constraint, or a failure mode. That contrast is essential for interpreting the term correctly in the wider literature.