Decoder Switching in Adaptive Systems

• Decoder switching is an adaptive decoding architecture that conditionally delegates processing to specialized pathways based on language structure, stream state, or task context.
• It is applied across various domains such as code-mixed NLP with dual decoders, ASR with MoE routing, consistent video decoding, long-term BCI control, and quantum error correction.
• Key design principles include conditional coordination, state consistency, and multi-objective balancing to optimize performance, accuracy, and computational efficiency.

Searching arXiv for papers on "decoder switching" and related uses across NLP, ASR, video streaming, BCI, and quantum error correction. Decoder switching denotes a family of architectures and procedures in which decoding is not treated as a single monolithic process, but is instead partitioned, routed, reconfigured, or conditionally delegated across multiple decoder pathways, experts, modes, or target representations. The term appears in distinct technical senses across recent literature: in code-mixed language modeling it refers to synchronized language-specific decoders coupled by inter-decoder cross-attention (Baral et al., 19 May 2025); in code-switching ASR it denotes Switch Transformer-style Mixture-of-Experts routing inside the decoder to exploit text-side language information (Ye et al., 2024); in interactive video streaming it is the problem of ensuring identical reconstruction when a decoder switches between pre-encoded streams (Dai et al., 2015); in long-term BCI it appears as latent-state switching among decoder submodels in a Markov-switching mixture-of-experts framework (Moly et al., 2022); in online ASR on GPU it refers to dynamic context switching that changes active biasing information per segment without rebuilding the decoding graph (Nigmatulina et al., 2023). A related but later formulation explicitly names “decoder switching” as a weak/strong decoder orchestration framework for real-time quantum error correction (Toshio et al., 29 Oct 2025). Across these settings, the common structural theme is conditional specialization: decoding behavior adapts to language, stream state, confidence, task mode, or external context rather than remaining fixed.

1. Dual-decoder synchronization in code-mixed language modeling

In code-mixed language modeling, decoder switching is instantiated by CMLFormer, which is built around the claim that code-mixed text should not be modeled with a single monolithic decoder (Baral et al., 19 May 2025). The model uses a shared encoder together with two synchronized decoders, one for the base language and one for the mixing language. In the paper’s setup, the encoder consumes the code-mixed sentence $C=\{c_1,\dots,c_{N_c}\}$, while the base-language decoder generates the aligned translation $B$ and the mixing-language decoder generates $M$. This separation is intended to reflect the structure of code-mixed sentences, where the constituent languages have different syntactic and semantic roles but remain tightly coupled.

The central architectural mechanism is dual-decoder inter-decoder cross-attention. Unlike the vanilla Transformer, CMLFormer adds a third attention layer in each decoder block so that each decoder can attend to the other decoder’s hidden states. If the decoder hidden states after encoder-decoder attention are denoted $a^b$ and $a^m$, then decoder-decoder cross-attention is

$$o^b = Attention(a^b, a^m, a^m) \qquad o^m = Attention(a^m, a^b, a^b). \tag{1}$$

These cross-attended outputs are then passed to the usual feed-forward and layer-normalization sublayers. The model description characterizes the decoders as “fully synchronized,” because each decoder depends on the other decoder’s current hidden states in every block (Baral et al., 19 May 2025).

The paper also introduces cross-decoder linear projections before cross-attention. These projections map one decoder’s hidden states into the latent space of the other decoder, allowing the base-language branch to attend to mixing-language representations and vice versa despite structural asymmetries between the two languages. Decoder switching in this setting is therefore not a hard gate between branches. Rather, it is a synchronized latent interaction between language-specific decoders trained on aligned targets (Baral et al., 19 May 2025).

This architecture is tied directly to the BiLTM objective, which uses the dual-decoder setup to generate base-language and mixing-language translations from a code-mixed input sentence $C$. The paper describes this as a multi-target autoregressive objective, with total BiLTM loss defined as the average of the two decoder losses $L_{BiLTM^b}$ and $L_{BiLTM^m}$. A plausible implication is that decoder switching here is best understood as coordinated specialization under alignment constraints rather than decoder replacement.

2. Switching-point learning and decoder interaction in CMLFormer

The second core component of CMLFormer is switching point learning, which formalizes where the language changes between consecutive tokens (Baral et al., 19 May 2025). The paper defines

$$C = \{c_1, C_2, ... , C_{N_c}\},\quad B = \{b_1,b_2, ... , b_{N_B}\},\quad M = \{m_1, m_2, ... , m_{N_M}\}$$

and

$B$0

where $B$1 is a 0–1 bit indicating a language transition before token $B$2, with $B$3 if $B$4, and $B$5 otherwise. The sequence begins with $B$6 because the first token cannot have a preceding transition.

Switching Point Prediction is an auxiliary encoder-side task in which, for each token position $B$7, the model predicts the probability $B$8 that a switch occurs at that position and is trained with binary cross-entropy against the ground-truth label sequence $B$9 (Baral et al., 19 May 2025). The supervision is aligned to subword tokenization by assigning each switching label only to the first subword of the corresponding word and masking all other subwords with $M$0, so that the loss ignores them. The intent is to preserve the original word-level switch structure.

CMLFormer combines standard Masked Language Modeling on all three sequences $M$1, $M$2, and $M$3 with several code-mixing-specific objectives: Bilingual Translated Sentence Prediction, Bilingual Language Translation Modeling, Token Language Classification, Switching Point Prediction, and Code-Mix Index prediction. The overall loss is given as

$M$4

The hyperparameter table provides the corresponding weights $M$5, $M$6, $M$7, $M$8, $M$9, and $a^b$0, although the OCR is described as somewhat inconsistent (Baral et al., 19 May 2025).

The global code-mixing signal is provided by CMI regression, where

$a^b$1

Here $a^b$2 is the number of tokens, $a^b$3 the number of switching points, $a^b$4 the number of tokens from the dominant/base language, and $a^b$5, $a^b$6 weighting factors. This sentence-level objective complements token-level switching supervision.

The empirical findings tie decoder interaction to switching-point learning. On HASOC-2021, adding BiLTM and SPP improves over a BERTbase model pre-trained on the same augmented data, with gains of 0.18 and 0.05 in F1, respectively (Baral et al., 19 May 2025). The paper interprets this as evidence that cross-lingual alignment and awareness of switching dynamics improve downstream classification. It also reports that adding BTSP, TLC, and CMI on top of those objectives can reduce performance, suggesting that too many objectives may create conflicting gradients or overburden the encoder. Attention analyses further show that CMLFormer assigns higher attention to tokens near language transitions, whereas BERT fails to focus on these points (Baral et al., 19 May 2025). The ablation on decoder cross-attention reports that the synchronous variant gives the smoothest and lowest loss trajectory, while removing cross-attention produces a noisier and less stable training curve and the asynchronous variant converges more slowly with higher loss.

3. MoE-based decoder switching in code-switching ASR

In SC-MoE, decoder switching has a different meaning: the introduction of Mixture-of-Experts or Switch Transformer-style routing inside the decoder so that the model can exploit text-side language information in Mandarin-English code-switching ASR (Ye et al., 2024). The paper argues that encoder-side routing alone is insufficient because “the decoder lacks the modeling ability for phonemic confusion when utilizing only MoE layer in the encoder.” The decoder-side mechanism is therefore intended to leverage language-specific information embedded in text, improve modeling of phonemic confusion across Mandarin and English, complement encoder-side acoustic routing, and improve unified streaming and non-streaming code-switching ASR performance.

The backbone is U2++, with a CTC encoder branch and an AED branch with L2R and R2L decoders (Ye et al., 2024). In the decoder, the standard transformer decoder is replaced by Switch Transformer decoder layers; specifically, each decoder is built as 1 standard transformer decoder layer plus 2 switch transformer decoder layers. The paper studies router variants R1, R2, and R3, and reports the best setup for the decoder as R3 extended to the full encoder-decoder system.

The decoder MoE experts are language-specialized text-side experts corresponding to Mandarin and English, making the decoder MoE a 2-class LID-oriented routing mechanism. This differs from the encoder’s streaming MoE, which has three experts—Mandarin, English, and blank (Ye et al., 2024). The decoder uses standard Switch Transformer top-1 routing: the router produces logits, softmax converts them to probabilities, and the highest-probability expert is selected. Although the paper prints the routing equations only for the streaming encoder MoE,

$a^b$7

$a^b$8

$a^b$9

$a^m$0

it states that the same Switch Transformer idea applies in the decoder, except routing is over 2 experts rather than 3 (Ye et al., 2024).

Training is joint. The total loss is

$a^m$1

with

$a^m$2

and

$a^m$3

For decoder routers specifically,

$a^m$4

Thus decoder-side routing is trained with language labels $a^m$5 using a cross-entropy objective rather than CTC (Ye et al., 2024).

The comparison between SC-MoE-Enc and SC-MoE-Enc-Dec quantifies the decoder contribution. SC-MoE-Enc-Dec yields modest MER reductions over SC-MoE-Enc in both streaming and non-streaming modes: streaming mixed MER improves from 9.50 to 9.42, and non-streaming mixed MER from 8.70 to 8.66 (Ye et al., 2024). The best model reports streaming 7.15 / 28.05 / 9.42 and non-streaming 6.50 / 26.36 / 8.66 for Man/Eng/Mixed, with an activated parameter count of 50.2M. The paper concludes that introducing MoE layers into the decoder potentially allows use of language-specific text information and mitigates some phonemic confusion issues (Ye et al., 2024).

4. Stream switching and identical reconstruction in video coding

In interactive video streaming, decoder switching refers to a different problem: the decoder may switch from one pre-encoded stream to another at a switching instant, but the side-information versions of the same future picture already in the decoder buffer may not be identical (Dai et al., 2015). If predictive coding continues from whichever reconstructed version happens to be available, drift occurs and later frames become inconsistent. The proposed solution is a single merge description—an M-frame—that allows the same target frame to be reconstructed regardless of which side-information frame is present at the decoder.

The method replaces DSC-style bit-plane and channel-coding merge logic with a piecewise constant merge operator applied directly to quantized transform coefficients (Dai et al., 2015). For block $a^m$6 and coefficient $a^m$7,

$a^m$8

and the merge function is

$a^m$9

Here $$o^b = Attention(a^b, a^m, a^m) \qquad o^m = Attention(a^m, a^b, a^b). \tag{1}$$0 is the step size and $$o^b = Attention(a^b, a^m, a^m) \qquad o^m = Attention(a^m, a^b, a^b). \tag{1}$$1 is a shift. The objective is to choose parameters so that all candidate side-information coefficients for a given block and frequency fall into the same constant interval of the piecewise constant function, thereby mapping to the same reconstructed value $$o^b = Attention(a^b, a^m, a^m) \qquad o^m = Attention(a^m, a^b, a^b). \tag{1}$$2.

The sufficient condition for identical merging over a block group $$o^b = Attention(a^b, a^m, a^m) \qquad o^m = Attention(a^m, a^b, a^b). \tag{1}$$3 is

$$o^b = Attention(a^b, a^m, a^m) \qquad o^m = Attention(a^m, a^b, a^b). \tag{1}$$4

where $$o^b = Attention(a^b, a^m, a^m) \qquad o^m = Attention(a^m, a^b, a^b). \tag{1}$$5 is the maximum pairwise side-information difference in that group (Dai et al., 2015). The feasible shift range is expressed using

$$o^b = Attention(a^b, a^m, a^m) \qquad o^m = Attention(a^m, a^b, a^b). \tag{1}$$6

with

$$o^b = Attention(a^b, a^m, a^m) \qquad o^m = Attention(a^m, a^b, a^b). \tag{1}$$7

and

$$o^b = Attention(a^b, a^m, a^m) \qquad o^m = Attention(a^m, a^b, a^b). \tag{1}$$8

Once $$o^b = Attention(a^b, a^m, a^m) \qquad o^m = Attention(a^m, a^b, a^b). \tag{1}$$9 and a valid $C$0 are transmitted, the decoder applies the same function to whichever side-information coefficient it has, and all candidate inputs yield the same output (Dai et al., 2015).

The paper distinguishes fixed target merging and optimized target merging. In the fixed target case, if $C$1 is the target coefficient and

$C$2

then choosing

$C$3

with $C$4, guarantees

$C$5

In the RD-optimized case, the reconstruction target itself is optimized: $C$6 At coefficient level the optimization is

$C$7

subject to

$C$8

Empirically, the merge-frame approach is reported to outperform both DSC-based D-frames and H.264 SP-frames while reducing decoder complexity because it avoids bit-plane decoding and channel decoding (Dai et al., 2015). In static view switching, fixed target merging achieved BD-rate reductions of about 31–40% relative to D-frame, with up to 40.1% improvement on Kendo. In bitrate adaptation, the RD-optimized M-frame achieved up to 65.6% BD-rate reduction over D-frame in the average case and up to 36.3% over SP-frame; in the worst case it still beat D-frame by up to 65.4% and SP-frame by up to 49.9%. For dynamic view switching, it achieved up to 57.5% BD-rate reduction over D-frame and 19.3% over SP-frame in the average case, and up to 58.7% and 36.4% in the worst case, respectively. In this literature, decoder switching is therefore a consistency problem at stream-transition points, solved by making decoder output independent of which side-information stream arrives.

5. State-switching decoders in BCI and runtime context switching in ASR

In long-term BCI, decoder switching is embodied by the Recursive Exponentially Weighted Markov-Switching multi-Linear Model, an adaptive online tensor-based decoder for asynchronous bimanual exoskeleton control (Moly et al., 2022). REW-MSLM is described as a fully adaptive mixture of experts built on a Markov-switching linear model, extended to tensor input and tensor output variables, and trained recursively using an adaptation inspired by REW-NPLS. The input feature space is partitioned into multiple regions, each modeled by a separate local multilinear function. Rather than using one global decoder, the model employs a piecewise-linear tensor model with state-dependent submodels.

The switching mechanism is defined through a latent state variable $C$9, with gating probability

$L_{BiLTM^b}$0

and continuous output

$L_{BiLTM^b}$1

The HMM transition matrix is

$L_{BiLTM^b}$2

In practice, the latent states correspond to task modes such as idle, left-hand translation, right-hand translation, left wrist rotation, and right wrist rotation (Moly et al., 2022). Decoder switching is thus simultaneously switching among multiple multilinear models, latent states, and task modes.

Adaptation is recursive and exponentially weighted. Each expert uses REW-NPLS with forgetting factor $L_{BiLTM^b}$3, where $L_{BiLTM^b}$4, while the HMM transition matrix is updated with forgetting factor $L_{BiLTM^b}$5, where $L_{BiLTM^b}$6 (Moly et al., 2022). The neural features are tensorized ECoG representations

$L_{BiLTM^b}$7

derived from 64-channel EpiCoG, 1-second windows with 100 ms step, and a complex continuous wavelet transform from 10 to 150 Hz.

The reported stability figures are central to the significance of the switching mechanism. Exoskeleton control remained functional for 167 days after the last model calibration and virtual avatar control for 203 days after the last model calibration (Moly et al., 2022). The decoder was not recalibrated for more than 5 months for the exoskeleton and more than 6.5 months for the avatar. The zero-slope hypothesis was not rejected for 16 of 18 indicators in the avatar experiment and 12 of 18 indicators in the exoskeleton experiment. Average accuracy was about 93 ± 3% for the avatar and about 92 ± 4% for the exoskeleton, with average F-score about 76 ± 9% and 75 ± 12%, respectively (Moly et al., 2022).

A distinct operational use of switching appears in GPU online ASR with contextual biasing (Nigmatulina et al., 2023). There, dynamic context switching means that the decoder can load and apply a different biasing context for each speech segment or stream at runtime, without rebuilding the whole decoding graph. The system keeps HCLG intact, stores only the indices of arcs to boost, and adjusts arc weights on-the-fly during decoding. The boosting discount factor is

$L_{BiLTM^b}$8

Only anticipated contexts are preloaded; runtime switching selects among those preloaded biasing sets rather than constructing new FSTs on the fly (Nigmatulina et al., 2023). The extra per-thread lookup complexity is $L_{BiLTM^b}$9, with arc indices kept sorted for binary search. On Earnings21, GPU contextual biasing reduces EntWER from 60.5 to 52.4 at endpoints while leaving WER at 21.4, and reported RTFX stays about 26.06 (Nigmatulina et al., 2023). This suggests a broader usage of “decoder switching” in which the decoder’s effective scoring regime changes segment by segment while the base graph remains fixed.

6. Weak/strong decoder switching in real-time quantum error correction

A later paper uses the term “decoder switching” explicitly for real-time quantum error correction (Toshio et al., 29 Oct 2025). The framework combines a fast, soft-output “weak decoder” with a slower, high-accuracy “strong decoder.” The weak decoder processes each decoding window and outputs a reliability indicator $L_{BiLTM^m}$0; only when reliability is low does the system switch to the strong decoder. The motivation is to keep average latency close to that of the weak decoder while approaching or surpassing the accuracy of the strong decoder.

The weak decoder is fast and hardware-friendly, with examples including MWPM with complementary gap as soft output and UF with cluster gap as soft output (Toshio et al., 29 Oct 2025). The strong decoder is slower but more accurate, with belief-matching given as the main example. The switching rule is thresholded:

• if $L_{BiLTM^m}$1, trust the weak decoder;
• if $L_{BiLTM^m}$2, switch to the strong decoder.

For MWPM, the complementary gap is

$L_{BiLTM^m}$3

and the signed version is

$L_{BiLTM^m}$4

For UF, the cluster gap $L_{BiLTM^m}$5 is defined by quotient-graph construction and Dijkstra’s shortest path, with

$L_{BiLTM^m}$6

The paper derives the logical error rate under switching as

$L_{BiLTM^m}$7

and the thresholded logical error rate as

$L_{BiLTM^m}$8

The online implementation is the double window decoding scheme. Each weak-decoder window has a commit region of size $L_{BiLTM^m}$9 and buffer region of size $$C = \{c_1, C_2, ... , C_{N_c}\},\quad B = \{b_1,b_2, ... , b_{N_B}\},\quad M = \{m_1, m_2, ... , m_{N_M}\}$$0, often chosen as

$$C = \{c_1, C_2, ... , C_{N_c}\},\quad B = \{b_1,b_2, ... , b_{N_B}\},\quad M = \{m_1, m_2, ... , m_{N_M}\}$$1

with strong-decoder window

$$C = \{c_1, C_2, ... , C_{N_c}\},\quad B = \{b_1,b_2, ... , b_{N_B}\},\quad M = \{m_1, m_2, ... , m_{N_M}\}$$2

The weak decoder continues sliding forward on fresh data while the strong decoder processes only the flagged region (Toshio et al., 29 Oct 2025).

The main sufficient conditions for preventing backlog divergence in a single code patch with one weak decoder and one strong decoder are

$$C = \{c_1, C_2, ... , C_{N_c}\},\quad B = \{b_1,b_2, ... , b_{N_B}\},\quad M = \{m_1, m_2, ... , m_{N_M}\}$$3

together with

$$C = \{c_1, C_2, ... , C_{N_c}\},\quad B = \{b_1,b_2, ... , b_{N_B}\},\quad M = \{m_1, m_2, ... , m_{N_M}\}$$4

The paper also gives

$$C = \{c_1, C_2, ... , C_{N_c}\},\quad B = \{b_1,b_2, ... , b_{N_B}\},\quad M = \{m_1, m_2, ... , m_{N_M}\}$$5

with

$$C = \{c_1, C_2, ... , C_{N_c}\},\quad B = \{b_1,b_2, ... , b_{N_B}\},\quad M = \{m_1, m_2, ... , m_{N_M}\}$$6

For $$C = \{c_1, C_2, ... , C_{N_c}\},\quad B = \{b_1,b_2, ... , b_{N_B}\},\quad M = \{m_1, m_2, ... , m_{N_M}\}$$7, $$C = \{c_1, C_2, ... , C_{N_c}\},\quad B = \{b_1,b_2, ... , b_{N_B}\},\quad M = \{m_1, m_2, ... , m_{N_M}\}$$8, and $$C = \{c_1, C_2, ... , C_{N_c}\},\quad B = \{b_1,b_2, ... , b_{N_B}\},\quad M = \{m_1, m_2, ... , m_{N_M}\}$$9, the estimated requirement is

$B$00

The numerical results use rotated surface code memory experiments at physical error rate $B$01 (Toshio et al., 29 Oct 2025). With fixed threshold $B$02 dB, decoder switching lowers the logical error rate relative to MWPM or UF alone, improves the error threshold by about 0.1%, and matches the threshold of belief-matching closely. For $B$03, MWPM plus belief-matching switching achieves a 3.6× lower logical error rate than MWPM alone and a 1.3× lower logical error rate than belief-matching alone; UF plus belief-matching gives about 6.1× lower than UF alone at the minimum point (Toshio et al., 29 Oct 2025). The switching rate exhibits threshold behavior: below the weak decoder’s threshold it decays exponentially with code distance $B$04. This suggests that, in this domain, decoder switching is a confidence-triggered exception-handling architecture.

7. Comparative interpretation and recurring design principles

Across these literatures, decoder switching is not a single mechanism but a recurring design pattern with different operational meanings. In CMLFormer, it is synchronized interaction between language-specific decoders trained on aligned base-language and mixing-language targets (Baral et al., 19 May 2025). In SC-MoE, it is expert routing within decoder layers so that text-side generation can specialize by language (Ye et al., 2024). In merge-frame video coding, it is the prevention of decoder-state divergence when switching among pre-encoded streams (Dai et al., 2015). In REW-MSLM, it is HMM-governed switching among task-specific multilinear subdecoders (Moly et al., 2022). In GPU ASR contextual biasing, it is runtime switching among preloaded biasing contexts without changing the underlying graph (Nigmatulina et al., 2023). In quantum error correction, it is thresholded delegation from a weak soft-output decoder to a strong decoder on low-confidence instances (Toshio et al., 29 Oct 2025).

Several general principles recur. One is specialization: separate decoder branches or experts are assigned to distinct languages, states, or confidence regimes rather than forcing a single decoder to absorb all variation. Another is conditional coordination: the branches are not necessarily independent, as shown by synchronized dual-decoder cross-attention in CMLFormer and weak/strong scheduling in double window decoding (Baral et al., 19 May 2025, Toshio et al., 29 Oct 2025). A third is state consistency under switching: in video streaming, the essential requirement is that the reconstructed frame be identical regardless of which predecessor stream is active (Dai et al., 2015). A fourth is multi-objective balancing: decoder switching often mediates between competing desiderata such as language specialization versus cross-lingual alignment, recognition accuracy versus computational efficiency, or speed versus accuracy (Baral et al., 19 May 2025, Ye et al., 2024, Toshio et al., 29 Oct 2025).

A common misconception is that decoder switching always denotes a hard router that activates exactly one of several full decoders. The literature does not support that generalization. CMLFormer explicitly does not model decoder switching as a hard gate between decoders, but as synchronized latent interaction (Baral et al., 19 May 2025). REW-MSLM uses probabilistic gating over expert outputs rather than simple winner-take-all selection (Moly et al., 2022). GPU ASR context switching changes active biasing sets while leaving the decoding graph fixed (Nigmatulina et al., 2023). Only some settings, such as top-1 Switch Transformer routing in SC-MoE or weak/strong escalation in quantum error correction, are close to hard switching (Ye et al., 2024, Toshio et al., 29 Oct 2025).

Taken together, these works indicate that decoder switching is best treated as an umbrella term for adaptive decoding architectures in which decoding behavior changes according to linguistic structure, stream provenance, latent state, context, or estimated reliability. This suggests that the most durable conceptual core of decoder switching is not any one algorithmic template, but the systematic replacement of uniform decoding with structured, conditional, and domain-aware decoding pathways.