Dynamic Right Context Masking

• Dynamic Right Context Masking is a technique that selectively alters future context via stochastic strategies to balance look-ahead benefits with computational constraints.
• It employs methods like per-chunk Bernoulli masking in ASR, rolling-window observation masking in LLM agents, and appended mask tokens in diffusion models to align training and inference conditions.
• Empirical evidence shows improvements such as up to a 13.9% reduction in WER for ASR and over 50% cost reduction in LLM settings, highlighting its efficiency and robustness.

Dynamic right context masking refers to a class of techniques for selectively omitting, randomizing, or replacing future or distal context information—either at training or inference time—using dynamic masking strategies that adapt the context fed to a model based on geometric locality, computational considerations, or stochastic schedule. The term spans multiple domains, including streaming automatic speech recognition (ASR), LLMs, and diffusion LLMs, with concrete algorithmic realizations tailored to the structural, efficiency, and robustness requirements of each setting. Common objectives include aligning training and inference conditions under constrained context availability, controlling computation, mitigating context overload, and improving robustness to context truncation.

1. Formalism and Algorithmic Structure

Dynamic right context masking for streaming ASR, as described in "Improving Streaming Speech Recognition With Time-Shifted Contextual Attention And Dynamic Right Context Masking" (Le et al., 21 Feb 2025), operates on chunked sequences of input frames. In this paradigm, audio is segmented into non-overlapping chunks of size $c$, each chunk optionally being extended with up to $r$ future ("right-context") frames according to a per-chunk Bernoulli schedule.

Let $T$ denote the total number of frames, with chunk index $i_t = \lfloor t/c \rfloor$. For each chunk $i$, an extension variable $\delta_i \sim \mathrm{Bernoulli}(p)\cdot r$ determines if additional right-context is included. The effective receptive field $C_i = c + \delta_i$, and the dynamic attention mask $M \in \{0,1\}^{T \times T}$ is given by:

$$M_{t,k} = \begin{cases} 1, & \text{if}\ i_t \cdot c - l \leq k < i_t \cdot c + C_{i_t} \ 0, & \text{otherwise} \end{cases}$$

Here, $l$ is left-context size. This masking is applied in both self-attention and convolutional layers to enforce a variable, stochastic right-context on each chunk during training, simulating diverse look-ahead scenarios found in deployment. During inference, techniques such as Time-Shifted Contextual Attention (TSCA) exploit this learned generality to seamlessly leverage any available right-context without distributional mismatch (Le et al., 21 Feb 2025).

In LLM-based agent settings, dynamic right context masking appears as "Observation Masking"—a rolling window where older environment observations are replaced by concise placeholders, preserving the most recent $r$0 observations verbatim. For a history $r$1 with $r$2, the masked trajectory is:

$r$3

with

$r$4

where $r$5 is a placeholder such as "Previous output omitted for brevity" (Lindenbauer et al., 29 Aug 2025).

2. Motivations and Theoretical Considerations

The primary objective of dynamic right context masking is to bridge the gap between real-time constraints and the potential utility of future or distal information, while maintaining computational and memory efficiency. In streaming ASR, fixed causal chunking limits access to future information, often degrading accuracy on tasks requiring look-ahead. Dynamic masking:

• Provides a curriculum over possible future context lengths, thus regularizing the model towards robust performance across look-ahead configurations.
• Reduces train-test distribution mismatch when inference-time tricks (e.g., TSCA) provide additional context not uniformly present during training (Le et al., 21 Feb 2025).

In LLM-based agent settings, large and verbose context histories rapidly exhaust LLM context windows and increase inference cost. Dynamic right context masking (as observation masking):

• Halves the number of context tokens (notably in environments where $r$6 of context is observation tokens) (Lindenbauer et al., 29 Aug 2025).
• Exploits the "lost-in-the-middle" phenomenon, where LLMs are much more sensitive to loss of recent than distant context, ensuring that essential recent information is preserved.

In diffusion language modeling, right-context masks (appended mask tokens) act as strong distractors ("attention sinks") and induce a locality bias, even in bidirectional architectures. Managing such masking is thus critical to context comprehension (Piskorz et al., 26 Nov 2025).

3. Empirical Performance and Sensitivity Analyses

In streaming ASR, dynamic right context masking, in combination with TSCA at inference, achieves up to $r$7 relative word error rate (WER) reduction over standard causal masking, with negligible added latency and preserved batch efficiency. Empirically, the optimal probability $r$8 of right-context extension was found to be approximately $r$9; lower $T$0 limits the model's ability to use look-ahead, while $T$1 leads to excessive leakage and test-train mismatch (Le et al., 21 Feb 2025). For example, the DRC+TSCA configuration with $T$2 achieved WERs of $T$3 (test-clean) and $T$4 (test-other) compared to $T$5 for pure causal chunking, a $T$6 improvement.

In LLM-based SE agents on the SWE-bench-Verified benchmark, rolling-window observation masking with $T$7 consistently reduces per-instance cost by $T$8 versus raw agent baselines, with solve rates matching or slightly exceeding those obtained with summarization-based context management (Lindenbauer et al., 29 Aug 2025). For Qwen3-Coder-480B, the solve rate improved from $T$9 to $i_t = \lfloor t/c \rfloor$0, and cost per instance dropped from $i_t = \lfloor t/c \rfloor$1 to $i_t = \lfloor t/c \rfloor$2.

In diffusion LLMs, appending right-context mask tokens triggers significant accuracy drop—$i_t = \lfloor t/c \rfloor$3 and $i_t = \lfloor t/c \rfloor$4 points for LLaDA-Base and LLaDA-Instruct respectively with $i_t = \lfloor t/c \rfloor$5 right-context masks; iterative unmasking can partially recover this loss at the expense of increased inference latency (Piskorz et al., 26 Nov 2025). Mask-agnostic loss fine-tuning reduces this robustness gap by $i_t = \lfloor t/c \rfloor$6.

4. Engineering, Implementation, and Domain-Specific Guidelines

In streaming ASR, DRC masks require only minor implementation adjustments:

• Mask generation can be fully vectorized, enabling efficient GPU batching.
• No runtime overhead is incurred; all mask stochasticity is at training, with inference free to exploit available context by mask logic alone.
• The chunk size $i_t = \lfloor t/c \rfloor$7 and maximum right-context $i_t = \lfloor t/c \rfloor$8 should be selected to balance accuracy gain against memory/compute footprint ($i_t = \lfloor t/c \rfloor$9 is effective).

For LLM agents, recommendations include:

• Use window $i$0 as initial setting; monitor cost/accuracy trade-off and domain-specific context composition for effective ablation.
• Re-tokenize after masking to ensure contiguous positional embeddings; no changes to model internals are required.
• Hybrid masking/summarization can be triggered adaptively, e.g., when the model appears stuck, for chain-of-thought reasoning (Lindenbauer et al., 29 Aug 2025).

For MDLMs, practical evaluation protocols must report the number of right-context masks and perform mask-sensitivity sweeps (e.g., 0–50 masks). Mask-agnostic fine-tuning yields robustness to varying mask counts without architectural changes (Piskorz et al., 26 Nov 2025).

5. Broader Implications and Limitations

Dynamic right context masking closes the train-test gap under variable look-ahead, supports seamless integration of future-context tricks (like TSCA), and reduces computation and memory cost without inducing accuracy loss in most regimes. However, several limitations are notable:

• Overzealous masking of right context (i.e., large $i$1 or $i$2) during training can cause performance collapse if inference rarely provides corresponding look-ahead.
• In diffusion LMs, long contiguous right-context mask spans that do not occur in the training curriculum significantly degrade performance, indicating a need for better-aligned training and evaluation regimes (Piskorz et al., 26 Nov 2025).
• In LLM agents, masking underperforms in certain regimes (notably chain-of-thought "thinking" modes on Gemini 2.5 Flash) indicating older information can be essential for specific agent strategies—potentially necessitating adaptive or hybrid approaches (Lindenbauer et al., 29 Aug 2025).

6. Comparative Overview

Domain	Masking Scheme	Core Benefit	Limitation
Streaming ASR (Le et al., 21 Feb 2025)	DRC mask (per chunk random)	Improves WER, enables TSCA	Sensitive to $i$3; memory grows with $i$4
LLM Agents (Lindenbauer et al., 29 Aug 2025)	Rolling window, placeholders	Halves context cost, matches solve rates	Underperforms in CoT for some models
Diffusion LMs (Piskorz et al., 26 Nov 2025)	Appending right-context masks	Simpler decoding, facilitates bidirectional models	Strong accuracy drops with many masks

Dynamic right context masking constitutes a general strategy for efficient, robust, and cost-effective context management across a range of modern sequence modeling tasks, demanding careful tuning and evaluation according to modality, model architecture, and downstream requirements.