Streaming-dLLM: Accelerated Diffusion LLM

• Streaming-dLLM is a training-free framework that accelerates diffusion-based LLM inference by applying suffix pruning and adaptive decoding.
• It achieves spatial efficiency by reducing context via attenuation-guided modeling and improves temporal efficiency with dynamic confidence thresholds.
• Empirical results demonstrate significant throughput gains and lower latency, making it ideal for long-form, latency-sensitive applications.

Streaming-dLLM refers to a training-free acceleration framework for diffusion-based LLMs (dLLMs) focused on optimizing spatial and temporal efficiency in diffusion decoding. It targets the inefficiencies of block-wise dLLM inference—specifically, spatial redundancy in attending to long, uninformative suffixes and temporal inefficiency from fixed masking schedules—by introducing suffix pruning and dynamic, confidence-aware decoding. Streaming-dLLM can be deployed as a plug-and-play module for dLLM inference, yielding significant improvements in throughput and latency with negligible impact on output quality (Xiao et al., 25 Jan 2026).

1. Diffusion-Based LLMs and Inference Inefficiency

Diffusion-based LLMs (dLLMs) generate target sequences by iterative refinement of masked token blocks. For a target sequence of length $L$, the initial state consists of a prompt $p_0$ followed by $L$ masked positions:

$x^{(0)} = [p_0, [MASK]_1, \ldots, [MASK]_L]$

Tokens are grouped into $N$ non-overlapping blocks of size $K$ (thus $L = N \cdot K$). At each diffusion step $t$ (for a total $T = N \cdot M$ diffusion steps), the model $f_\theta$ predicts logits for all masked positions:

$$z^{(t)} = f_\theta(x^{(t)}), \quad c_i^{(t)} = \max \mathrm{Softmax}(z_i^{(t)}), \quad \hat{x}_i^{(t)} = \arg\max \mathrm{Softmax}(z_i^{(t)})$$

A selection rule determines which masked positions are updated. Bidirectional attention across prompt plus all masked tokens enables superior global coherence relative to autoregressive models and allows tokens within a block to be finalized in parallel. However, standard dLLM inference attends over the entire masked suffix at each step, incurring unnecessary computation as $L$ increases, and applies fixed confidence thresholds, causing either excess waiting for high-confidence tokens or premature updates for uncertain positions (Xiao et al., 25 Jan 2026).

2. Spatial Acceleration: Suffix Pruning via Attenuation-Guided Modeling

Streaming-dLLM introduces an attenuation-guided suffix modeling strategy. Empirically, attention scores from the current block to distant suffix blocks decay rapidly, indicating that only a narrow window of suffix blocks and the end-of-sequence token provide significant contextual utility. The framework constructs a sliding window of $w$ contiguous suffix blocks $(K$ tokens per block$)$ and includes the final token to capture necessary position information. The effective context at step $t$ is then:

$$\tilde{x}^{(t)} = S_\text{prefix}^{(t)} \cup S_\text{current}^{(t)} \cup \tilde{S}_\text{suffix}^{(t)}$$

where $S_\text{prefix}^{(t)}$ covers the prompt and all decoded blocks, $S_\text{current}^{(t)}$ the active block, and $\tilde{S}_\text{suffix}^{(t)}$ the pruned subset of the suffix. This reduces the per-layer attention cost from $O((|\text{prefix}|+|\text{current}|+L) \cdot d)$ to $$O((|\text{prefix}|+|\text{current}|+w \cdot K +1)\cdot d)$$. Empirical results confirm that using a much smaller context (e.g., $w=32$ for $K=32$) does not degrade output quality (Xiao et al., 25 Jan 2026).

3. Temporal Acceleration: Dynamic Decoding with Adaptive Thresholding

Classical dLLMs employ a fixed confidence threshold $\tau$ to determine when masked positions are updated. Streaming-dLLM instead adopts an adaptive thresholding rule:

$$\tau^{(t)} = \tau_0 \cdot (1 - \alpha \cdot (1 - r_\text{mask}))$$

where $r_\text{mask}$ is the fraction of still-masked positions in the current block, $\tau_0 \in (0,1)$ is a base threshold, and $\alpha \in [0,1]$ determines how aggressively the threshold adapts. At each iteration, all tokens $i$ with $c_i^{(t)} \geq \tau^{(t)}$ are unmasked; if none pass the threshold, the most confident token is updated to ensure progress. An early exit mechanism halts decoding if a high-confidence EOS token is produced, thus saving unnecessary iterations for converged outputs. This dynamic policy improves sample efficiency and model throughput (Xiao et al., 25 Jan 2026).

4. Streaming-dLLM Inference Algorithm

Streaming-dLLM simply replaces full-suffix attention with pruning (as above) and fixed-threshold unmasking with dynamic thresholding and early exit. No model retraining is required. The inference pseudocode is as follows:

x[0] = [p_0, MASK^L] for c in 0...N-1: prefix = [p_0] + decoded blocks < c KV_prefix = f_theta.KV_encode(prefix) mask_positions = current block indices for t in 0...M-1: current = x^{(t)}[current block] pruned_suffix = next w blocks after c + end position x_tilde = prefix + current + pruned_suffix {z_i, c_i} = f_theta.forward_query(x_tilde, KV_prefix) r_mask = fraction masked in current block tau = tau_0 * (1 - alpha * (1 - r_mask)) to_update = {i | c_i >= tau} if empty(to_update): to_update = {argmax_i c_i} update x_i for i in to_update if any updated x_i is EOS: return full sequence if all i in current block unmasked: break

Practical values typically set $w \in [32,128]$ ($K=32$), $\tau_0 = 0.9$, and $\alpha \in [0.3, 0.6]$ (Xiao et al., 25 Jan 2026).

5. Empirical Performance and Comparative Evaluation

Streaming-dLLM achieves substantial improvements in speed and computational efficiency while preserving or slightly improving output quality:

• Throughput: Up to $68.2 \times$ speedup on MBPP@512 tokens; up to $225.3 \times$ for $L=2048$.
• Latency: Up to $85.5\%$ reduction in per-sample inference time.
• Quality: Output accuracy remains within $\pm 0.5\%$ of the full-suffix baseline. Compared to dKV-Cache, Prefix-Cache, and Fast-dLLM baselines, Streaming-dLLM delivers higher throughput and comparable or superior accuracy (e.g., on GSM8K@512, Fast-dLLM achieves $25.8$ TPS vs. $69.8$ TPS for Streaming-dLLM). Ablation studies show that each component—suffix pruning, dynamic decoding, early exit—contributes to the overall speedup (Xiao et al., 25 Jan 2026).

Method	Speedup (vs baseline)	Quality Δ
Suffix pruning	$\sim1.8\times$	Slight gain
+ Dynamic decoding	$\sim2.0\times$	Minor change
+ Early exit	$\sim2.7\times$	None

6. Implementation and Deployment Considerations

• Hyperparameter tuning: The suffix window ($w$) controls the balance between speed and quality; recommended $w$ values are $[32,128]$. Setting $\tau_0$ too low or $\alpha$ too high may cause premature unmasking and reduced quality.
• Applicability: Best suited for long-form generation tasks ($L \gg |\text{prompt}|$), block-wise diffusion architectures (e.g., Dream-7B, LLaDA-1.5/8B), and interactive systems where latency is critical.
• Plug-and-play: Streaming-dLLM operates as a training-free wrapper, requiring only modifications to the inference routine without retraining the underlying dLLM (Xiao et al., 25 Jan 2026).

7. Significance and Broader Impact

Streaming-dLLM represents a practical advance for accelerating natural language generation in diffusion-based LLMs, effectively addressing inefficiencies that scale with output length and enabling near-real-time inference in settings that previously suffered from substantial computational overhead. Its design principles—attenuation-guided context pruning, adaptive masking, and early exit—may influence efficient inference in other non-autoregressive generative frameworks and facilitate broader adoption of dLLMs in latency-sensitive and high-throughput applications (Xiao et al., 25 Jan 2026).