Introspective Strided Decoding (ISD) Overview

• ISD is an inference algorithm that combines parallel token block proposals with an introspection step verifying each token against AR distributions.
• It employs adaptive stride control and single-pass introspection, ensuring tokens meet autoregressive quality while optimizing decoding speed.
• Empirical evaluations show ISD can achieve up to 2.6x speedup and enhanced throughput over traditional DLMs, maintaining comparable output quality.

Introspective Strided Decoding (ISD) is an inference algorithm central to Introspective Diffusion LLMs (I-DLMs), designed to merge the parallel generation efficiency of diffusion-style block prediction with the self-consistency guarantees of autoregressive (AR) models. ISD achieves this by including an introspection step, which verifies proposed tokens against the model’s own causal next-token distribution in a single forward pass, enabling high-throughput generation without sacrificing output quality equivalence to AR decoding (Yu et al., 13 Apr 2026).

1. Objectives and Operational Principles

ISD is devised to address the introspective inconsistency commonly observed in classical Diffusion LLMs (DLMs), where generated tokens may not align with what the model would produce autoregressively. The primary objectives and workflow of ISD are as follows:

• Parallel stride generation: At each decoding step, the model proposes a block of $N$ new tokens in a single forward pass, not one at a time as in AR decoding.
• Introspective consistency enforcement: Each proposed token is introspected—that is, its log-probability under the AR model (the causal anchor distribution $p$) is recomputed and compared to the proposal distribution ($q$), directly enforcing alignment between diffusion-style proposals and AR likelihoods.
• Adaptive stride control: ISD accepts high-confidence proposals in parallel. Tokens failing the introspective acceptance criterion are resampled with a fallback to smaller stride, ensuring that, in the worst case, AR-style sequential decoding is recovered.
• Single-pass introspection: The same forward pass that proposes $N$ tokens also computes the AR anchor distributions for consistency checking, incurring no additional compute overhead.
• Provable distribution equivalence: By construction, ISD can, via the $p/q$ acceptance criterion, output a sample distribution-identical to that produced by pure AR decoding of the base model (Yu et al., 13 Apr 2026).

2. Mathematical Foundation

At decoding step $t$ with prefix $x_1, \ldots, x_k$, ISD operates as follows:

• Proposal generation: $N$ mask tokens are appended to the prefix, and the logit-shifted, causal-attention LLM $M$ runs a single forward pass. This yields stride logits for positions $k, \ldots, k + N - 1$, from which proposal distributions $p$0 are obtained. The first, $p$1, yields the exact AR next-token.
• Introspection anchor: These same positions are immediately re-evaluated by feeding the proposed tokens back as "clean" input, producing anchor logits $p$2—the causal AR distributions.
• Acceptance criterion: For each proposed token $p$3, acceptance probability is computed as

$p$4

• Sequence-wide acceptance rate: The introspective acceptance rate is defined as

$p$5

where $p$6 is the output sequence length.

• Stride update rule: Upon rejection at position $p$7, the proposal is resampled from the normalized positive residual $p$8, and all subsequent proposals are discarded. The effective stride adaptively matches the number of accepted tokens.

Tokens-per-forward (TPF) efficiency is described by:

$p$9

where $q$0 is the uniform per-token proposal acceptance rate. With $q$1 and $q$2, TPF is approximately $q$3–$q$4 (Yu et al., 13 Apr 2026).

3. ISD Algorithmic Workflow

ISD is implemented in a three-phase step:

1. Stride-Propose: Append $q$5 masks to the prefix. The forward pass provides "stride logits," from which proposals are sampled ($q$6 is always AR-exact).
2. Introspect: Prepare a new input by appending accepted proposals as "clean" context, then perform a forward pass to yield AR anchor distributions $q$7.
3. Acceptance and adaptive stride: For each proposal, calculate $q$8 (as above). If accepted, commit it; otherwise, resample from the residual and truncate the stride. If all $q$9 proposals are accepted, a bonus token can be sampled, extending the stride.

Annotated Python-style pseudocode is provided in the original work, reflecting these stages; the mask handling, proposal/anchor logic, KV-cache management, and stride adaptation are all precisely aligned for efficient batch and cache reuse (Yu et al., 13 Apr 2026).

4. Systems Optimizations and Serving Infrastructure

ISD leverages AR-inherited model architecture and serving optimizations:

• Causal attention and logit shifting: I-DLM training guarantees causal masking and logit shifts, maintaining compatibility with pure AR "extend" operations.
• Batching and cache utilization: ISD is integrable with AR-style continuous batching and paged KV-caches; no diffusion-specialized cache or commit step is needed.
• Single attention kernel per layer: By restricting extended inputs to small $N$0 lengths, attention kernels are fused and efficient, reducing the overhead of per-layer kernel launches.
• Stationary-batch scheduler: To mitigate decode-stage CPU overhead breaking the flow of tightly chained ISD steps, stationary-batch decoding loops reuse batch objects, update metadata in-place (using captured CUDA graphs), and defer non-critical I/O to background threads, thus recovering or exceeding AR throughput at scale.

5. Computational Complexity and Empirical Throughput

ISD is evaluated with respect to overhead (OH), defined as forward-query tokens per output token. For AR, $N$1. For ISD at acceptance $N$2, stride $N$3:

• $N$4
• $N$5
• Compute efficiency: $N$6

For typical settings ($N$7, $N$8): $N$9, $p/q$0, yielding notably higher FLOP efficiency compared to prior DLM approaches (SDAR, TiDAR).

Empirical results on NVIDIA H100 (I-DLM-8B, $p/q$1):

• $p/q$2 per-request speedup over AR for $p/q$3-token generations.
• At concurrency $p/q$4, $p/q$5–$p/q$6 higher throughput versus LLaDA-2.1-mini (16B), $p/q$7–$p/q$8 versus SDAR (8B).
• Peak single-request TPS: $p/q$9 (ISD) vs $t$0 (AR) and $t$1 (SDAR) (Yu et al., 13 Apr 2026).

6. Output Quality and Benchmark Performance

Across diverse benchmarks in mathematics, coding, and instruction-following, I-DLM using ISD demonstrates quality matching or exceeding that of its same-scale AR baseline (Qwen3), while outperforming previous DLMs:

Benchmark	Qwen3-8B (AR)	I-DLM-8B (ISD)	LLaDA-2.1-mini (16B)
AIME-24 (math)	73.1	69.6	43.3
LiveCodeBench-v6	50.3	45.7	30.4
MATH-500 (math)	95.8	96.8	85.0
HumanEval (code)	95.1	93.3	86.0
IFEval (instr)	84.7	84.7	83.2

Key observations:

• I-DLM-8B achieves a 26.3-point improvement over LLaDA-2.1-mini on AIME-24 and a 15.3-point increase on LiveCodeBench-v6, despite having half the parameter count.
• I-DLM-8B accuracy is, on average, within ±1 point of the AR baseline over 15 benchmarks, evidencing the effectiveness of ISD for attaining AR-level output quality (Yu et al., 13 Apr 2026).

7. Significance and Implications

ISD represents the first decoding method for DLMs that achieves AR-equivalent quality through a single-pass, parallel decoding architecture. By leveraging introspective self-consistency via causal masking, logit shift, and p/q acceptance, ISD closes the longstanding DLM–AR quality gap and unlocks substantial throughput gains. This compatibility with mature AR serving infrastructure, along with principled distributional guarantees and high empirical efficiency, positions ISD as a foundational inference method in the context of large-scale, high-concurrency LLM deployment (Yu et al., 13 Apr 2026).