---
title: Block-wise Autoregressive Decoding (BAR)
url: https://www.emergentmind.com/topics/block-wise-autoregressive-decoding-bar
type: topic
---

# Block-wise Autoregressive Decoding (BAR)

Block-wise Autoregressive Decoding (BAR) is a family of decoding procedures that preserves autoregressive dependence across coarse-grained blocks while exploiting parallel prediction or iterative parallel denoising within each block. In its original form, introduced as blockwise parallel decoding, BAR proposes multiple future tokens in parallel and then verifies the longest prefix that matches the base autoregressive model’s greedy choices; in later diffusion-based formulations, it factorizes the sequence at block granularity and refines the contents of each block in parallel before committing the block and advancing to the next one [1811.03115][2404.09221][2508.19529]. Across language modeling, multimodal generation, and robotic control, BAR has become a bridge between strictly sequential decoding and fully parallel generation, with repeated emphasis on the same structural idea: sequentiality is retained at block boundaries, while computation inside a block is amortized or parallelized [2606.26493][2605.13382].

## 1. Historical emergence and scope

The canonical starting point is “Blockwise Parallel Decoding for Deep Autoregressive Models” [1811.03115], which framed generation acceleration as a predict–verify–accept loop. Instead of emitting one token at a time, the decoder proposes a block of \(k\) tokens in parallel, verifies how many of them the original scoring model would have selected under greedy decoding, and accepts the longest matching prefix. This formulation was presented for deep autoregressive sequence-to-sequence models, with experiments in machine translation and image super-resolution, and it established the central latency argument: generation is sequential in standard autoregressive decoding, but verification of several candidate future positions can be parallelized when the underlying architecture supports it [1811.03115].

Subsequent work broadened both the terminology and the design space. “Exploring and Improving Drafts in Blockwise Parallel Decoding” explicitly treats BPD and BAR interchangeably and studies block drafts produced by multiple prediction heads, together with refinement by \(n\)-gram and neural language models [2404.09221]. In discrete diffusion language models, BAR came to denote a different but structurally related regime: responses are partitioned into fixed-size blocks, blocks are generated sequentially, and each block is denoised or sampled in parallel under a blockwise likelihood factorization [2508.19529]. Later systems such as BlockVLA, BARD, Nemotron-TwoTower, DDTree, and S2D2 show that BAR is no longer confined to a single algorithmic recipe; it now names a broader decoding pattern spanning exact greedy verification, block diffusion, speculative verification, and multimodal or action-conditioned generation [2605.13382][2604.16514][2606.26493][2604.12989][2603.25702].

A common misconception is that BAR is a fully non-autoregressive method. The literature does not support that reading. The defining invariant is block-level chronology: whether drafts are produced by parallel heads or by within-block diffusion, the next block is conditioned on previously committed blocks, and future blocks are not available as context during generation [2508.19529][2605.13382].

## 2. Formal definitions and algorithmic variants

For standard autoregressive decoding, the sequence likelihood is written tokenwise as
\[
p(y_{1:T}\mid c)=\prod_{t=1}^T p(y_t\mid c,y_{<t}),
\]
which requires \(T\) serial forward passes in the usual left-to-right regime [2604.12989]. BAR replaces this finest-grained factorization with a block decomposition. In the diffusion-language-model formulation, with response partitioned into blocks \(\bb^{(1)},\dots,\bb^{(M)}\) of size \(B\),
\[
p_\theta(\bx_{1:L})
=
\prod_{a=1}^M
p_\theta\!\bigl(\bb^{(a)}\mid \context^{(a)},t=0\bigr),
\]
and the corresponding block-wise negative log-likelihood sums over blocks rather than tokens [2508.19529]. BlockVLA uses the same principle for action-token generation:
\[
p(x\mid c)=\prod_{k=1}^K p(x_{(k)}\mid x_{(<k)},c),
\]
where \(x_{(k)}\) is block \(k\) and \(x_{(<k)}\) is the concatenation of previous blocks [2605.13382]. Nemotron-TwoTower states the same idea as
\[
\log p_\theta(x)=\sum_{b=1}^{B}\log p_\theta(x_b\mid x_{<b}),
\]
with fixed-size non-overlapping blocks [2606.26493].

In the original BPD formulation, BAR is a three-stage procedure. First, the model predicts a draft block of \(h\) future tokens in parallel. Second, the base autoregressive model verifies these proposals by re-evaluating each proposed position under the appropriate extended prefix. Third, the decoder accepts the longest contiguous prefix of the draft that exactly matches the greedy autoregressive decode [2404.09221][1811.03115]. Because verification is against the original model, exact BPD preserves the greedy-LM output; what changes is the number of serial calls needed to obtain that output [2404.09221].

In block-diffusion formulations, the internal mechanics differ but the block-level chronology remains. A block is initialized as fully masked, then denoised over a small number of steps, with all masked positions in the block updated in parallel. BlockVLA describes this as “predict-and-refine” within each block: start a new block as fully masked, run \(S\) denoising iterations, select a fraction of the most confident predictions to fix according to a predetermined schedule, and re-mask the remainder until the block is fully filled [2605.13382]. Nemotron-TwoTower uses the same high-level loop with a trainable denoiser tower and a frozen autoregressive context tower, while S2D2 emphasizes that a block-diffusion model becomes autoregressive when the block size is reduced to one, allowing the same pretrained model to act as both drafter and verifier [2606.26493][2603.25702].

## 3. Training objectives, supervision alignment, and bridging AR with diffusion

A central issue in diffusion-based BAR is alignment between training and inference. “Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding” argues that classical supervised fine-tuning with random masking over the entire response is mismatched to semi-autoregressive block decoding because it introduces noisy prefixes, suffix leakage, and granularity mismatch [2508.19529]. Its remedy is Blockwise SFT: partition the response into fixed-size blocks, sample one active block, keep all preceding tokens clean and frozen, fully hide future tokens, and compute loss only on the active block. The paper formalizes this as a block-local diffusion surrogate and states that the training objective remains a variational upper bound on the true block-wise NLL, while Theorem 3.4 shows unbiased gradients via block/time sampling [2508.19529].

BlockVLA addresses the same transition from pretrained autoregressive backbones to diffusion-style block decoding in robotic policies. Pretraining is standard next-token cross-entropy on full action sequences and multimodal prefixes, after which the model is switched to the block diffusion loss \(L_{BD}(\theta)\). The paper states that all other transformer weights remain as in the pretrained AR model and argues that, because the block-wise causal mask still respects global chronology, no catastrophic distribution mismatch arises [2605.13382]. Its default masking strategy is Diffusion Forcing, conditioning block \(b\) on noisy previous blocks because this better simulates inference conditions, whereas Teacher Forcing is described as slightly less robust [2605.13382].

BARD develops a staged path from autoregressive vision-language models to block-wise autoregressive diffusion models. Its progressive supervised block merging uses a geometric block schedule \(\{4,8,16,32\}\), warm-starting each larger-block stage from the previous one, followed by stage-wise intra-dVLM distillation from a fixed small-block diffusion anchor [2604.16514]. The paper explicitly reports that direct autoregressive-to-diffusion distillation is poorly aligned and can even hurt performance, whereas distillation within the diffusion regime is consistently effective [2604.16514]. This suggests that, in BAR systems built on diffusion denoising, preserving training–inference congruence at block granularity is not a secondary implementation detail but a primary determinant of performance.

## 4. Architectural mechanisms: verification, caching, attention, and towers

Classical BAR depends on efficient verification. In BPD, the key object is the accepted prefix length: the decoder accepts the longest contiguous prefix of the draft that matches the greedy autoregressive model, discarding the rest [2404.09221]. The 2024 draft-analysis paper further defines *block efficiency* \(B\) as the average number of tokens accepted per serial call to the BPD model, with a standard greedy autoregressive decoder corresponding to \(B=1\) [2404.09221]. This acceptance-centric viewpoint reappears in speculative decoding, where the practical objective is to increase accepted length per verifier call rather than merely to make parallel proposals.

Block-diffusion BAR introduces different systems bottlenecks, notably KV caching and within-block attention. BlockVLA states that completed blocks can be appended to the KV cache and subsequent block denoising reuses them as fixed causal context, so prefix KV-cache reuse reduces the effective cost of iterative denoising [2605.13382]. Its attention pattern gives full bidirectional attention within the multimodal prefix \(c=[BOS,v,p,l]\), forbids the prefix from attending to action tokens, and allows action blocks to attend to the prefix plus appropriate previous blocks under a causal block mask [2605.13382]. BARD formalizes the same principle at the block level: the \(i\)th noisy block may attend to the fixed multimodal context \(Q\), all clean previous blocks, and itself, but not to future clean or noisy blocks [2604.16514].

Nemotron-TwoTower splits BAR into two explicit computational roles. A frozen AR context tower causally processes clean tokens and maintains KV caches and Mamba boundary states, while a trainable diffusion denoiser tower uses bidirectional block attention within the current block and cross-attention to the layer-aligned caches of the context tower [2606.26493]. Time conditioning is provided through adaptive LayerNorm, and denoiser Mamba layers are initialized from the context tower’s recurrent states at the previous block boundary [2606.26493]. This decoupling is specific to TwoTower, but it makes explicit a general BAR theme: past context representation and current-block refinement need not be implemented by the same computational path.

Speculative variants alter the verification structure rather than the block factorization itself. DDTree constructs a draft tree from the per-position marginals of a block diffusion drafter, then verifies all tree nodes in one target-model forward pass using an ancestor-only attention mask, under which each node attends only to its own context and its ancestors in the tree [2604.12989]. S2D2 inserts a speculative AR verification step into block-diffusion decoding and uses routing policies such as minimum-span, score-threshold, hysteresis, and contextual-bandit (UCB) rules to decide when verification is worth its cost [2603.25702]. In both cases, BAR remains the outer scaffold; the innovation lies in how the block proposal is validated and consumed.

## 5. Empirical results and quantitative trade-offs

The empirical literature consistently reports latency reductions or throughput gains, but the precise trade-off depends on whether BAR is exact greedy verification, draft refinement, block diffusion, or self-speculative verification.

| Setting | Reported result | Source |
|---|---|---|
| Original blockwise parallel decoding | iteration reductions of up to 2x with no loss in quality, or up to 7x in exchange for a slight decrease in performance; real-time speedups of up to 4x | [1811.03115] |
| Refined block drafts | +5–21% increase in block efficiency across diverse datasets | [2404.09221] |
| DDTree over DFlash | HumanEval (8B): \(4.84\times \rightarrow 6.90\times\), \(\tau: 6.61 \rightarrow 9.67\); GSM8K (8B): \(4.78\times \rightarrow 6.75\times\), \(\tau: 6.57 \rightarrow 9.54\); Alpaca (4B): \(2.03\times \rightarrow 3.32\times\), \(\tau: 3.11 \rightarrow 5.35\) | [2604.12989] |
| BlockVLA on a single NVIDIA RTX 4090 | AR throughput \(\approx 23\) tokens/s; DDVLA \(\approx 56\) tokens/s; BlockVLA \(=186.7\) tokens/s, \(3.3\times\) faster than DDVLA and \(8\times\) faster than AR | [2605.13382] |
| Nemotron-TwoTower on 2× H100 GPUs | 98.7 % of AR baseline quality and \(2.42 \times\) higher wall-clock generation throughput | [2606.26493] |
| S2D2 on SDAR | up to \(4.7\times\) speedup over autoregressive decoding, and up to \(1.57\times\) over a tuned dynamic decoding baseline while improving accuracy by up to \(4.5\) points | [2603.25702] |

These numbers are accompanied by domain-specific observations. In the original 2018 study, exact BAR on WMT’14 EN→DE achieved mean accepted block size \(\approx 1.7\)–\(1.9\), corresponding to \(\approx 2\times\) fewer iterations, while distillation plus fine-tuning increased mean accepted block size and yielded real-time speedup \(\approx 3.3\times\) with a small BLEU decrease [1811.03115]. The 2024 draft-refinement study reported that raw BAR drafts exhibit an unnaturally high rate of token repetition, with consecutive repetition ranging from 20% up to 75% depending on task, and it linked higher efficiency to lower head entropy in early prediction heads [2404.09221]. In DDTree, all \(10\times 3\times 2=60\) settings strictly improved both speedup and acceptance length over vanilla DFlash [2604.12989].

In robotic control, BlockVLA reports not only inference acceleration but faster convergence. On LIBERO at 50k steps, OpenVLA (AR) achieved 76.5%, DDVLA 83.2%, and BlockVLA 91.7%; BlockVLA reached 80% success in only \(\sim 15k\) fine-tuning steps, whereas DDVLA took \(\sim 30k\), and on LIBERO-Long it hit 60% in 15k steps versus 3.8% for DDVLA [2605.13382]. On SimplerEnv, across 4 pick-and-place tasks and 60k steps, BlockVLA matched or slightly exceeded DDVLA success counts while using only \(S=2\) diffusion passes per block versus 12 in DDVLA [2605.13382]. These results indicate that, in embodied domains, BAR is being evaluated not only as a decoding acceleration technique but also as a training and control-policy design choice.

## 6. Applications, misconceptions, and open research directions

BAR now appears across several application classes. The original blockwise parallel decoding paper covers machine translation and image super-resolution [1811.03115]. The 2024 draft-analysis work studies language modeling, question answering, and summarization, including LAMBADA, SQuAD V1, CNN/DailyMail, SAMSum, MultiNews, XSUM, and NewsRoom [2404.09221]. DDTree benchmarks reasoning, code, and instruction tasks on Qwen3 target models [2604.12989]. Blockwise SFT focuses on GSM8K, MATH, and MetaMathQA [2508.19529]. BARD targets vision-language models, Nemotron-TwoTower targets large-scale diffusion language modeling with pretrained autoregressive context, and BlockVLA applies BAR to Vision-Language-Action policies for LIBERO and SimplerEnv [2604.16514][2606.26493][2605.13382].

One persistent misconception is that BAR guarantees quality preservation. That claim is only true for exact verification variants that explicitly back off to the original autoregressive model; both the 2018 paper and the 2024 draft-refinement study state that exact verification preserves the greedy-LM output, so generation quality is identical in that regime [1811.03115][2404.09221]. Diffusion BAR methods instead expose explicit quality–throughput fronts. Nemotron-TwoTower reports a default operating point of \(\gamma=0.8\), block size \(S=16\), yielding 98.7 % of baseline quality at \(2.42\times\) throughput, while larger speedups are available at larger quality loss [2606.26493]. BARD likewise reports graceful quality degradation as block size increases and attributes quality recovery to progressive supervised block merging and stage-wise distillation [2604.16514].

A second misconception is that block size is a purely engineering hyperparameter. The block-size consistency study in Blockwise SFT shows that accuracy peaks when \(B_{\rm train}=B_{\rm infer}\) and degrades for large mismatches, with a visible “hot-band” along the diagonal of the train–infer block-size grid [2508.19529]. BlockVLA similarly reports that \(B=14\) tokens per block gave best results and that smaller \(B=7\) under-performed [2605.13382]. This suggests that block granularity functions as part of the model’s effective factorization, not merely as a runtime knob.

Open directions are stated explicitly across the literature. The original BAR paper points to extending beyond greedy decoding, scaling to larger language models, reducing memory overhead in parallel verification, and designing more advanced drafting-head architectures and training criteria [1811.03115]. The 2024 draft-analysis study highlights dynamic halting and adaptive rescoring based on head-entropy profiles [2404.09221]. DDTree identifies variable-length blocks, adaptive budgets, and hybrid schemes combining autoregressive and diffusion drafters as promising avenues [2604.12989]. S2D2 contributes a training-free self-speculative route for block-diffusion models, implying that future BAR systems may increasingly mix blockwise diffusion proposals with localized autoregressive critics rather than choosing between them outright [2603.25702].

Source: https://www.emergentmind.com/topics/block-wise-autoregressive-decoding-bar