Bidirectional Autoregressive Latent Diffusion

• Bidirectional autoregressive latent diffusion is a design space that integrates tokenized latent representations, iterative denoising, and flexible ordering to enhance generative performance.
• These methods employ innovative attention masking and corruption strategies to balance bidirectional context with autoregressive dynamics across diverse applications like text-to-motion and video analysis.
• Empirical evaluations in systems such as BAD, LaDiR, VidLaDA, and BARD demonstrate improved metrics and efficiency over traditional left-to-right generation frameworks.

Bidirectional autoregressive latent diffusion denotes a family of generative formulations that combine iterative denoising in a latent or tokenized representation with an ordered rollout, while relaxing the strictly left-to-right conditioning of classical causal autoregression. Recent usage spans several distinct constructions: BAD models text-to-motion via permutation-based corruption and hybrid bidirectional/causal attention over discrete motion tokens (Hosseyni et al., 2024); LaDiR performs reasoning in VAE latent blocks with bidirectional attention within each block and causal ordering across blocks (Kang et al., 6 Oct 2025); VidLaDA replaces causal video-language decoding with a bidirectional diffusion LLM over masked response tokens (He et al., 25 Jan 2026); and BARD learns latent magnetohydrodynamic dynamics that run both forward and backward in time with shared weights (Scheinker, 28 Jun 2026). A related compression-oriented case, ZeroGVC, remains fundamentally causal but introduces an optional future-reference pathway, showing that the boundary of the concept is not canonical (Gao et al., 21 Jun 2026).

1. Terminological scope and recurring formulations

The literature does not present a single universal factorization under the label. Instead, closely related papers instantiate different combinations of three ingredients: a compressed or tokenized internal representation, an iterative denoising process, and a directional structure that is not purely left-to-right. This suggests that “bidirectional autoregressive latent diffusion” is best treated as a design space rather than a fixed architecture.

System	Domain	Bidirectional-autoregressive construction
BAD	Text-to-motion	Bidirectional attention among unmasked tokens plus permuted causal attention among masked tokens (Hosseyni et al., 2024)
LaDiR	Text reasoning	Bidirectional attention within a latent reasoning block, causal attention across blocks (Kang et al., 6 Oct 2025)
VidLaDA	Video understanding	Full bidirectional attention over visual, prompt, and masked response tokens; iterative denoising replaces causal next-token decoding (He et al., 25 Jan 2026)
BARD	Magnetohydrodynamics	Shared latent diffusion dynamics used for forward and backward temporal prediction (Scheinker, 28 Jun 2026)
ZeroGVC	Video compression	Causal autoregressive latent diffusion prior with an optional future I-frame reference (Gao et al., 21 Jun 2026)

Across these systems, the autoregressive unit is not uniform. BAD orders masked motion tokens through a sampled permutation, LaDiR orders latent reasoning blocks, BARD orders latent time steps, and ZeroGVC orders P-frames inside a GOP. VidLaDA is the main exception: it is described as being in the spirit of a bidirectional autoregressive latent diffusion-style model, but it is explicitly not autoregressive in the causal-mask sense, because its central intervention is to replace the causal decoder with a bidirectional diffusion LLM (He et al., 25 Jan 2026).

2. Structural principles: latent state, ordering, and bidirectionality

A common structural move is to separate representation learning from generative dynamics. In BAD, motion is tokenized by a VQ-VAE into a discrete sequence $X=\{x_1,\dots,x_T\}$, after which generation occurs in token space rather than directly in 3D pose space (Hosseyni et al., 2024). In LaDiR, each chain-of-thought sentence is encoded by a $\beta$-VAE into a block of continuous latent thought tokens $$\mathbf{Z}^{(b)}=\{z^{(b)}_1,\ldots,z^{(b)}_{L_b}\}$$, with a frozen pretrained LLM decoder preserving textual interpretability (Kang et al., 6 Oct 2025). In BARD, each MHD field is mapped from $512\times 512$ to $16\times 16\times 4$, a $256\times$ compression, and the diffusion model operates on the resulting latent trajectories rather than on field grids directly (Scheinker, 28 Jun 2026). ZeroGVC similarly performs reconstruction in the latent space of WanVAE, compressing the denoising trajectory rather than transmitting dense latent residuals (Gao et al., 21 Jun 2026). VidLaDA uses multimodal token sequences $V,P,R_t$, where video tokens and prompt tokens remain visible while masked response tokens are iteratively denoised (He et al., 25 Jan 2026).

Bidirectionality is likewise implemented in several non-equivalent ways. BAD defines a hybrid attention mask,

$$\text{att}_{\text{hyb}}=\text{att}_{\text{bi}}+\text{att}_{\text{per}},$$

where unmasked tokens receive bidirectional attention and masked tokens are generated under a permuted causal order (Hosseyni et al., 2024). LaDiR uses a related but block-structured mask: attention is bidirectional within a reasoning block and causal across blocks, so a single reasoning step can be holistically revised while the overall chain remains ordered (Kang et al., 6 Oct 2025). VidLaDA removes the causal mask entirely and applies full bidirectional attention to the entire multimodal sequence, arguing that video evidence is distributed across space and time rather than naturally organized for left-to-right decoding (He et al., 25 Jan 2026). BARD implements bidirectionality not through token attention but through temporal dynamics, using one parameter set for both $c_d=+1$ and $c_d=-1$ transitions (Scheinker, 28 Jun 2026). ZeroGVC preserves causal low-delay generation as the default, with bidirectionality appearing only as an optional future I-frame reference that does not alter the primary forward factorization (Gao et al., 21 Jun 2026).

3. Corruption and denoising objectives

The denoising mechanisms span discrete diffusion, masked denoising, continuous latent diffusion, flow matching, and codebook-steered consistency sampling. BAD uses permutation-based corruption over discrete motion tokens. Rather than masking in a fixed left-to-right order or replacing tokens with a single absorbing state, it masks a subset of tokens using multiple mask tokens from a Maskbook and reconstructs them under a randomized autoregressive order. Its main objective couples masked-token prediction with observed-token modeling:

$\beta$0

with masked-token dependencies explicitly preserved through the permutation-induced causal structure (Hosseyni et al., 2024).

VidLaDA uses masked discrete diffusion over response tokens. At diffusion time $\beta$1, each response token is independently replaced by $\beta$2 with probability $\beta$3, while video tokens $\beta$4 and prompt tokens $\beta$5 remain visible. The denoising objective is

$\beta$6

where $\beta$7. The paper contrasts this with standard autoregressive decoding, for which the visibility frequency of token $\beta$8 is $\beta$9, producing an early-receptive-field bias and an information gap relative to bidirectional decoding (He et al., 25 Jan 2026).

LaDiR formulates latent reasoning as continuous-time generative flow. A clean latent block $$\mathbf{Z}^{(b)}=\{z^{(b)}_1,\ldots,z^{(b)}_{L_b}\}$$0 is interpolated with Gaussian noise $$\mathbf{Z}^{(b)}=\{z^{(b)}_1,\ldots,z^{(b)}_{L_b}\}$$1 according to

$$\mathbf{Z}^{(b)}=\{z^{(b)}_1,\ldots,z^{(b)}_{L_b}\}$$2

and the network is trained to match the target velocity field $$\mathbf{Z}^{(b)}=\{z^{(b)}_1,\ldots,z^{(b)}_{L_b}\}$$3 by minimizing

$$\mathbf{Z}^{(b)}=\{z^{(b)}_1,\ldots,z^{(b)}_{L_b}\}$$4

The same paper also presents the $$\mathbf{Z}^{(b)}=\{z^{(b)}_1,\ldots,z^{(b)}_{L_b}\}$$5-VAE objective

$$\mathbf{Z}^{(b)}=\{z^{(b)}_1,\ldots,z^{(b)}_{L_b}\}$$6

which defines the structured latent reasoning space before diffusion is applied (Kang et al., 6 Oct 2025).

BARD adopts a DDPM-style latent noising process for physical-field evolution. The target latent $$\mathbf{Z}^{(b)}=\{z^{(b)}_1,\ldots,z^{(b)}_{L_b}\}$$7 is corrupted as

$$\mathbf{Z}^{(b)}=\{z^{(b)}_1,\ldots,z^{(b)}_{L_b}\}$$8

and the denoiser is conditioned on neighboring latents and direction:

$$\mathbf{Z}^{(b)}=\{z^{(b)}_1,\ldots,z^{(b)}_{L_b}\}$$9

Training minimizes the conditional noise-prediction loss $512\times 512$0 (Scheinker, 28 Jun 2026).

ZeroGVC is distinct in that the transmitted object is the denoising trajectory itself. In each few-step denoising stage, the encoder computes the residual to the target latent and approximates it by a sparse combination of deterministically regenerated Gaussian codebook atoms. This yields a codebook-guided latent correction path that both encoder and decoder can reproduce from shared seeds, transmitted atom indices, and quantized coefficients (Gao et al., 21 Jun 2026).

4. Representative systems and empirical behavior

BAD is a text-to-motion system built on a simple VQ-VAE tokenizer with codebook size $512\times 512$1, code dimension $512\times 512$2, and downsampling rate $512\times 512$3, followed by an 18-layer transformer with hidden size 1024 and 16 attention heads. The first two layers are cross-attention layers, and text conditioning is supplied by a pre-trained CLIP model through a prepended sentence embedding and word-level cross-attention. On HumanML3D, BAD reports FID $512\times 512$4 for CBS and $512\times 512$5 for OAAS, compared with MMM at $512\times 512$6; on KIT-ML, MMM reports FID $512\times 512$7, while BAD reports $512\times 512$8 for CBS and $512\times 512$9 for OAAS. The paper also reports strong performance on temporal inpainting, outpainting, prefix prediction, and suffix completion, particularly in FID (Hosseyni et al., 2024).

LaDiR is a post-training reasoning framework in which each chain-of-thought sentence becomes one latent block, decoded back to text by a frozen pretrained LLM. Two robustness augmentations are used during VAE training: latent Gaussian noise and input token substitution with probability $16\times 16\times 4$0. On seven mathematical reasoning benchmarks, the paper reports average pass@1 of $16\times 16\times 4$1, with per-benchmark scores of MATH $16\times 16\times 4$2, GSM8K $16\times 16\times 4$3, Fresh-Gaokao-Math-2023 $16\times 16\times 4$4, DeepMind-Math $16\times 16\times 4$5, College-Math $16\times 16\times 4$6, OlympiaBench-Math $16\times 16\times 4$7, and TheoremQA $16\times 16\times 4$8. On Countdown planning, LaDiR reports CD-4 pass@1 $16\times 16\times 4$9, pass@100 $256\times$0, diversity $256\times$1, and CD-5 pass@1 $256\times$2, pass@100 $256\times$3, diversity $256\times$4. An ablation on Countdown-4 further reports $256\times$5 for the flow-matching $256\times$6-parameterization, compared with $256\times$7 for $256\times$8-prediction, $256\times$9 for $V,P,R_t$0-prediction, $V,P,R_t$1 for $V,P,R_t$2-prediction, and $V,P,R_t$3 for MSE (Kang et al., 6 Oct 2025).

VidLaDA keeps a standard video-LMM front end but replaces the language backbone with a masked diffusion transformer initialized from LLaDA-V or LLaDA-8B. Its vision encoder is SigLIP2-SO400M, and visual embeddings are spatially pooled by reshaping into a 2D grid and applying $V,P,R_t$4 bilinear interpolation, which reduces token length by $V,P,R_t$5. The model is evaluated as a video-language understanding system rather than a generative simulator, and the paper reports that it outperforms diffusion baselines and rivals autoregressive systems such as Qwen2.5-VL and LLaVA-Video. It also reports intra-frame robustness to moving salient patches to later positions, inter-frame robustness across event positions and frame sparsity, and causal-inference examples in which long-range temporal links are better captured. For inference, MARS-Cache delivers over $V,P,R_t$6 speedup, with roughly $V,P,R_t$7–$V,P,R_t$8 throughput gains in chain-of-thought inference and about $V,P,R_t$9–$$\text{att}_{\text{hyb}}=\text{att}_{\text{bi}}+\text{att}_{\text{per}},$$0 TPS over Dual-Cache while preserving reasoning accuracy (He et al., 25 Jan 2026).

BARD is a latent diffusion surrogate for 2D MHD. The dataset consists of synthetic trajectories generated from randomized initial density, pressure, and velocity, with the magnetic field initialized from a random vector potential built as a mixture of 100 Gaussians. Each simulation has resolution $$\text{att}_{\text{hyb}}=\text{att}_{\text{bi}}+\text{att}_{\text{per}},$$1, 100 time steps over 1 second, and the reported split is 500 training trajectories and 50 test trajectories. The model jointly predicts density, pressure, velocity components, and magnetic field components; VAE reconstructions are reported to visually match original fields well, and predicted power spectra for velocity and magnetic fields agree well with ground truth up to high wavenumbers. The paper reports that rollout error grows with autoregressive depth, that self-supervised consistency tracks this growth, and that UNet converges faster while ViT can match or slightly exceed performance with more data and training (Scheinker, 28 Jun 2026).

ZeroGVC is a training-free generative codec that uses Causal Forcing or Causal Forcing++ as the autoregressive diffusion prior, OneDC as the I-frame codec, and WanVAE for latent mapping. The reported evaluation covers HEVC Classes B, C, D, E, UVG, and MCL-JCV using the first 96 frames of each sequence, intra period 32, low-delay configuration, and metrics LPIPS, DISTS, FID, and DOVER, with MS-SSIM in supplementary material. Implementation choices include codebook size $$\text{att}_{\text{hyb}}=\text{att}_{\text{bi}}+\text{att}_{\text{per}},$$2, denoising steps $$\text{att}_{\text{hyb}}=\text{att}_{\text{bi}}+\text{att}_{\text{per}},$$3, atom count $$\text{att}_{\text{hyb}}=\text{att}_{\text{bi}}+\text{att}_{\text{per}},$$4 varied from 64 to 384, and tiled latent processing for high-resolution videos. The paper reports strong perceptual reconstruction quality at ultra-low bitrates without additional training, with example runtimes of about 96 ms encode / 124 ms decode per frame for CF++ with $$\text{att}_{\text{hyb}}=\text{att}_{\text{bi}}+\text{att}_{\text{per}},$$5, 144 ms / 163 ms for CF++ with $$\text{att}_{\text{hyb}}=\text{att}_{\text{bi}}+\text{att}_{\text{per}},$$6, and 195 ms / 202 ms for CF with $$\text{att}_{\text{hyb}}=\text{att}_{\text{bi}}+\text{att}_{\text{per}},$$7. The optional bidirectional reference mode is reported to improve perceptual quality without extra bitrate cost (Gao et al., 21 Jun 2026).

5. Inference-time behavior: parallelism, caching, adaptive compute, and self-supervision

One of the main motivations for these models is to escape the rigidity of one-pass left-to-right decoding. BAD performs iterative decoding under a cosine mask schedule,

$$\text{att}_{\text{hyb}}=\text{att}_{\text{bi}}+\text{att}_{\text{per}},$$8

and offers two generation strategies: Order-Agnostic Autoregressive Sampling, which follows a sampled permutation, and Confidence-Based Sampling, which retains high-confidence predictions while re-masking low-confidence ones for refinement. The empirical distinction is that OAAS tends to improve retrieval alignment and MM-Dist, whereas CBS improves FID most strongly (Hosseyni et al., 2024).

LaDiR makes test-time compute an explicit control variable. Inference begins from Gaussian noise, denoises latent reasoning blocks iteratively, and continues until the model emits the special token <SOA>, after which answer decoding proceeds. The paper reports that increasing denoising steps from 5 to 10 gives a large gain, 10 to 30 adds more, and 50 yields the best results among those tested. It also introduces diversity-guided sampling through increased initial noise scale and a repulsion force over batch latents, so multiple reasoning trajectories can be explored in parallel rather than by independent autoregressive sampling (Kang et al., 6 Oct 2025).

VidLaDA exposes the opposite side of bidirectional refinement: computational cost. Vanilla diffusion-language decoding recomputes full bidirectional attention over the multimodal sequence at every step, so MARS-Cache introduces asynchronous cache refreshing, frame-wise chunk attention, and adaptive anchor token searching. Refresh rates depend on layer group and modality, with deeper layers refreshed more frequently than shallow ones and text refreshed more often than vision. Frame-wise chunk attention restricts visual refresh to local neighborhoods such as $$\text{att}_{\text{hyb}}=\text{att}_{\text{bi}}+\text{att}_{\text{per}},$$9, while anchor tokens preserve global connectivity. Additional engineering choices include anchor search only at the first decoding step, once per layer group, and optional relocation of anchor tokens to the beginning of each frame for memory locality (He et al., 25 Jan 2026).

BARD uses bidirectionality not only for generation but also for calibration. If a forward rollout operator $c_d=+1$0 followed by a backward rollout operator $c_d=+1$1 were exact, their composition would be the identity. The paper therefore defines a measurable self-supervised consistency error,

$c_d=+1$2

and proposes trusting rollouts only while $c_d=+1$3. It also notes that stochastic diffusion sampling permits either deterministic probability-flow sampling or sample-averaged consistency estimates, with the spread across samples serving as an additional uncertainty estimate. The same framework is extended to sparse-diagnostic correction through bounded extremum seeking, allowing partial measurements such as density alone to re-anchor the latent rollout (Scheinker, 28 Jun 2026).

ZeroGVC addresses a different deployment problem: low-delay video coding. Because the decoder reproduces the same few-step denoising path from shared codebook seeds and compact atom/coefficients, the method is naturally compatible with streaming AR video generation. Its optional bidirectional reference mode uses the next I-frame latent as auxiliary context when available; because the next I-frame is already transmitted for the next GOP, this adds no bitrate. The paper further notes that RoPE requires a non-naïve cache treatment: future reference keys are stored before rotation and then re-rotated using their true future frame indices to preserve temporal geometry (Gao et al., 21 Jun 2026).

6. Conceptual distinctions, limitations, and unresolved boundaries

A persistent source of confusion is that the phrase joins three terms—bidirectional, autoregressive, and latent diffusion—that are each realized differently across papers. VidLaDA demonstrates that a system may be “autoregressive latent diffusion-style” in spirit while explicitly rejecting causal-mask autoregression: its central claim is that video understanding suffers from asymmetric receptive field, visibility-frequency imbalance, positional bias, and loss of global spatiotemporal context under standard AR masking, and that fully bidirectional masked denoising remedies this (He et al., 25 Jan 2026). BAD shows the converse case: a model can remain genuinely autoregressive over masked-token dependencies, but in a random permutation order rather than a fixed left-to-right one (Hosseyni et al., 2024). LaDiR occupies an intermediate position, with local bidirectionality and global causal ordering (Kang et al., 6 Oct 2025). ZeroGVC remains fundamentally causal despite its optional future reference, so it is better characterized as a causal autoregressive latent diffusion codec with a bidirectional extension rather than as a fully bidirectional generative model (Gao et al., 21 Jun 2026).

The representation layer is also non-uniform. BAD explicitly differs from conventional latent diffusion because it operates over discrete VQ-VAE motion tokens rather than continuous Gaussian latents, making it closer to discrete diffusion or denoising over tokens (Hosseyni et al., 2024). LaDiR and BARD are standard latent-diffusion cases in the stronger sense of continuous VAE latents (Kang et al., 6 Oct 2025, Scheinker, 28 Jun 2026). VidLaDA uses a diffusion LLM over multimodal tokens, and its “latent” character is tied to tokenized multimodal representations rather than to a separate continuous latent variable model (He et al., 25 Jan 2026). This diversity indicates that the field has not converged on a single ontological meaning of latent diffusion within bidirectional-autoregressive settings.

The limitations reported by the papers are correspondingly heterogeneous. BAD uses a simple VQ-VAE rather than an RVQ tokenizer and still relies on iterative decoding; sequence length handling is kept simple and broader modality extensions are left for future work (Hosseyni et al., 2024). LaDiR also depends on iterative decoding and presently uses a simple sentence-to-block setup, though its results suggest that adaptive test-time scaling is valuable (Kang et al., 6 Oct 2025). BARD is evaluated only on synthetic 2D MHD, with no hard physics constraints yet enforced; its cycle consistency is necessary rather than sufficient, because forward and backward errors could in principle cancel (Scheinker, 28 Jun 2026). VidLaDA’s full bidirectional attention creates an inference bottleneck that requires specialized caching and locality mechanisms (He et al., 25 Jan 2026). ZeroGVC remains sensitive to I-frame quality, and its bidirectional reference is usable only when the next I-frame is already available, even though it introduces no additional bitrate (Gao et al., 21 Jun 2026).

Taken together, these works define a broad methodological tendency rather than a finalized architecture. The shared intuition is that latent denoising permits iterative correction, while some form of bidirectional context counters the information bottlenecks of strictly causal generation. The main disagreement lies in where bidirectionality should enter: token attention, block structure, time reversal, or auxiliary references. That unresolved boundary is itself a defining feature of the current literature.