Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion

Abstract: Diffusion LLMs (DLMs) promise parallel, order-agnostic generation, but on standard benchmarks they have historically lagged behind autoregressive models in sample quality and diversity. Recent continuous flow and diffusion approaches over token embeddings have narrowed this gap, suggesting continuous state spaces are highly effective for language. In this work, we further close the autoregressive gap by modeling text as a continuous diffusion process over fixed-width binary bitstreams. Our approach represents semantic tokens as analog bit sequences and utilizes a matched-filter residual parameterization to isolate contextual learning from analytic independent-bit posteriors. Crucially, we adopt a stochastic sampler that applies Langevin-type corrections gated by the entropy-rate profile, automatically concentrating stochasticity in high-information regions while remaining nearly deterministic elsewhere. On the One Billion Word Benchmark (LM1B), our 130M-parameter bitstream model reaches a generative perplexity ($\GenPPL$) of $59.76$ at matched real-data entropy ($4.31$) using 256 neural function evaluations (NFEs), decisively outperforming prior DLM baselines and reaching the autoregressive reference. On OpenWebText (OWT), our stochastic sampler establishes a new continuous-DLM Pareto frontier, achieving $\GenPPL=27.06$ at an entropy of $5.26$ using $4\times$ fewer steps than previous 1024-NFE baselines. As an additional architectural benefit, bitstream diffusion removes the $\mathcal{O}(V)$ vocabulary scaling bottleneck shared by standard DLMs. By predicting $\mathcal{O}(\log V)$ bitwise logits via semantic bit-patching, our model yields a reduced memory footprint and higher throughput, demonstrating a scalable paradigm for language generation as vocabulary sizes grow.

• The paper introduces a continuous bitstream diffusion approach with analytic residual parameterization and entropy-gated stochastic sampling to closely match autoregressive language model quality.
• It demonstrates significant improvements in generative perplexity, sample entropy, and computational efficiency on benchmarks like LM1B and OpenWebText.
• The method eliminates the traditional O(V) logit bottleneck, achieving faster training and inference while offering controllable quality-diversity trade-offs.

Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion

Introduction and Motivation

This work addresses the persistent quality and entropy deficit of non-autoregressive language generation using diffusion models (DLMs) when compared to autoregressive (AR) transformers. Recent continuous-state DLMs have reduced this gap, but challenges with discretization, vocabulary-size scaling, sample diversity, and high-fidelity text generation remain. This paper introduces a method that models text as a continuous diffusion process over bitstreams rather than token embeddings, leveraging matched-filter analytic posteriors and entropy-adaptive stochastic sampling to further close the quality and diversity gap relative to strong AR and DLM baselines.

Methodology

The core innovation is a representation of text as sequences of fixed-width binary bitstreams, decoded into semantic tokens post-generation. Diffusion operates directly in this space using a variance-exploding (VE) Gaussian corruption model. The denoiser is trained to operate on noisy bit vectors, with the following principal components:

• Matched-Filter Residual Parameterization: The network is tasked solely with modeling contextual residuals, while the analytic independent-bit posterior under Gaussian corruption is computed in closed form. This optimally divides modeling capacity between local denoising (analytic, noncontextual) and long-range semantics (learned, contextual).
• Score-Matching Objective with Self-Conditioning: Training optimizes a weighted binary MSE loss (score matching), directly connecting model outputs to the continuous score required for sampling. Self-conditioning further stabilizes and enhances sample quality.
• Entropy-Rate Adaptive Noise Schedule: The noise allocation during both training and inference is statically adapted using an online estimate of local entropy production, ensuring that solver resolution and stochasticity concentrate in information-active regions.
• Entropy-Gated Stochastic Sampling: Sequence generation uses a deterministic probability-flow sampler augmented with full-band Langevin-like stochastic corrections ("churn") whose strength is gated by local entropy, ensuring the sampler remains conservative in low-information regions but injects appropriate diversity where uncertainty remains.
• Sequence Diffusion Transformer (SDT): The architecture patches $m = [\log_2V]$ consecutive analog bits into a single semantic token embedding, processed by a standard Transformer, fully decoupling context length $T$ from raw bitstream length $Tm$.

Empirical Results

Sample Quality and Diversity

On LM1B, the 130M parameter model achieves a generative perplexity of 59.76 at an entropy of 4.31 using just 256 neural function evaluations (NFEs), outperforming LangFlow's GenPPL of 92.24 by over 30 points and surpassing the AR baseline's 66.70, while strictly matching real text entropy. On OpenWebText (OWT), at GenPPL 27.06/entropy 5.26, it sets a new continuous DLM frontier and uses 4x fewer denoising steps than prior SOTA baselines. This demonstrates that the proposed bitstream diffusion with entropy-gated stochasticity achieves both AR-level perplexity and healthy token entropy, correcting over-contraction/entropy collapse endemic to prior DLMs.

Computational Efficiency

Bitstream diffusion removes the $O(V)$ vocabulary-wide logit output that bottlenecks almost all DLMs, reducing it to $O(\log V)$ per token. This translates directly into substantial memory and throughput gains:

• On LM1B: 1.64x peak memory and 2.3x throughput improvement in training; 10x lower memory and 2x faster inference generation.
• On OWT: 2.46x memory and 3.3x throughput in training; 19x lower memory and 2.65x throughput improvement in generation at batch sizes where token-level baselines already reach single-GPU limits.

These scaling gains become strictly more pronounced as vocabulary and context lengths increase.

Ablation and Analysis

• Removing the analytic matched-filter term causes a >22 GenPPL point drop, confirming the benefit of the split residual parameterization.
• Replacing score matching with cross-entropy worsens deterministic GenPPL, and stochasticity only regains quality at the cost of lower entropy—a poor quality-diversity tradeoff.
• A discrete DLM on bits (SEDD-style) is ablated and scores significantly worse than the token-level discrete baseline, underlining that continuous-state modeling is critical.
• Stochastic churn applied on an entropy-rate grid is both necessary and sufficient for matching AR quality while retaining sample entropy. Excessive stochasticity degrades sample quality, while too little yields over-contracted, under-diverse generations.

Theoretical and Practical Implications

This study demonstrates that the gap in language modeling generation quality between AR and DLMs is not a necessary consequence of continuity, but contingent on representation, residual parameterization, and careful, information-adaptive sampling. The results suggest that for large-scale and multilingual models, or for generative unification of heterogeneously tokenized modalities (text, images, audio), the logarithmic output scaling of bitstream diffusion could be instrumental, effectively sidestepping the prohibitive $O(V)$ bottleneck that currently hinders token-based DLMs.

Moreover, the entropy-gated stochastic process provides a test-time quality-diversity control knob, with the entropy-rate noise grid and full-band churn forming a predictor-corrector loop naturally aligned to where the reverse process needs the most stochasticity. This mechanism is validated both empirically and via continuous-time SDE analysis.

Limitations and Future Directions

Currently demonstrated at $\sim$130M scale on widely used language benchmarks, it remains an open question whether the same improvements hold at multi-billion or larger parameter counts, or for tasks requiring long-context reasoning or code generation. Inference with DLMs remains more expensive in wall-clock terms due to iterative denoising compared to greedy AR decoding. Advances in fast samplers, distillation techniques, or hybridization with AR/flow-based models may be required for practical deployment in latency-critical applications.

Conclusion

This study establishes strong empirical and architectural evidence that continuous bitstream diffusion, equipped with analytic residual parameterization and entropy-gated stochastic sampling, can close the AR/DLM gap on canonical benchmarks while also enabling strictly better vocabulary and context scaling. The approach yields Pareto-optimal quality-diversity tradeoffs, robust scaling behavior, and test-time controllability, opening a promising path for non-autoregressive LM development, especially in domains where scaling or heterogeneity make conventional vocabularized outputs infeasible.

Reference:

"Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion" (2605.07013)