Masked Bit AutoRegressive Modeling (BAR)

• Masked Bit AutoRegressive Modeling (BAR) is a discrete generative approach that factors token prediction into bit-level steps using a masked prediction head.
• The framework leverages a causal transformer and iterative unmasking schedule to generate latent tokens more efficiently than traditional large-vocabulary softmax methods.
• BAR-Fixed and BAR-Learned variants adjust coordinate-wise masking schedules to optimize decoding orders, enhancing both image synthesis and tabular data modeling.

Searching arXiv for the cited BAR-related papers to ground the article in current records. Looking up (Yu et al., 9 Feb 2026) and (Garg et al., 24 Nov 2025). Masked Bit AutoRegressive modeling (BAR) denotes a family of discrete generative modeling formulations that combine autoregressive factorization with masked prediction. In visual generation, BAR is defined as a scalable framework in which a causal transformer predicts discrete latent tokens by progressively generating their constituent bits with a masked bit modeling head (Yu et al., 9 Feb 2026). In related work on masked diffusion over discrete tokens, BAR-Fixed and BAR-Learned denote models whose continuous-time variational objective decomposes exactly into a weighted sum of autoregressive losses over decoding orders, with the weights determined by a multivariate masking schedule (Garg et al., 24 Nov 2025). These usages are distinct but technically adjacent: both place masking inside an autoregressive view of discrete generation.

1. Terminology and scope

The acronym BAR appears in two related senses in the cited literature. One sense is the image-generation framework introduced in "Autoregressive Image Generation with Masked Bit Modeling" (Yu et al., 9 Feb 2026). The other is the learned-order autoregressive interpretation of masked diffusion on discrete tokens, instantiated as BAR-Fixed and BAR-Learned in tabular synthesis experiments in "Masked Diffusion Models are Secretly Learned-Order Autoregressive Models" (Garg et al., 24 Nov 2025).

Usage	Domain	Defining mechanism
BAR	Discrete latent image generation	Predict each latent token through its constituent bits using a masked bit modeling head
BAR-Fixed / BAR-Learned	Discrete-token masked diffusion and tabular synthesis	Learn or fix coordinate-wise masking schedules that induce weights over autoregressive orders

The first formulation is operational: it specifies a tokenization, a bit-level factorization, a transformer backbone, and a multi-round unmasking procedure for sampling. The second is theoretical and algorithmic: it shows that masked diffusion with multivariate schedules is exactly a learned-order autoregressive model, and then uses that result to learn favorable decoding orders in practice. A plausible implication is that BAR should be understood less as a single architecture than as a design pattern in which masked reconstruction is used to control autoregressive generation order, granularity, or both.

2. Bit-level autoregressive formulation

In the image-generation formulation, BAR treats generation as an autoregressive model over discrete latent tokens, but does not predict each token index with one large softmax. Instead, each token is decomposed into bits and generated bit-by-bit (Yu et al., 9 Feb 2026). Let $N$ be the number of latent tokens, $C$ the codebook size, $k=\lceil \log_2 C\rceil$ the number of bits per token, $x_i\in\{0,\dots,C-1\}$ the $i$-th latent token, and $\mathbf b_i=(b_{i,1},\dots,b_{i,k})\in\{0,1\}^k$ its binary expansion. The standard token-level factorization is

$p(x_{1:N})=\prod_{i=1}^N p(x_i\mid x_{<i}),$

and BAR further factorizes each term as

$$p(x_i\mid x_{<i})=p(b_{i,1:k}\mid x_{<i})=\prod_{j=1}^k p(b_{i,j}\mid b_{i,<j},x_{<i}).$$

Flattening the entire latent stream into a single bit sequence $\mathbf b=(b_1,\dots,b_{Nk})$ yields

$p(\mathbf b)=\prod_{t=1}^{Nk} p(b_t\mid b_{<t}).$

Training uses masked-bit modeling. A random subset of bits in each $C$0 is replaced by a mask symbol, and the model reconstructs the full bit-vector. If $C$1 is a masking-ratio sampler and $C$2, then the head predicts

$C$3

where $C$4 is the transformer hidden state conditioned on $C$5 (Yu et al., 9 Feb 2026).

At inference time, prediction is performed through a small fixed unmasking schedule $C$6. Starting from $C$7, the model repeatedly computes logits from the masked bit string, fills the next $C$8 masked positions by sampling or argmax, and finally decodes the completed bit string back to the token index $C$9. This makes token prediction iterative at the bit level while remaining autoregressive at the token level.

3. Architecture, objective, and computational profile

BAR’s backbone is a causal transformer similar to RAR or a GPT-style ViT, with depth $k=\lceil \log_2 C\rceil$0 equal to 24 for BAR-B and 32 for BAR-L, width $k=\lceil \log_2 C\rceil$1 equal to 768 or 1280, and components including attention heads, RoPE positional embeddings, SwiGLU feed-forward blocks, RMSNorm, and repeated class conditioning (Yu et al., 9 Feb 2026). After causal self-attention over previous tokens $k=\lceil \log_2 C\rceil$2, the hidden vector $k=\lceil \log_2 C\rceil$3 is passed to the Masked Bit Modeling (MBM) head $k=\lceil \log_2 C\rceil$4.

The MBM head is a lightweight 3-layer SwiGLU MLP with adaptive layer norms. If the partially masked bit-string $k=\lceil \log_2 C\rceil$5 is embedded into a $k=\lceil \log_2 C\rceil$6 tensor and a learned mask-token embedding is used for masked positions, then

$k=\lceil \log_2 C\rceil$7

Each $k=\lceil \log_2 C\rceil$8 is a logit for bit $k=\lceil \log_2 C\rceil$9. Because $x_i\in\{0,\dots,C-1\}$0, the head scales as $x_i\in\{0,\dots,C-1\}$1 rather than $x_i\in\{0,\dots,C-1\}$2, which is one of the central computational motivations of the method (Yu et al., 9 Feb 2026).

The training objective is the bit-wise cross-entropy over all token positions and bit positions,

$x_i\in\{0,\dots,C-1\}$3

supplemented by standard weight decay, optional dropout on class embeddings, and classifier-free guidance preparation through class-label dropping (Yu et al., 9 Feb 2026).

Sampling uses KV-cache reuse in the transformer. Each new token requires one transformer forward pass to obtain $x_i\in\{0,\dots,C-1\}$4 and then $x_i\in\{0,\dots,C-1\}$5 MBM-head calls, with 4–6 rounds described as typical. Since the MBM head is only $x_i\in\{0,\dots,C-1\}$6 per token and $x_i\in\{0,\dots,C-1\}$7 diffusion’s 1,000 steps, the paper states that BAR runs tens of times faster than diffusion (Yu et al., 9 Feb 2026). This suggests that BAR’s efficiency is achieved not by abandoning autoregression, but by moving the high-cardinality output layer into a low-dimensional masked-bit subproblem.

4. Codebook scaling and compression ratio

A central claim of the image-generation BAR paper is that the performance gap between discrete and continuous pipelines arises primarily from the total number of bits allocated in the latent space, that is, the compression ratio, rather than from an intrinsic inferiority of discrete tokenizers (Yu et al., 9 Feb 2026). BAR is presented as a mechanism for exploiting very large codebooks without incurring the standard training-cost or performance penalties associated with large-vocabulary token prediction.

The latent capacity of the discrete representation is measured by the bit budget

$x_i\in\{0,\dots,C-1\}$8

where $x_i\in\{0,\dots,C-1\}$9 is the spatial downsampling factor. Pixel-level bits are

$i$0

and the compression ratio is

$i$1

As $i$2 grows, $i$3 grows linearly, so the discrete bottleneck becomes less lossy (Yu et al., 9 Feb 2026).

The reported sweep ranges from $i$4 to $i$5, and reconstruction FID is said to decrease in lock-step as bits increase, eventually matching or surpassing continuous VAEs at similar bit budget (Yu et al., 9 Feb 2026). The technical significance is that BAR reframes large codebooks from a softmax-scaling problem into a bit-sequence prediction problem. A plausible implication is that codebook scaling becomes a first-class optimization axis rather than an impractical edge case.

5. Learned-order autoregression from masked diffusion

The learned-order formulation begins with masked diffusion over a discrete token sequence $i$6, where each coordinate $i$7 has its own strictly decreasing masking schedule $i$8 satisfying $i$9 and $\mathbf b_i=(b_{i,1},\dots,b_{i,k})\in\{0,1\}^k$0 (Garg et al., 24 Nov 2025). In the forward process, coordinate $\mathbf b_i=(b_{i,1},\dots,b_{i,k})\in\{0,1\}^k$1 is replaced by a mask token at a random time $\mathbf b_i=(b_{i,1},\dots,b_{i,k})\in\{0,1\}^k$2 whose CDF is

$\mathbf b_i=(b_{i,1},\dots,b_{i,k})\in\{0,1\}^k$3

and the joint density is

$\mathbf b_i=(b_{i,1},\dots,b_{i,k})\in\{0,1\}^k$4

Using the Ho–King reparameterization or the continuous-time limit of the discrete ELBO, the variational bound on $\mathbf b_i=(b_{i,1},\dots,b_{i,k})\in\{0,1\}^k$5 becomes the denoising loss

$\mathbf b_i=(b_{i,1},\dots,b_{i,k})\in\{0,1\}^k$6

If $\mathbf b_i=(b_{i,1},\dots,b_{i,k})\in\{0,1\}^k$7 is the permutation obtained by sorting the transition times,

$\mathbf b_i=(b_{i,1},\dots,b_{i,k})\in\{0,1\}^k$8

then grouping trajectories by their implied order yields an exact decomposition of the ELBO: $\mathbf b_i=(b_{i,1},\dots,b_{i,k})\in\{0,1\}^k$9 with

$p(x_{1:N})=\prod_{i=1}^N p(x_i\mid x_{<i}),$0

The paper states this as a proposition: the masked-diffusion objective is precisely a weighted autoregressive loss over orders (Garg et al., 24 Nov 2025).

This result has two immediate consequences. First, if all $p(x_{1:N})=\prod_{i=1}^N p(x_i\mid x_{<i}),$1 are identical, then symmetry gives $p(x_{1:N})=\prod_{i=1}^N p(x_i\mid x_{<i}),$2, recovering the any-order autoregressive interpretation with uniform mixing over orders. Second, if the schedules differ across coordinates, permutation invariance is broken, some orders receive larger weight, and the model specializes to a favorable decoding order during training (Garg et al., 24 Nov 2025). The parameterization

$p(x_{1:N})=\prod_{i=1}^N p(x_i\mid x_{<i}),$3

is given as an example of a learnable schedule family. In tabular experiments, the learned version is trained by learning each $p(x_{1:N})=\prod_{i=1}^N p(x_i\mid x_{<i}),$4 via REINFORCE/RLOO (Garg et al., 24 Nov 2025).

6. Empirical findings and interpretive issues

On ImageNet-256$p(x_{1:N})=\prod_{i=1}^N p(x_i\mid x_{<i}),$5256 with classifier-free guidance, BAR-L with 1.1B parameters achieves $p(x_{1:N})=\prod_{i=1}^N p(x_i\mid x_{<i}),$6, while the paper reports comparison numbers of 1.13 for RAE with 839M parameters, 1.65 for FlowAR, 1.55 for MAR, 1.24 for xAR, and approximately 1.63–1.26 for diffusion baselines such as DiMR and DDT (Yu et al., 9 Feb 2026). The smaller BAR-B with 415M parameters achieves $p(x_{1:N})=\prod_{i=1}^N p(x_i\mid x_{<i}),$7, matching RAE, and samples at 24 images per second on a single H200, described as a 3.7$p(x_{1:N})=\prod_{i=1}^N p(x_i\mid x_{<i}),$8 speedup. BAR-B/2 is reported at 150 images per second, with a slight FID increase but still better quality than most one-step diffusion methods (Yu et al., 9 Feb 2026).

In the learned-order masked-diffusion setting, two models with approximately 86K parameters are instantiated: BAR-Fixed with $p(x_{1:N})=\prod_{i=1}^N p(x_i\mid x_{<i}),$9, and BAR-Learned with $$p(x_i\mid x_{<i})=p(b_{i,1:k}\mid x_{<i})=\prod_{j=1}^k p(b_{i,j}\mid b_{i,<j},x_{<i}).$$0 (Garg et al., 24 Nov 2025). On six standard tabular datasets—Adult, Default, Beijing, Shoppers, Magic, and News—the validation negative log-likelihood is stated to be uniformly lower for BAR-Learned. The paper further reports that learned schedules concentrate masking later on “hard” columns and earlier on “easy” ones, and that data-fidelity metrics such as $$p(x_i\mid x_{<i})=p(b_{i,1:k}\mid x_{<i})=\prod_{j=1}^k p(b_{i,j}\mid b_{i,<j},x_{<i}).$$1-precision, $$p(x_i\mid x_{<i})=p(b_{i,1:k}\mid x_{<i})=\prod_{j=1}^k p(b_{i,j}\mid b_{i,<j},x_{<i}).$$2-recall, and Trend match or exceed BAR-Fixed while remaining competitive with much larger Gaussian-diffusion baselines. The Adult example gives 13.1524 for BAR-Learned and 13.1861 for BAR-Fixed (Garg et al., 24 Nov 2025).

Two recurrent misconceptions are addressed directly by these results. One concerns discrete latent models: the image-generation paper argues against the view that discrete tokenizers are intrinsically inferior, attributing the gap primarily to latent bit allocation and compression ratio (Yu et al., 9 Feb 2026). The other concerns masked diffusion: the learned-order paper shows that apparent order-agnosticism is a consequence of using a univariate schedule, not a structural property of the objective itself (Garg et al., 24 Nov 2025). Taken together, these points suggest that BAR is best interpreted as a reparameterization strategy for discrete generation in which performance depends critically on how information capacity and generation order are exposed to the model.