Autoregressive Boltzmann Generators

• Autoregressive Boltzmann Generators (ArBG) are autoregressive formulations that factorize Boltzmann distributions into normalized conditional probabilities, enabling exact log-likelihood computation.
• In molecular systems, ArBG bypasses constraints of flow-based methods by using decoder-only causal transformers, facilitating efficient ancestral sampling and reweighting without complex invertibility requirements.
• For pairwise interacting binary spins, ArBG provides an exact physics-aware neural realization that mirrors Hamiltonian couplings, yielding improved free energy and observable estimations.

Searching arXiv for the cited ArBG papers to ground the article in current sources. Autoregressive Boltzmann Generators (ArBG) are autoregressive formulations of Boltzmann distributions in which normalized conditional factors provide tractable log-likelihoods and ancestral sampling. In current arXiv usage, the term covers two technically distinct programs: a diffeomorphism-free, likelihood-tractable alternative to flow-based Boltzmann Generators for sampling molecular equilibrium ensembles, and exact autoregressive realizations of the Boltzmann distribution for pairwise interacting binary spins whose first-layer parameters are in one-to-one correspondence with the Hamiltonian’s couplings and fields (Rehman et al., 25 Jun 2026, Biazzo, 2023).

1. Boltzmann factorization and the autoregressive viewpoint

For molecular systems, the target equilibrium density is written as

$$\pi(x)=Z^{-1}\exp(-\beta U(x)), \qquad Z=\int \exp(-\beta U(x))\,dx,$$

with Cartesian coordinates $x \in \mathbb{R}^{3n}$, potential energy $U(x)$, and $\beta = 1/(k_B T)$. The ArBG proposal is an autoregressive density

$$q_\theta(x)=\prod_{i=1}^d q_\theta(x_i \mid x_{<i}), \qquad \log q_\theta(x)=\sum_{i=1}^d \log q_\theta(x_i \mid x_{<i}),$$

where $d=3n$ after flattening coordinates into scalar components. This factorization yields exact, fast log-likelihoods and supports self-normalized importance sampling (SNIS), with weights proportional to $\exp(-\beta U(x))/q_\theta(x)$, so that the unknown partition function cancels in the normalized estimator (Rehman et al., 25 Jun 2026).

For pairwise interacting binary spins, the same chain-rule principle is exact for any fixed ordering:

$$p(\boldsymbol{s})=\prod_{i=1}^N p(s_i \mid s_{<i}),$$

where

$$H(\boldsymbol{s})=-\sum_{i<j}J_{ij}s_i s_j-\sum_{i=1}^N h_i s_i, \qquad p(\boldsymbol{s})=\frac{1}{Z}e^{-\beta H(\boldsymbol{s})}.$$

The 2023 formulation derives exact conditional probabilities for general pairwise Ising Hamiltonians and then expresses each conditional as a neural module with a strictly causal first layer, residual structure, and recurrent updates (Biazzo, 2023).

The shared idea is therefore not a single architecture but a probabilistic program: represent the Boltzmann law directly through normalized autoregressive conditionals rather than through a global transport map. This suggests a unifying contrast with approaches that obtain tractable likelihoods only through invertibility constraints or global partition-function manipulations.

2. Molecular ArBGs as a departure from flow-based Boltzmann Generators

In molecular equilibrium sampling, Boltzmann Generators address the difficulty that the equilibrium distribution is highly multimodal and molecular dynamics spends most of its time in local vibrations instead of crossing large energy barriers. The mainstream BG literature has relied on normalizing flows, but the 2026 ArBG work argues that discrete-time flows are constrained to be bijections, typically with a simple prior, hence are homeomorphisms, while continuous-time flows require an ODE solve of an augmented system and expensive divergence estimation for likelihood evaluation. ArBG departs from that paradigm by directly modeling the target density via local conditionals in data space, avoiding any learned diffeomorphism, any Jacobian determinant, and any ODE solver (Rehman et al., 25 Jun 2026).

The construction flattens $x \in \mathbb{R}^{3n}$ into $x \in \mathbb{R}^{3n}$0 scalar coordinates ordered residue-by-residue, with sidechains placed after the residue backbone atoms. A decoder-only causal transformer parameterizes $x \in \mathbb{R}^{3n}$1. Under causal masking, one transformer forward pass yields all conditionals. Complexity scales linearly in the number of coordinates in terms of conditional evaluations, while the likelihood remains exact because the model is normalized one conditional at a time.

Three conditional parameterizations are studied. The first uses discretized mixture density networks following PixelCNN++, with mixture-of-logistics or Gaussian-mixture conditional masses computed by CDF differences over uniform bins. The second, and empirically most stable, is the uniform-bin parameterization:

$x \in \mathbb{R}^{3n}$2

with dequantization at inference time. Its log-likelihood differs from a discrete cross-entropy by a constant $x \in \mathbb{R}^{3n}$3. The paper gives a corresponding irreducible error decomposition:

$x \in \mathbb{R}^{3n}$4

The discrete factors can match the true bin masses exactly, and the remaining error is the non-uniformity of the true density inside each bin. Increasing $x \in \mathbb{R}^{3n}$5 reduces this term monotonically.

The significance of this formulation is explicit in the paper’s comparison with flow-based BGs. Discrete flows preserve topology and struggle to represent disconnected supports or “holes,” whereas ArBG does not learn a transport map from a simple prior at all. Continuous-time flows relax architectural invertibility constraints but make SNIS costly because likelihoods require divergence integration along trajectories. ArBG’s exact likelihoods “in one pass” are therefore central not merely as an implementation convenience but as the mechanism that makes reweighting computationally practical.

3. Training, inference, and architectural scaling in molecular systems

Training proceeds by maximum likelihood on discretized Cartesian coordinates. Reference equilibrium samples are centered and standardized, each coordinate is discretized to $x \in \mathbb{R}^{3n}$6 uniform bins, and teacher forcing minimizes

$x \in \mathbb{R}^{3n}$7

For MDN variants, the objective is the discretized mixture likelihood based on CDF differences. No invertibility constraint and no Jacobian term enter the optimization (Rehman et al., 25 Jun 2026).

Inference uses sequential ancestral decoding. For the uniform-bin variant, one initializes an empty structure, optionally clamps coordinates or conditions, then for each coordinate computes logits over bins, samples a bin with optional temperature scaling, dequantizes within the bin, and accumulates

$x \in \mathbb{R}^{3n}$8

After evaluating $x \in \mathbb{R}^{3n}$9, the importance weight is

$U(x)$0

and SNIS yields

$U(x)$1

For free energies along a collective variable $U(x)$2, the weighted density estimate gives

$U(x)$3

Architecturally, the molecular ArBG uses a decoder-only causal transformer with causal self-attention, positional encoding such as rotary, RMSNorm, SwiGLU feed-forward blocks, FlashAttention, and KV caching. Per-token embeddings include atom type and residue type; the transferable model further conditions on residue position and sequence length. Optimization practice reported in the paper uses the Muon optimizer with cosine learning-rate schedule, and large models are trained in float32 because bf16 can be unstable on the largest configurations.

The inference temperature $U(x)$4 is an explicit control parameter. The reported pattern is that $U(x)$5 benefits larger peptides such as Chignolin, whereas $U(x)$6 can help on small systems such as tetrapeptides. This does not alter the exact likelihood formula of the trained model; rather, it changes the proposal distribution used for downstream reweighting. A plausible implication is that the autoregressive proposal can be tuned toward a sharper or broader proposal family while preserving unbiased observable estimation through SNIS.

4. Sequential interventions, benchmarks, and the Robin model

A distinctive feature of the molecular ArBG framework is its support for sequential inference-time interventions through Autoregressive Twisted SMC. At generation step $U(x)$7, the paper defines twisted intermediate densities

$U(x)$8

where twist functions are designed to favor physically plausible partial structures. The molecularly informed twist rescales partial trajectories using capped-residue partial energies:

$U(x)$9

A population of partial trajectories is propagated, reweighted, and systematically resampled at residue boundaries. The reported effect is early rejection of partials with high steric clashes or strained geometries, thereby reducing wasted full-molecule evaluations (Rehman et al., 25 Jun 2026).

The empirical program evaluates single peptides and transferable peptide families. The single-peptide systems are alanine dipeptide (ALDP), tri-alanine (AL3), alanine tetrapeptide (AL4), hexa-alanine (AL6), and the 10-residue decapeptide Chignolin (GYDPETGTWG). The transferable setting uses the ManyPeptidesMD dataset with families of 4-residue and 8-residue peptides, in which Robin is trained once and tested zero-shot on unseen sequences. The primary metrics are the energy 2–Wasserstein distance E-W$\beta = 1/(k_B T)$0, the torus 2–Wasserstein distance T-W$\beta = 1/(k_B T)$1 in torsional space, and TICA-W$\beta = 1/(k_B T)$2 on the leading two TICA components; ESS is explicitly not used as a primary metric.

Using $\beta = 1/(k_B T)$3 energy evaluations for SNIS, the reported Chignolin result is ArBG (tuned $\beta = 1/(k_B T)$4) with $\beta = 1/(k_B T)$5 and $\beta = 1/(k_B T)$6, compared with SBG at $\beta = 1/(k_B T)$7 and $\beta = 1/(k_B T)$8. For AL6, ArBG (tuned $\beta = 1/(k_B T)$9) achieves $$q_\theta(x)=\prod_{i=1}^d q_\theta(x_i \mid x_{<i}), \qquad \log q_\theta(x)=\sum_{i=1}^d \log q_\theta(x_i \mid x_{<i}),$$0. For AL4, ArBG (tuned $$q_\theta(x)=\prod_{i=1}^d q_\theta(x_i \mid x_{<i}), \qquad \log q_\theta(x)=\sum_{i=1}^d \log q_\theta(x_i \mid x_{<i}),$$1) achieves $$q_\theta(x)=\prod_{i=1}^d q_\theta(x_i \mid x_{<i}), \qquad \log q_\theta(x)=\sum_{i=1}^d \log q_\theta(x_i \mid x_{<i}),$$2, while without tuning ($$q_\theta(x)=\prod_{i=1}^d q_\theta(x_i \mid x_{<i}), \qquad \log q_\theta(x)=\sum_{i=1}^d \log q_\theta(x_i \mid x_{<i}),$$3) it attains $$q_\theta(x)=\prod_{i=1}^d q_\theta(x_i \mid x_{<i}), \qquad \log q_\theta(x)=\sum_{i=1}^d \log q_\theta(x_i \mid x_{<i}),$$4 with stronger T-W$$q_\theta(x)=\prod_{i=1}^d q_\theta(x_i \mid x_{<i}), \qquad \log q_\theta(x)=\sum_{i=1}^d \log q_\theta(x_i \mid x_{<i}),$$5 than flows. On AL3 and ALDP, the method is reported as state-of-the-art or competitive on E-W$$q_\theta(x)=\prod_{i=1}^d q_\theta(x_i \mid x_{<i}), \qquad \log q_\theta(x)=\sum_{i=1}^d \log q_\theta(x_i \mid x_{<i}),$$6, and competitive on T-W$$q_\theta(x)=\prod_{i=1}^d q_\theta(x_i \mid x_{<i}), \qquad \log q_\theta(x)=\sum_{i=1}^d \log q_\theta(x_i \mid x_{<i}),$$7.

Robin is a 132 million parameter transferable decoder-only transformer with 16 layers, 8 heads with head dimension 64, channel width 768, SwiGLU expansion 4, and $$q_\theta(x)=\prod_{i=1}^d q_\theta(x_i \mid x_{<i}), \qquad \log q_\theta(x)=\sum_{i=1}^d \log q_\theta(x_i \mid x_{<i}),$$8 bins. It conditions on atom type, residue type, position in sequence, and sequence length through a non-causal conditioning branch. Averaged across 30 unseen 8AA test systems at $$q_\theta(x)=\prod_{i=1}^d q_\theta(x_i \mid x_{<i}), \qquad \log q_\theta(x)=\sum_{i=1}^d \log q_\theta(x_i \mid x_{<i}),$$9 evaluations, Prose yields $d=3n$0 and Robin yields $d=3n$1, approximately a 61% reduction; Robin’s T-W$d=3n$2 is 1.902 versus 2.019 for Prose. At a smaller budget of $d=3n$3, Robin’s E-W$d=3n$4 is 4.251 versus 10.038 for Prose. Robin is slower per sample than Prose at the same batch size, for example about 29 versus 66 samples/s on 8AA, because it decodes scalar coordinates rather than atom tokens, but the paper reports better target T-W$d=3n$5 and TICA-W$d=3n$6 at fixed computational or energy-evaluation budgets.

These results delimit a specific meaning of “generator” in the molecular setting: not merely rapid ancestral decoding, but a proposal distribution whose exact likelihood enables physically corrected estimation after sampling. The emphasis on reweighted accuracy, rather than raw sample quality alone, is central to the benchmark design.

5. Exact spin-system ArBGs and physics-aware neural architectures

In the 2023 usage, ArBG refers to exact autoregressive realizations of the Boltzmann distribution of pairwise interacting binary spin systems. For a fixed ordering of spins, the exact conditional derived in the paper is

$d=3n$7

with

$d=3n$8

Equivalently,

$d=3n$9

where the effective field includes the explicit past-spin contribution and a correction from the future-spin partition functions (Biazzo, 2023).

The exact architecture, termed H2ARNN, makes the network–physics correspondence explicit. The first-layer outputs are

$\exp(-\beta U(x))/q_\theta(x)$0

and the conditional becomes

$\exp(-\beta U(x))/q_\theta(x)$1

with

$\exp(-\beta U(x))/q_\theta(x)$2

The first layer is strictly causal, with a triangular masked pattern; the coefficient from $\exp(-\beta U(x))/q_\theta(x)$3 to $\exp(-\beta U(x))/q_\theta(x)$4 is $\exp(-\beta U(x))/q_\theta(x)$5, the bias is $\exp(-\beta U(x))/q_\theta(x)$6, and the weights from $\exp(-\beta U(x))/q_\theta(x)$7 to $\exp(-\beta U(x))/q_\theta(x)$8 are $\exp(-\beta U(x))/q_\theta(x)$9. The recurrence

$$p(\boldsymbol{s})=\prod_{i=1}^N p(s_i \mid s_{<i}),$$0

supports $$p(\boldsymbol{s})=\prod_{i=1}^N p(s_i \mid s_{<i}),$$1 total complexity with caching across $$p(\boldsymbol{s})=\prod_{i=1}^N p(s_i \mid s_{<i}),$$2.

Because the exact second layer is exponentially large for arbitrary pairwise models, the paper develops tractable reductions for special systems. For the Curie–Weiss model,

$$p(\boldsymbol{s})=\prod_{i=1}^N p(s_i \mid s_{<i}),$$3

the finite-$$p(\boldsymbol{s})=\prod_{i=1}^N p(s_i \mid s_{<i}),$$4 conditional CWN is exact and uses $$p(\boldsymbol{s})=\prod_{i=1}^N p(s_i \mid s_{<i}),$$5 parameters across all conditionals, while the thermodynamic-limit approximation CW$$p(\boldsymbol{s})=\prod_{i=1}^N p(s_i \mid s_{<i}),$$6 uses $$p(\boldsymbol{s})=\prod_{i=1}^N p(s_i \mid s_{<i}),$$7 parameters total. For the Sherrington–Kirkpatrick model with couplings drawn i.i.d. from $$p(\boldsymbol{s})=\prod_{i=1}^N p(s_i \mid s_{<i}),$$8, the future partition functions are approximated via replica-theoretic RS and $$p(\boldsymbol{s})=\prod_{i=1}^N p(s_i \mid s_{<i}),$$9-RSB ansätze, yielding tractable nested nonlinear forms whose parameter count scales as $$H(\boldsymbol{s})=-\sum_{i<j}J_{ij}s_i s_j-\sum_{i=1}^N h_i s_i, \qquad p(\boldsymbol{s})=\frac{1}{Z}e^{-\beta H(\boldsymbol{s})}.$$0.

Training has two modes. To approximate a known Boltzmann law, the paper minimizes the variational free energy

$$H(\boldsymbol{s})=-\sum_{i<j}J_{ij}s_i s_j-\sum_{i=1}^N h_i s_i, \qquad p(\boldsymbol{s})=\frac{1}{Z}e^{-\beta H(\boldsymbol{s})}.$$1

which equals $$H(\boldsymbol{s})=-\sum_{i<j}J_{ij}s_i s_j-\sum_{i=1}^N h_i s_i, \qquad p(\boldsymbol{s})=\frac{1}{Z}e^{-\beta H(\boldsymbol{s})}.$$2 up to an additive constant independent of $$H(\boldsymbol{s})=-\sum_{i<j}J_{ij}s_i s_j-\sum_{i=1}^N h_i s_i, \qquad p(\boldsymbol{s})=\frac{1}{Z}e^{-\beta H(\boldsymbol{s})}.$$3. When fitting to data drawn from an unknown Boltzmann distribution, maximum likelihood is used:

$$H(\boldsymbol{s})=-\sum_{i<j}J_{ij}s_i s_j-\sum_{i=1}^N h_i s_i, \qquad p(\boldsymbol{s})=\frac{1}{Z}e^{-\beta H(\boldsymbol{s})}.$$4

The reported training setup uses ADAM with learning rate $$H(\boldsymbol{s})=-\sum_{i<j}J_{ij}s_i s_j-\sum_{i=1}^N h_i s_i, \qquad p(\boldsymbol{s})=\frac{1}{Z}e^{-\beta H(\boldsymbol{s})}.$$5, batch size 2000, up to 1000 epochs, and $$H(\boldsymbol{s})=-\sum_{i<j}J_{ij}s_i s_j-\sum_{i=1}^N h_i s_i, \qquad p(\boldsymbol{s})=\frac{1}{Z}e^{-\beta H(\boldsymbol{s})}.$$6-annealing from 0.1 to 2.0 in steps of 0.05.

Empirically, the paper reports that CWN matches the analytic free energy to within plotting resolution, CW$$H(\boldsymbol{s})=-\sum_{i<j}J_{ij}s_i s_j-\sum_{i=1}^N h_i s_i, \qquad p(\boldsymbol{s})=\frac{1}{Z}e^{-\beta H(\boldsymbol{s})}.$$7 significantly outperforms shallow baselines with only $$H(\boldsymbol{s})=-\sum_{i<j}J_{ij}s_i s_j-\sum_{i=1}^N h_i s_i, \qquad p(\boldsymbol{s})=\frac{1}{Z}e^{-\beta H(\boldsymbol{s})}.$$8 parameters, and SK$$H(\boldsymbol{s})=-\sum_{i<j}J_{ij}s_i s_j-\sum_{i=1}^N h_i s_i, \qquad p(\boldsymbol{s})=\frac{1}{Z}e^{-\beta H(\boldsymbol{s})}.$$9RSB yields lower free energies than MADE and RS/1L baselines, especially as $x \in \mathbb{R}^{3n}$0 grows. Overlap distributions in the glassy phase are better matched by SK$x \in \mathbb{R}^{3n}$1RSB, and in an inverse Ising test on SK, maximum-likelihood training of SK1RSB recovers first-layer weights strongly correlated with true $x \in \mathbb{R}^{3n}$2.

6. Comparisons, limitations, and open directions

The two ArBG lines share exact autoregressive normalization but differ in object, inductive bias, and tractability regime. In molecular systems, ArBG is presented as a likelihood-tractable alternative to flow-based Boltzmann Generators for continuous atomistic coordinates, with exact reweighting against a physical energy and with transformer scaling borrowed from modern LLM practice (Rehman et al., 25 Jun 2026). In spin systems, ArBG is an exact architecture-level realization of the Boltzmann distribution itself for pairwise binary models, in which the first-layer parameters are fixed by the Hamiltonian and tractability depends on model-specific reductions of future partition functions (Biazzo, 2023).

Several limitations are explicit. Molecular ArBGs have ordering sensitivity, an irreducible within-bin mismatch under uniform binning, no built-in $x \in \mathbb{R}^{3n}$3 equivariance or explicit quotienting by global translation, rotation, or permutation symmetries, and potential autoregressive bias in long-range dependence modeling. Larger systems with sharper features may require more bins, and model capacity and memory scale with $x \in \mathbb{R}^{3n}$4. In the spin setting, the exact H2ARNN is exponentially large for arbitrary pairwise models; practical ArBGs therefore rely on special solvable structures such as Curie–Weiss or on approximations such as RS and $x \in \mathbb{R}^{3n}$5-RSB for Sherrington–Kirkpatrick. Training at very low temperatures in highly frustrated systems remains hard, with potential mode collapse and rugged objectives.

The comparative claims are correspondingly specific. In the molecular setting, exact and cheap log-likelihoods make SNIS feasible without Jacobian determinants, ODE solves, or divergence estimation, and sequential generation permits substructure-level steering through twisted SMC. In the spin setting, exactness for any ordering follows from the chain rule, but the cost of exact future marginalization is the main bottleneck, so the contribution is the derivation of physics-aware neural reductions rather than a generic scalable recipe. This suggests that “ArBG” is best understood as a broader methodological family: direct autoregressive modeling of Boltzmann distributions with tractable conditionals, instantiated either as a scalable generative framework for molecular equilibrium ensembles or as an exact physics-informed neural construction for discrete interacting systems.