Same Graph, Different Likelihoods: Calibration of Autoregressive Graph Generators via Permutation-Equivalent Encodings

Abstract: Autoregressive graph generators define likelihoods via a sequential construction process, but these likelihoods are only meaningful if they are consistent across all linearizations of the same graph. Segmented Eulerian Neighborhood Trails (SENT), a recent linearization method, converts graphs into sequences that can be perfectly decoded and efficiently processed by LLMs, but admit multiple equivalent linearizations of the same graph. We quantify violations in assigned negative log-likelihood (NLL) using the coefficient of variation across equivalent linearizations, which we call Linearization Uncertainty (LU). Training transformers under four linearization strategies on two datasets, we show that biased orderings achieve lower NLL on their native order but exhibit expected calibration error (ECE) two orders of magnitude higher under random permutation, indicating that these models have learned their training linearization rather than the underlying graph. On the molecular graph benchmark QM9, NLL for generated graphs is negatively correlated with molecular stability (AUC $=0.43$), while LU achieves AUC $=0.85$, suggesting that permutation-based evaluation provides a more reliable quality check for generated molecules. Code is available at https://github.com/lauritsf/linearization-uncertainty

• The paper introduces Linearization Uncertainty (LU), the coefficient of variation of negative log-likelihood across equivalent graph encodings, to measure whether autoregressive generators assign consistent probabilities to the same graph.
• The paper finds that biased training orderings produce lower native NLL but cause up to 10× NLL increases and major calibration failures under random permutations, with revisit tokens driving the largest errors.
• The paper shows that generation NLL can misidentify unstable molecules, while LU predicts stability with AUC 0.85 using only scalar likelihoods, making permutation-based evaluation a practical quality-control tool.

Autoregressive graph generators assign likelihoods to graphs through a sequential factorization, which presupposes that the ordering chosen for linearization does not affect the assigned probability. This paper examines that presupposition directly. Using Segmented Eulerian Neighborhood Trail (SENT) encodings [autograph2024], which are permutation-equivalent in the sense that every linearization of a graph decodes to the identical topology, the authors formalize a linearization-invariance requirement and introduce a diagnostic, Linearization Uncertainty (LU), that quantifies violations of it. The central finding is stark: autoregressive transformers trained on biased orderings learn the training linearization rather than the underlying graph, and the resulting likelihoods are not only inconsistent across orderings but actively misleading as quality signals.

Formalizing linearization-invariance

Let $\phi: \mathcal{S} \to \mathcal{G}$ denote the SENT decoding map. Because $\phi$ is many-to-one and all sequences in $\phi^{-1}(G)$ decode to the same graph, any consistent distribution over graphs must assign equal likelihood to every linearization of $G$. An autoregressive model $p_\theta(s) = \prod_t p_\theta(s_t \mid s_{<t})$ trained by teacher forcing does not enforce this constraint, and the constraint is non-trivial to satisfy: SENT sequence length depends on the traversal, so equivalent linearizations can differ in length and in their entire conditional factorization.

The paper's metric, Linearization Uncertainty, is the coefficient of variation of NLL across $K$ sampled linearizations of a graph:

$$\mathrm{LU}(G) = \frac{\sigma(\{\mathcal{L}(s_1), \dots, \mathcal{L}(s_K)\})}{\mu(\{\mathcal{L}(s_1), \dots, \mathcal{L}(s_K)\})}$$

where $\mathcal{L}(s) = -\log p_\theta(s)$. A key practical property is that LU requires only scalar NLL values from $K$ forward passes; it does not require access to full logits. LU is evaluated in two settings: under the model's native training strategy (internal consistency) and under random permutations (robustness to out-of-distribution orderings). The authors position LU as complementary to Expected Calibration Error (ECE): ECE measures whether per-token probabilities match empirical frequencies, while LU measures whether equivalent inputs receive equivalent likelihoods. A model that assigns different NLL to the same graph under different orderings does not define a well-structured distribution over graphs at all, let alone a calibrated one.

The paper also distinguishes Generation NLL, the likelihood of the model's own sampled trajectory, from mean permutation NLL computed over algorithmic re-linearizations of the decoded graph. A gap between these two indicates specialization to the training linearization.

Experimental design

The evaluation covers a data-scarce regime (the synthetic Planar dataset with $N = 128$ training graphs, and small QM9 subsets) and a data-rich regime (full QM9, $\phi$0k training molecules after explicit-hydrogen preprocessing). The backbone is a 12-layer Llama transformer following the SENT/autograph setup, with constrained decoding at inference time on QM9 that guarantees decodable graph topology without enforcing chemical validity. Four linearization strategies are compared, all producing valid SENT encodings of the same graphs but inducing different inductive biases: Random Order (maximally diverse; a fresh linearization every epoch), Min-Degree First (deferring hubs), Max-Degree First (front-loading hubs), and Anchor Expansion (anchored at the maximum-degree node, expanding leaf-first).

The robustness trade-off on QM9

In the data-rich regime, all strategies train stably, and biased strategies achieve lower native NLL per token with shorter sequences, reflecting traversal biases that minimize chordal back-references. Anchor Expansion attains the best native numbers (NLL/token $\phi$1, LU $\phi$2, ECE $\phi$3). Under randomized evaluation, however, these advantages collapse:

Strategy	NLL/token (Native → Random)	LU (Native → Random)	ECE (Native → Random)
Random	0.336 → 0.336	0.083 → 0.083	0.000 → 0.000
Min-Degree	0.250 → 1.564	0.049 → 0.383	0.001 → 0.171
Max-Degree	0.244 → 2.240	0.039 → 0.277	0.002 → 0.205
Anchor	0.222 → 2.246	0.027 → 0.217	0.001 → 0.245

NLL rises by up to $\phi$4 and ECE by two orders of magnitude off-distribution. The implication is direct: biased models act as better density estimators only under their native ordering, and their apparent likelihood quality is an artifact of memorized ordering statistics rather than graph-level understanding. Random Order shows no degradation, since its native strategy already samples uniformly.

The full $\phi$5 cross-evaluation decomposition localizes the failure mechanism. New Node tokens (first occurrences of node indices, a grammatical prediction) remain well-calibrated across orderings, with ECE at or below $\phi$6 for Min-Degree. Revisit tokens (cycle-closing back-references, a topological prediction) account for the largest failure, with off-diagonal ECE reaching approximately $\phi$7 for Min-Degree and $\phi$8 for Anchor Expansion. Node and edge label calibration also degrade substantially. The models therefore exploit topological sorting cues from their training ordering to resolve cycle-closing decisions, and lose this ability on unfamiliar orderings.

Memorization in the data-scarce regime

On Planar ($\phi$9), biased strategies exhibit a characteristic overfitting signature: validation NLL rises after roughly 5k training steps, yet composite generative quality (VUN) continues to improve. The paper's decomposition shows this apparent improvement is driven entirely by Validity; Uniqueness degrades to 90–95% and Novelty to 75–85% as the models increasingly reproduce training graphs, with Anchor Expansion's novelty collapsing to roughly 60%. Random Order sustains high values across all three components, because the combinatorially many permutations act as inherent data augmentation. The memorization persists at scale: on QM9 subsets, biased strategies recover competitive Uniqueness only at $\phi^{-1}(G)$0, and their sequence diversity saturates early at all dataset sizes. A practical consequence is that in low-data regimes, aggregate quality metrics can mask memorization unless decomposed.

The NLL paradox and LU as a quality signal

The most consequential result concerns self-assessment of generated molecules. Under constrained decoding, Generation NLL is negatively correlated with chemical stability: unstable molecules receive lower (more confident) NLL than stable ones, yielding AUC $\phi^{-1}(G)$1 for binary stability prediction — worse than chance. The proposed explanation is that unstable molecules arise from traversals that happen to align well with the training linearization, so every token prediction is confident even though the global structure violates valency. Mean permutation NLL alone is close to chance (AUC $\phi^{-1}(G)$2), but the variance across orderings is diagnostic: molecules whose low NLL depended on a specific traversal show high LU, whereas genuinely confident graph-level likelihoods are stable across orderings.

LU achieves AUC $\phi^{-1}(G)$3 for stability prediction, compared to $\phi^{-1}(G)$4 for Generation NLL and $\phi^{-1}(G)$5 for ECE, which requires full logit access. The signal is sample-efficient: even at $\phi^{-1}(G)$6 permutations, LU reaches AUC $\phi^{-1}(G)$7, far exceeding the Generation NLL baseline, and Anchor Expansion reaches $\phi^{-1}(G)$8 by $\phi^{-1}(G)$9. Random Order models converge more slowly (still at $G$0 at $G$1), indicating that unbiased models require more permutations for a stable LU estimate. The operational implication is that Generation NLL should not be used alone as a quality filter for generated molecules; permutation-based LU provides a more reliable, logit-free check.

A secondary observation concerns self-assessment more broadly: mean permutation NLL consistently exceeds Generation NLL, and generated sequences are systematically longer than algorithmically re-linearized versions of the same molecules (e.g., 127 vs. 89 tokens/graph for Random Order). A mechanistic analysis attributes this to traversal inefficiency rather than molecular complexity: generated sequences incur roughly $G$2 as many chordal back-references, often because the model learns valid but inefficient habits such as deferring explicit hydrogens to the end of the sequence, each requiring a separate back-reference.

Limitations and open questions

The paper is candid about scope. The stability-predictor results are established only on QM9, a constrained small-molecule benchmark; the authors note that more chemically diverse benchmarks such as MOSES or GuacaMol would be needed to test whether the Generation NLL inversion and the LU signal generalize. The entire analysis is confined to the SENT framework, and the invariance argument relies on SENT's permutation-equivalence property; encodings without this property would confound model-level violations with encoding-level differences, which is precisely what distinguishes this study from prior ordering comparisons. Finally, while the $G$3-sensitivity of LU is characterized on QM9, whether LU converges at comparable $G$4 on larger graphs remains open, and the slower convergence of Random Order models suggests the answer may be strategy-dependent.

Conclusion

Because SENT encodings are permutation-equivalent, any non-zero coefficient of variation of NLL across orderings is attributable to the model, not the encoding. The paper shows that autoregressive transformers trained on biased linearizations converge to their training order rather than the graph structure: they achieve lower native NLL but suffer up to $G$5 NLL inflation and two-orders-of-magnitude ECE degradation under permutation, with Revisit tokens as the dominant failure mode. Most pointedly, Generation NLL is anti-correlated with molecular stability (AUC $G$6) while permutation-based LU identifies unstable molecules with AUC $G$7 using only scalar likelihoods. Training with random linearization emerges as a simple intervention for permutation-consistent likelihoods, and permutation-based evaluation is recommended as a standard secondary check when autoregressive models generate structured objects.