AST-Driven Structural Entropy

• AST-driven structural entropy is an information-theoretic method that quantifies program structure by analyzing empirical distributions of depth-bounded AST subtrees.
• It employs both structure-only and structure+value encodings with metrics like Jensen–Shannon divergence and structural cross-entropy to assess consistency in code generation.
• The approach provides a reference-free, language-agnostic evaluation tool that distinguishes between syntactic stability and semantic equivalence in generated code.

Searching arXiv for the cited papers and closely related work on AST-driven structural entropy. AST-driven structural entropy denotes an entropy-based analysis of program structure in which code is parsed into abstract syntax trees (ASTs), local subtree patterns are counted, and the resulting empirical distributions are compared to characterize structural variability. In current arXiv usage, the phrase is most directly associated with a reference-free method for evaluating the stability of LLM code generation under repeated sampling for a fixed prompt: the method extracts depth-bounded AST subtrees, converts their relative frequencies into probability distributions, and measures agreement with Jensen–Shannon divergence and a Structural Cross-Entropy ratio (Song et al., 19 Aug 2025). More broadly, it belongs to a wider family of structural-entropy constructions in which entropy is attached not to raw strings alone, but to motif populations, recurring patches, or soft structural partitions (Han et al., 2019, Javerzat et al., 28 May 2026, 2505.14040).

1. Conceptual scope and definition

The central object in AST-driven structural entropy is not the token sequence of a program, but the distribution of local syntactic structures induced by its AST. For any fixed prompt, repeated generations can be treated as repeated samples from a model-specific structural process. Stability is then operationalized as the extent to which these generations induce similar distributions over AST subtree patterns. In the formulation introduced for code generation, the method is explicitly reference-free, language-agnostic, and execution-independent, and it is positioned against evaluation regimes such as pass@k, BLEU, and CodeBLEU, which target correctness or reference overlap rather than repeated-run structural consistency (Song et al., 19 Aug 2025).

This usage of “structural entropy” is information-theoretic rather than thermodynamic. The relevant entropy is computed over symbolic subtree types and their multiplicities. That places the method in continuity with earlier work that defined a Shannon information entropy over local structural motif populations, specifically Voronoi polyhedra in a model glass former, as a qualitative measure of configurational entropy (Han et al., 2019). It is also conceptually adjacent to work that estimates entropy from the decay rate of recurring structural patterns as patch size increases, thereby connecting entropy reduction to the persistence of amorphous order (Javerzat et al., 28 May 2026).

A common misconception is that AST-driven structural entropy is a correctness metric. It is not. The method for code generation stability does not require tests, does not rely on a gold solution, and does not address semantic equivalence or execution behavior directly (Song et al., 19 Aug 2025). Another misconception is that “structural entropy” has a single universal meaning across domains. In graph clustering it may denote a differentiable soft-assignment objective over graph partitions (2505.14040); in correlated materials it may denote a cooperative free-energy advantage arising from phonon and spin entropy in a structural phase (Zhu et al., 1 Jul 2026). AST-driven structural entropy refers specifically to entropy over program structure represented through ASTs.

2. AST representation and induced probability distributions

The AST-based formulation begins by parsing each generated program into an AST, denoted for two samples as $\mathcal{T}_A$ and $\mathcal{T}_B$. For a fixed depth parameter $d \in \mathbb{N}$, and for every node $v$ in an AST $\mathcal{T}$, the extracted local object is the rooted depth-bounded subtree

$\mathrm{sub}(v;d),$

defined as the fragment rooted at $v$ that contains all descendants up to depth $d$ (Song et al., 19 Aug 2025). The representation is therefore local and multiplicity-aware: each AST contributes one truncated subtree per node, and repeated patterns count multiple times.

Two canonical encodings are defined. The structure-only encoding is

$\sigma_{\text{struct} : \mathrm{sub}(v;d)\;\mapsto\;(\text{node-type}(v),\; (\text{node-type}(c_1),\dots,\text{node-type}(c_k))),$

where $c_1,\dots,c_k$ are the immediate children of $\mathcal{T}_B$0. This preserves syntactic form while abstracting away identifier names and literal values. The structure+value encoding is

$\mathcal{T}_B$1

where $\mathcal{T}_B$2 is the exact source text covered by $\mathcal{T}_B$3 for leaves, or a sentinel $\mathcal{T}_B$4 for internal nodes. The distinction is substantive: the structural-only view isolates control-flow and expression-shape regularities, whereas the token-aware view becomes sensitive to identifiers, literals, and concrete lexical choices (Song et al., 19 Aug 2025).

The extracted encodings form multisets

$\mathcal{T}_B$5

If $\mathcal{T}_B$6 denotes the countable universe of canonical subtree encodings, then the aligned support is the joint union

$\mathcal{T}_B$7

For $\mathcal{T}_B$8, the multiplicity functions are

$\mathcal{T}_B$9

with totals

$d \in \mathbb{N}$0

The empirical subtree distributions are then defined as

$d \in \mathbb{N}$1

where $d \in \mathbb{N}$2 is a smoothing constant used for cross-entropy calculations (Song et al., 19 Aug 2025).

This construction makes the entropy object prompt-conditional and sample-conditional. The code-generation paper defines distributions pairwise over generations rather than as a single pooled prompt-level distribution. That design choice matters: the resulting scores quantify stability between generated programs for the same prompt, not a corpus-level LLM prior over AST fragments (Song et al., 19 Aug 2025).

3. Divergence measures and structural interpretation

AST-driven structural entropy in the code-generation setting is operationalized through two complementary comparisons: a directed Structural Cross-Entropy ratio and a symmetric Jensen–Shannon divergence–based similarity (Song et al., 19 Aug 2025).

The directed component begins with

$d \in \mathbb{N}$3

The normalized ratio is

$d \in \mathbb{N}$4

Its interpretation is also given explicitly: $d \in \mathbb{N}$5 Because this measure is directed, $d \in \mathbb{N}$6 need not equal $d \in \mathbb{N}$7. It is designed to highlight missing high-probability patterns: if $d \in \mathbb{N}$8 assigns low probability to subtree types that are common in $d \in \mathbb{N}$9, then $v$0 increases and the ratio decreases. The relation

$v$1

shows that the ratio is monotone in the KL divergence, although no separate normalized KL score is introduced (Song et al., 19 Aug 2025).

The symmetric component uses the midpoint distribution

$v$2

and defines

$v$3

With logarithm base 2,

$v$4

and the corresponding similarity is

$v$5

This score is bounded and symmetric. In AST terms, it measures structural overlap between two generations at the level of subtree frequency distributions. High $v$6 indicates broad agreement in subtree vocabularies and frequencies; low $v$7 indicates substantial structural divergence (Song et al., 19 Aug 2025).

The coexistence of the two measures is significant. JSD can remain high even when SCE drops. The paper interprets that pattern as evidence that two outputs retain broad AST similarity while differing in whether one output adequately explains the high-probability local patterns of the other. This distinction becomes especially visible when the structure+value encoding is used, because token-aware distributions are more sensitive to identifier-level and literal-level variation (Song et al., 19 Aug 2025).

4. Workflow, complexity, and empirical behavior in code generation

The evaluation protocol is defined at the prompt level. For each prompt, the model is queried five times independently, producing five distinct outputs per prompt. Each output is parsed into an AST; one extracts the multiset of depth-bounded subtrees; one computes pairwise scores among the five outputs; and the final prompt-level stability signal is the average of those pairwise comparisons. Model-level reporting is then obtained by averaging across prompts (Song et al., 19 Aug 2025).

The end-to-end computational pipeline has three phases. First, code is generated repeatedly for the same prompt. Second, each generated program is parsed and converted into a multiset of canonical subtree symbols under either the structural-only or structure+value encoding. Third, for each pair of generations one forms the support union $v$8, constructs $v$9 and $\mathcal{T}$0, computes $\mathcal{T}$1 and $\mathcal{T}$2, and averages the results. The paper states that subtree extraction runs in $\mathcal{T}$3 time, where $\mathcal{T}$4 is the number of AST nodes and $\mathcal{T}$5 is the subtree depth bound, and that scoring after distribution construction is $\mathcal{T}$6. When depth $\mathcal{T}$7 is fixed, $\mathcal{T}$8 grows at most linearly in the number of AST nodes, which motivates the claim that the method is a lightweight addition to code-generation evaluation (Song et al., 19 Aug 2025).

The empirical study uses Python / BigCodeBench and SQL / Spider, and evaluates LLaMA-3.1 8B instruct, Qwen-2.5 7B instruct, and Qwen-2.5-Coder 7B. On Python / BigCodeBench, the reported scores include, for example, Qwen-2.5-Coder 7B with SCE(structural): 0.823, SCE: 0.722, JSD(structural): 0.951, and JSD: 0.918, while LLaMA-3.1 8B it records SCE(structural): 0.798, SCE: 0.656, JSD(structural): 0.940, and JSD: 0.898. On SQL / Spider, Qwen-2.5 7B it records SCE(structural): 0.886, SCE: 0.850, JSD(structural): 0.971, and JSD: 0.958, while Qwen-2.5-Coder 7B records SCE(structural): 0.822, SCE: 0.781, JSD(structural): 0.962, and JSD: 0.946 (Song et al., 19 Aug 2025).

The principal empirical pattern is that JSD values are consistently high, often above 0.9, whereas SCE values are noticeably lower, especially in token-aware form. The reported interpretation is that repeated outputs are broadly similar in AST structure, yet still differ in finer local pattern realization. This creates an evaluation dimension that pass@k does not expose. A large pass@1 versus pass@5 gap indicates variability in correctness, but it does not reveal whether the variability is deep syntactic reorganization or relatively superficial token-level fluctuation. The AST entropy framework is designed to make precisely that distinction visible (Song et al., 19 Aug 2025).

5. Relation to broader structural-entropy research

AST-driven structural entropy is one instance of a broader methodological pattern in which entropy is defined over structured objects rather than unstructured symbol sequences. In the metallic-glass study, the relevant structured objects are local atomic motifs identified by Voronoi tessellation of inherent structures. Their populations define

$\mathcal{T}$9

which is used as a qualitative measure for configurational entropy. The paper argues that cooling favors a limited set of more centrosymmetric motifs, making the motif distribution more uneven and lowering Shannon entropy, with a pronounced kink at the glass transition (Han et al., 2019). The formal resemblance to AST-driven subtree histograms is direct: both constructions quantify structural diversity by entropy over discrete local types.

A second neighboring line of work estimates entropy from recurrence statistics of spatial patches rather than from motif histograms. There the core object is the correlation integral

$\mathrm{sub}(v;d),$0

and the entropy density is extracted from the scaling law

$\mathrm{sub}(v;d),$1

In that formulation, smaller entropy implies that larger patches recur more often, or equivalently that structural correlations persist over larger length scales (Javerzat et al., 28 May 2026). This suggests a possible AST interpretation in which rooted subtrees or local AST neighborhoods play the role of patches and recurrence decay with subtree size yields a corpus-level structural entropy rate. That interpretation is a transfer, not an implemented code-generation benchmark in the cited work.

A third lineage uses structural entropy as a differentiable graph-partition objective. In DeSE, the entropy is hierarchical and soft-assignment based: $\mathrm{sub}(v;d),$2 with soft volume and cut defined by

$\mathrm{sub}(v;d),$3

This objective is optimized jointly with graph refinement and an edge-based cross-entropy loss in unsupervised graph clustering (2505.14040). A plausible implication is that AST-derived program graphs, including AST plus control-flow or data-flow edges, could be treated with analogous soft structural entropy objectives, although the cited paper itself assumes an undirected homogeneous graph rather than a directed typed syntax graph.

A fourth neighboring literature concerns entropy-regularized structural inference under annealing. In entropy-regularized optimal transport, the entropic solution

$\mathrm{sub}(v;d),$4

is annealed as $\mathrm{sub}(v;d),$5 to recover hard assignments, but the analysis identifies Premature Mode Collapse and a Thermodynamic Speed Limit

$\mathrm{sub}(v;d),$6

The paper’s direct subject is matching and permutations, not ASTs, yet it provides a dynamical theory of entropy removal in latent structural inference (Liu, 30 Jan 2026). This suggests that if AST-based models use temperature or entropy smoothing over latent syntax structures, fixed aggressive annealing schedules may also risk premature commitment to an incorrect discrete structure; that extrapolation is explicitly broader than the paper’s tested setting.

Finally, in the Nb$\mathrm{sub}(v;d),$7Cl$\mathrm{sub}(v;d),$8 study, “structural entropy” refers to a phase-dependent free-energy contribution jointly driven by phonon and spin entropy. The high-temperature $\mathrm{sub}(v;d),$9 phase is stabilized by softer phonons and larger paramagnetic spin entropy, whereas the low-temperature $v$0 phase is favored by interlayer dimerization, harder phonons, and quenched spin entropy (Zhu et al., 1 Jul 2026). This usage is thermodynamic rather than information-theoretic. Its relevance to AST-driven work is primarily terminological: it shows that “structural entropy” may refer either to a state-counting functional over explicit structures or to an entropy contribution associated with a structural phase.

6. Limitations, misconceptions, and research directions

The AST-based code-generation method has several explicit limitations. It depends on valid AST parsing, so malformed outputs or parser-specific AST conventions can affect results. It is sensitive to subtree depth $v$1: small $v$2 captures only shallow local structure, while larger $v$3 captures more context but increases sparsity and cost. The representation depends on AST granularity and parser encoding choices, and the structure+value variant is more discriminative but can become sparse and overly sensitive to identifier and literal variation. The empirical scope is also limited to two benchmarks and three models (Song et al., 19 Aug 2025).

Several interpretive cautions follow from those design choices. High structural similarity does not establish semantic equivalence, and low structural similarity does not prove semantic instability. Two semantically equivalent programs with different syntax may still look different structurally. Repeated identifier renaming can depress token-aware scores even when algorithmic structure is unchanged. Very shallow subtree representations may overstate stability by ignoring longer-range control or data dependencies. These are not defects in the formalism so much as boundaries on what the measured entropy actually represents (Song et al., 19 Aug 2025).

The most productive research directions arise by combining the existing strands rather than by collapsing them into a single notion. One direction is to move from pairwise prompt-level subtree histograms toward recurrence-based entropy rates over larger AST patches, in the spirit of the structural-pattern scaling law used for liquids and glasses (Javerzat et al., 28 May 2026). Another is to move from flat AST subtree distributions toward differentiable soft partitions on AST-derived graphs, following the soft-assignment entropy construction of graph clustering (2505.14040). A third is to study whether entropy-reduction schedules in latent syntax models should be treated as stability-constrained continuation processes rather than fixed schedules, as suggested by the annealing analysis of entropy-regularized structural inference (Liu, 30 Jan 2026).

Taken together, the literature indicates that AST-driven structural entropy is best understood as a family of information-theoretic procedures for quantifying structural diversity, recurrence, or stability in syntax trees. In its most direct current form, it measures repeated-run consistency of code generation by comparing empirical distributions of depth-bounded AST subtrees (Song et al., 19 Aug 2025). In a broader methodological sense, it also points toward a general program-structure calculus in which entropy is attached to local syntax motifs, recurring rooted patterns, or soft syntactic modules, with different choices emphasizing different notions of stability, diversity, and structural order.