---
title: AST-Driven Structural Entropy
url: https://www.emergentmind.com/topics/ast-driven-structural-entropy
type: topic
---

# AST-Driven Structural Entropy

Searching arXiv for the cited papers and closely related work on AST-driven structural entropy.
AST-driven structural entropy denotes an entropy-based analysis of program structure in which code is parsed into abstract syntax trees (ASTs), local subtree patterns are counted, and the resulting empirical distributions are compared to characterize structural variability. In current arXiv usage, the phrase is most directly associated with a reference-free method for evaluating the stability of large language model code generation under repeated sampling for a fixed prompt: the method extracts depth-bounded AST subtrees, converts their relative frequencies into probability distributions, and measures agreement with Jensen–Shannon divergence and a Structural Cross-Entropy ratio [2508.14288]. More broadly, it belongs to a wider family of structural-entropy constructions in which entropy is attached not to raw strings alone, but to motif populations, recurring patches, or soft structural partitions [1907.03695], [2605.29945], [2505.14040].

## 1. Conceptual scope and definition

The central object in AST-driven structural entropy is not the token sequence of a program, but the distribution of local syntactic structures induced by its AST. For any fixed prompt, repeated generations can be treated as repeated samples from a model-specific structural process. Stability is then operationalized as the extent to which these generations induce similar distributions over AST subtree patterns. In the formulation introduced for code generation, the method is explicitly **reference-free**, **language-agnostic**, and **execution-independent**, and it is positioned against evaluation regimes such as pass@k, BLEU, and CodeBLEU, which target correctness or reference overlap rather than repeated-run structural consistency [2508.14288].

This usage of “structural entropy” is information-theoretic rather than thermodynamic. The relevant entropy is computed over symbolic subtree types and their multiplicities. That places the method in continuity with earlier work that defined a **Shannon information entropy** over local structural motif populations, specifically Voronoi polyhedra in a model glass former, as a qualitative measure of configurational entropy [1907.03695]. It is also conceptually adjacent to work that estimates entropy from the decay rate of recurring structural patterns as patch size increases, thereby connecting entropy reduction to the persistence of amorphous order [2605.29945].

A common misconception is that AST-driven structural entropy is a correctness metric. It is not. The method for code generation stability does not require tests, does not rely on a gold solution, and does not address semantic equivalence or execution behavior directly [2508.14288]. Another misconception is that “structural entropy” has a single universal meaning across domains. In graph clustering it may denote a differentiable soft-assignment objective over graph partitions [2505.14040]; in correlated materials it may denote a cooperative free-energy advantage arising from phonon and spin entropy in a structural phase [2607.00599]. AST-driven structural entropy refers specifically to entropy over program structure represented through ASTs.

## 2. AST representation and induced probability distributions

The AST-based formulation begins by parsing each generated program into an AST, denoted for two samples as $\mathcal{T}_A$ and $\mathcal{T}_B$. For a fixed depth parameter $d \in \mathbb{N}$, and for every node $v$ in an AST $\mathcal{T}$, the extracted local object is the rooted depth-bounded subtree
\[
\mathrm{sub}(v;d),
\]
defined as the fragment rooted at $v$ that contains all descendants up to depth $d$ [2508.14288]. The representation is therefore local and multiplicity-aware: each AST contributes one truncated subtree per node, and repeated patterns count multiple times.

Two canonical encodings are defined. The **structure-only encoding** is
\[
\sigma_{\text{struct} : \mathrm{sub}(v;d)\;\mapsto\;(\text{node-type}(v),\; (\text{node-type}(c_1),\dots,\text{node-type}(c_k))),
\]
where $c_1,\dots,c_k$ are the immediate children of $v$. This preserves syntactic form while abstracting away identifier names and literal values. The **structure+value encoding** is
\[
\sigma_{\text{value} : \mathrm{sub}(v;d)\;\mapsto\;(\text{node-type}(v),\; \text{lexeme}(v),\; (\text{node-type}(c_1),\dots,\text{node-type}(c_k))),
\]
where $\text{lexeme}(v)$ is the exact source text covered by $v$ for leaves, or a sentinel $\varnothing$ for internal nodes. The distinction is substantive: the structural-only view isolates control-flow and expression-shape regularities, whereas the token-aware view becomes sensitive to identifiers, literals, and concrete lexical choices [2508.14288].

The extracted encodings form multisets
\[
S_A \;=\; \bigl\{\sigma\bigl(\mathrm{sub}(v;d)\bigr)\,\bigm|\, v\in \mathcal{T}_A\bigr\}, \qquad S_B \;=\; \bigl\{\sigma\bigl(\mathrm{sub}(v;d)\bigr)\,\bigm|\, v\in \mathcal{T}_B\bigr\}.
\]
If $\Sigma^{\!*}$ denotes the countable universe of canonical subtree encodings, then the aligned support is the joint union
\[
U \;=\; S_A \cup S_B \;=\; \{u_1,\dots,u_m\}, \qquad m = |U|.
\]
For $u \in U$, the multiplicity functions are
\[
c_A(u) \;=\; \#\{\,s\in S_A : s = u\,\}, \qquad c_B(u) \;=\; \#\{\,s\in S_B : s = u\,\},
\]
with totals
\[
n_A \;=\; \sum_{u\in U} c_A(u), \qquad n_B \;=\; \sum_{u\in U} c_B(u).
\]
The empirical subtree distributions are then defined as
\[
P(u) \;=\; \frac{c_A(u)}{n_A}, \qquad Q(u) \;=\; \max\!\Bigl(\frac{c_B(u)}{n_B},\,\varepsilon\Bigr), \qquad \sum_{u\in U} P(u) \;=\; \sum_{u\in U} Q(u) \;=\; 1,
\]
where $0<\varepsilon\ll 1$ is a smoothing constant used for cross-entropy calculations [2508.14288].

This construction makes the entropy object prompt-conditional and sample-conditional. The code-generation paper defines distributions pairwise over generations rather than as a single pooled prompt-level distribution. That design choice matters: the resulting scores quantify stability between generated programs for the same prompt, not a corpus-level language model prior over AST fragments [2508.14288].

## 3. Divergence measures and structural interpretation

AST-driven structural entropy in the code-generation setting is operationalized through two complementary comparisons: a directed **Structural Cross-Entropy ratio** and a symmetric **Jensen–Shannon divergence–based similarity** [2508.14288].

The directed component begins with
\[
H(P,Q)\;=\;-\sum_{u\in U}P(u)\log Q(u),\qquad H(Q)\;=\;-\sum_{u\in U}Q(u)\log Q(u).
\]
The normalized ratio is
\[
S_{\mathrm{CE}(P,Q)\;=\; \frac{H(Q)}{H(P,Q)},\qquad 0<S_{\mathrm{CE}\le 1.
\]
Its interpretation is also given explicitly:
\[
S_{\mathrm{CE}(P,Q)=1 \ \Longleftrightarrow\ P=Q;\quad S_{\mathrm{CE}\to 0 \ \text{as}\ Q\ \text{fails to explain}\ P.
\]
Because this measure is directed, $S_{\mathrm{CE}(P,Q)$ need not equal $S_{\mathrm{CE}(Q,P)$. It is designed to highlight missing high-probability patterns: if $Q$ assigns low probability to subtree types that are common in $P$, then $H(P,Q)$ increases and the ratio decreases. The relation
\[
D_{\mathrm{KL}(P\parallel Q)=H(P,Q)-H(P)
\]
shows that the ratio is monotone in the KL divergence, although no separate normalized KL score is introduced [2508.14288].

The symmetric component uses the midpoint distribution
\[
M=\tfrac12(P+Q)
\]
and defines
\[
D_{\mathrm{JS}(P\parallel Q) \;=\; \tfrac12D_{\mathrm{KL}(P\parallel M) \;+\; \tfrac12D_{\mathrm{KL}(Q\parallel M) \;=\; H(M)-\tfrac12\!\bigl[H(P)+H(Q)\bigr].
\]
With logarithm base 2,
\[
D_{\mathrm{JS} \in [0,1],
\]
and the corresponding similarity is
\[
S_{\mathrm{JSD}(P,Q)\;=\;1-D_{\mathrm{JS}(P\parallel Q)\in[0,1].
\]
This score is bounded and symmetric. In AST terms, it measures structural overlap between two generations at the level of subtree frequency distributions. High $S_{\mathrm{JSD}$ indicates broad agreement in subtree vocabularies and frequencies; low $S_{\mathrm{JSD}$ indicates substantial structural divergence [2508.14288].

The coexistence of the two measures is significant. JSD can remain high even when SCE drops. The paper interprets that pattern as evidence that two outputs retain broad AST similarity while differing in whether one output adequately explains the high-probability local patterns of the other. This distinction becomes especially visible when the structure+value encoding is used, because token-aware distributions are more sensitive to identifier-level and literal-level variation [2508.14288].

## 4. Workflow, complexity, and empirical behavior in code generation

The evaluation protocol is defined at the prompt level. For each prompt, the model is queried **five times independently**, producing **five distinct outputs per prompt**. Each output is parsed into an AST; one extracts the multiset of depth-bounded subtrees; one computes pairwise scores among the five outputs; and the final prompt-level stability signal is the average of those pairwise comparisons. Model-level reporting is then obtained by averaging across prompts [2508.14288].

The end-to-end computational pipeline has three phases. First, code is generated repeatedly for the same prompt. Second, each generated program is parsed and converted into a multiset of canonical subtree symbols under either the structural-only or structure+value encoding. Third, for each pair of generations one forms the support union $U$, constructs $P$ and $Q$, computes $S_{\mathrm{CE}$ and $S_{\mathrm{JSD}$, and averages the results. The paper states that subtree extraction runs in $O(n,d)$ time, where $n$ is the number of AST nodes and $d$ is the subtree depth bound, and that scoring after distribution construction is $O(|U|)$. When depth $d$ is fixed, $|U|$ grows at most linearly in the number of AST nodes, which motivates the claim that the method is a lightweight addition to code-generation evaluation [2508.14288].

The empirical study uses **Python / BigCodeBench** and **SQL / Spider**, and evaluates **LLaMA-3.1 8B instruct**, **Qwen-2.5 7B instruct**, and **Qwen-2.5-Coder 7B**. On Python / BigCodeBench, the reported scores include, for example, **Qwen-2.5-Coder 7B** with **SCE(structural): 0.823**, **SCE: 0.722**, **JSD(structural): 0.951**, and **JSD: 0.918**, while **LLaMA-3.1 8B it** records **SCE(structural): 0.798**, **SCE: 0.656**, **JSD(structural): 0.940**, and **JSD: 0.898**. On SQL / Spider, **Qwen-2.5 7B it** records **SCE(structural): 0.886**, **SCE: 0.850**, **JSD(structural): 0.971**, and **JSD: 0.958**, while **Qwen-2.5-Coder 7B** records **SCE(structural): 0.822**, **SCE: 0.781**, **JSD(structural): 0.962**, and **JSD: 0.946** [2508.14288].

The principal empirical pattern is that JSD values are consistently high, often above 0.9, whereas SCE values are noticeably lower, especially in token-aware form. The reported interpretation is that repeated outputs are broadly similar in AST structure, yet still differ in finer local pattern realization. This creates an evaluation dimension that pass@k does not expose. A large pass@1 versus pass@5 gap indicates variability in correctness, but it does not reveal whether the variability is deep syntactic reorganization or relatively superficial token-level fluctuation. The AST entropy framework is designed to make precisely that distinction visible [2508.14288].

## 5. Relation to broader structural-entropy research

AST-driven structural entropy is one instance of a broader methodological pattern in which entropy is defined over structured objects rather than unstructured symbol sequences. In the metallic-glass study, the relevant structured objects are local atomic motifs identified by Voronoi tessellation of inherent structures. Their populations define
\[
S_{\rm Shannon} = - \sum_{i=1}^{n} P_i(x_i)\ln P_i(x_i),
\]
which is used as a qualitative measure for configurational entropy. The paper argues that cooling favors a limited set of more centrosymmetric motifs, making the motif distribution more uneven and lowering Shannon entropy, with a pronounced kink at the glass transition [1907.03695]. The formal resemblance to AST-driven subtree histograms is direct: both constructions quantify structural diversity by entropy over discrete local types.

A second neighboring line of work estimates entropy from recurrence statistics of spatial patches rather than from motif histograms. There the core object is the correlation integral
\[
C_m^{(d)}(\epsilon) = \frac{1}{\mathcal{N}} \sum_{I=1}^{\mathcal{N}} \left[ n_I^{(d)}(\epsilon)\right]^{m-1},
\]
and the entropy density is extracted from the scaling law
\[
C_m^{(d)}(\epsilon) \simeq c\, \epsilon^{\nu} \exp\!\left[-d(m - 1)s^{\mathrm{R\acute{e}nyi}_{m}\right].
\]
In that formulation, smaller entropy implies that larger patches recur more often, or equivalently that structural correlations persist over larger length scales [2605.29945]. This suggests a possible AST interpretation in which rooted subtrees or local AST neighborhoods play the role of patches and recurrence decay with subtree size yields a corpus-level structural entropy rate. That interpretation is a transfer, not an implemented code-generation benchmark in the cited work.

A third lineage uses **structural entropy** as a differentiable graph-partition objective. In **DeSE**, the entropy is hierarchical and soft-assignment based:
\[
H^{\mathcal{T}_{sa}(G)=\sum_{k=1}^{h} H_{sa}(G;k),
\]
with soft volume and cut defined by
\[
vol^k[i] = D (C^k)_i, \qquad g_i^k = vol^k[i]-vol^k_{in}[i]
       = vol^k[i] - (C^k_i)^\top W (C^k_i).
\]
This objective is optimized jointly with graph refinement and an edge-based cross-entropy loss in unsupervised graph clustering [2505.14040]. A plausible implication is that AST-derived program graphs, including AST plus control-flow or data-flow edges, could be treated with analogous soft structural entropy objectives, although the cited paper itself assumes an undirected homogeneous graph rather than a directed typed syntax graph.

A fourth neighboring literature concerns entropy-regularized structural inference under annealing. In entropy-regularized optimal transport, the entropic solution
\[
P^\varepsilon = \mathcal S_\varepsilon(C)
\]
is annealed as $\varepsilon \to 0$ to recover hard assignments, but the analysis identifies **Premature Mode Collapse** and a **Thermodynamic Speed Limit**
\[
\delta_t \le O(\epsilon_t^2)\cdot R.
\]
The paper’s direct subject is matching and permutations, not ASTs, yet it provides a dynamical theory of entropy removal in latent structural inference [2601.23039]. This suggests that if AST-based models use temperature or entropy smoothing over latent syntax structures, fixed aggressive annealing schedules may also risk premature commitment to an incorrect discrete structure; that extrapolation is explicitly broader than the paper’s tested setting.

Finally, in the Nb$_3$Cl$_8$ study, “structural entropy” refers to a phase-dependent free-energy contribution jointly driven by phonon and spin entropy. The high-temperature $\alpha$ phase is stabilized by softer phonons and larger paramagnetic spin entropy, whereas the low-temperature $\beta$ phase is favored by interlayer dimerization, harder phonons, and quenched spin entropy [2607.00599]. This usage is thermodynamic rather than information-theoretic. Its relevance to AST-driven work is primarily terminological: it shows that “structural entropy” may refer either to a state-counting functional over explicit structures or to an entropy contribution associated with a structural phase.

## 6. Limitations, misconceptions, and research directions

The AST-based code-generation method has several explicit limitations. It depends on valid AST parsing, so malformed outputs or parser-specific AST conventions can affect results. It is sensitive to subtree depth $d$: small $d$ captures only shallow local structure, while larger $d$ captures more context but increases sparsity and cost. The representation depends on AST granularity and parser encoding choices, and the structure+value variant is more discriminative but can become sparse and overly sensitive to identifier and literal variation. The empirical scope is also limited to two benchmarks and three models [2508.14288].

Several interpretive cautions follow from those design choices. High structural similarity does not establish semantic equivalence, and low structural similarity does not prove semantic instability. Two semantically equivalent programs with different syntax may still look different structurally. Repeated identifier renaming can depress token-aware scores even when algorithmic structure is unchanged. Very shallow subtree representations may overstate stability by ignoring longer-range control or data dependencies. These are not defects in the formalism so much as boundaries on what the measured entropy actually represents [2508.14288].

The most productive research directions arise by combining the existing strands rather than by collapsing them into a single notion. One direction is to move from pairwise prompt-level subtree histograms toward recurrence-based entropy rates over larger AST patches, in the spirit of the structural-pattern scaling law used for liquids and glasses [2605.29945]. Another is to move from flat AST subtree distributions toward differentiable soft partitions on AST-derived graphs, following the soft-assignment entropy construction of graph clustering [2505.14040]. A third is to study whether entropy-reduction schedules in latent syntax models should be treated as stability-constrained continuation processes rather than fixed schedules, as suggested by the annealing analysis of entropy-regularized structural inference [2601.23039].

Taken together, the literature indicates that AST-driven structural entropy is best understood as a family of information-theoretic procedures for quantifying structural diversity, recurrence, or stability in syntax trees. In its most direct current form, it measures repeated-run consistency of code generation by comparing empirical distributions of depth-bounded AST subtrees [2508.14288]. In a broader methodological sense, it also points toward a general program-structure calculus in which entropy is attached to local syntax motifs, recurring rooted patterns, or soft syntactic modules, with different choices emphasizing different notions of stability, diversity, and structural order.

Source: https://www.emergentmind.com/topics/ast-driven-structural-entropy