Structural Cross-Entropy Ratio

• Structural Cross-Entropy Ratio is a normalized measure that compares the Shannon entropy of an empirical distribution of AST substructures with its cross-entropy counterpart.
• It is applied in evaluating code generation stability by comparing structure-only and structure+value encodings of abstract syntax trees.
• The metric, bounded between 0 and 1, provides a directional evaluation of how well one distribution covers high-probability structural patterns in another.

Structural Cross-Entropy Ratio denotes a structure-sensitive normalization of cross-entropy whose meaning depends on context, and the term is not yet fully standardized across the literature. Its clearest explicit formulation appears in the evaluation of code-generation stability, where empirical distributions over depth-bounded abstract-syntax-tree substructures are compared through a directional ratio $S_{\mathrm{CE}}(P,Q)=H(Q)/H(P,Q)$ (Song et al., 19 Aug 2025). Elsewhere, closely related ideas appear as conceptual extensions of cross-entropy to structured labels, graphs, latent-variable organization, and cost-benefit analysis, but several papers state that a “structural cross-entropy ratio” is either not given as a separate named formula, not a literal formula, or not explicitly defined (Kulick et al., 2014, Huang et al., 2024, 2505.14040). The resulting literature is therefore best read as a family of related constructions rather than a single universally fixed invariant.

1. Terminological scope and uses

The phrase has at least two distinct roles in current work. In one role, it is a concrete statistic computed from two empirical structural distributions. In another, it names or motivates a broader balance between ordinary cross-entropy and some notion of structure, such as partitions of the label space, latent-space graph organization, or compression of structured objects.

Context	Structural component	Status of the ratio
LLM code generation stability	AST subtree frequency distributions	Explicit formula (Song et al., 19 Aug 2025)
Iterative information gathering	Belief-change under asymmetric cross-entropy/KL objectives	Conceptual only (Kulick et al., 2014)
Cost-benefit and structural variants	Bounded distortion terms, normalization by $\mathcal{H}_{\max}$	Candidate foundation (Chen et al., 2019)
Probabilistic coding and graph clustering	Structural-entropy regularization plus CE terms	Not a literal or explicit ratio (Huang et al., 2024, 2505.14040)

A closely related but distinct line of work studies structured cross-entropy loss for classification. There, structure is introduced through a random partition of the label set, and the resulting loss aggregates ordinary cross-entropies over coarsenings of the target space rather than forming a ratio (Lucena, 2022). This broadens the conceptual background of the term, but it does not by itself define a Structural Cross-Entropy Ratio.

2. Explicit formulation from AST-based code generation stability

The most operational definition treats stability as consistency of structural choices made by a model across multiple generations for the same prompt, independently of functional correctness or string-level similarity (Song et al., 19 Aug 2025). For each generated program, the procedure parses the abstract syntax tree, extracts at each node a rooted subtree up to fixed depth $d$, and applies a canonical encoding $\sigma(\mathrm{sub}(v;d))$.

Two encodings are defined. The structure-only encoding records only node types and suppresses identifiers and literals. The structure+value encoding records node types together with exact tokens or lexemes. These induce two corresponding metric families: a structural-only view of control-flow and syntactic shape, and a token-aware view that is sensitive to identifier-level and literal-level variability (Song et al., 19 Aug 2025).

For two outputs $A$ and $B$, let $S_A$ and $S_B$ be the multisets of subtree encodings, $U=S_A\cup S_B$ the combined support, and $c_A(u), c_B(u)$ the counts of pattern $\mathcal{H}_{\max}$0. The empirical distributions are

$\mathcal{H}_{\max}$1

where $\mathcal{H}_{\max}$2, $\mathcal{H}_{\max}$3, and $\mathcal{H}_{\max}$4 is a small smoothing constant to avoid logs of zero (Song et al., 19 Aug 2025).

The cross-entropy of $\mathcal{H}_{\max}$5 relative to $\mathcal{H}_{\max}$6 and the Shannon entropy of $\mathcal{H}_{\max}$7 are

$\mathcal{H}_{\max}$8

The Structural Cross-Entropy Ratio is then

$\mathcal{H}_{\max}$9

This ratio equals $d$0 if and only if $d$1, and it tends toward $d$2 as $d$3 fails to cover high-probability regions of $d$4 (Song et al., 19 Aug 2025). In practical use, the metric is computed pairwise between all generations for a prompt and then averaged. The reported complexity is $d$5, where $d$6 is the total number of AST nodes and $d$7 is the subtree depth, and the method is described as reference-free, language-agnostic, and execution-independent (Song et al., 19 Aug 2025).

3. Information-theoretic interpretation

The underlying ingredients are classical. For distributions $d$8 and $d$9, cross-entropy is

$\sigma(\mathrm{sub}(v;d))$0

and Kullback-Leibler divergence is

$\sigma(\mathrm{sub}(v;d))$1

This asymmetry matters: $\sigma(\mathrm{sub}(v;d))$2 (Kulick et al., 2014).

The directional behavior of $\sigma(\mathrm{sub}(v;d))$3 follows the same general logic. In the code-generation setting, the question is not merely whether two outputs overlap structurally, but whether the structural patterns emphasized by $\sigma(\mathrm{sub}(v;d))$4 are well explained by $\sigma(\mathrm{sub}(v;d))$5 (Song et al., 19 Aug 2025). This is why the metric is contrasted with Jensen-Shannon divergence, which is symmetric and measures overall overlap rather than directional coverage (Song et al., 19 Aug 2025). A plausible implication is that Structural Cross-Entropy Ratio should be read as a normalized measure of explanatory adequacy: it discounts mismatch by the intrinsic uncertainty of the comparison distribution $\sigma(\mathrm{sub}(v;d))$6.

Related work on iterative information gathering provides an instructive precedent. Maximizing expected cross-entropy between old and new belief is equivalent to maximizing expected $\sigma(\mathrm{sub}(v;d))$7, and the paper notes that the “structural cross-entropy ratio” is not given there as a separate named formula, but that the intuition is tied to cross-entropy, entropy, and the asymmetry of KL divergence (Kulick et al., 2014). That connection is conceptual rather than definitional.

4. Structuralization mechanisms beyond the explicit ratio

One route to structuralization modifies the target space itself. In structured classification, the label set is equipped with a distribution over partitions, and the structured empirical cross-entropy loss becomes

$\sigma(\mathrm{sub}(v;d))$8

where $\sigma(\mathrm{sub}(v;d))$9 is the predicted probability mass assigned to the block containing the true label (Lucena, 2022). This preserves the cross-entropy form while making “similar” mistakes less costly than “dissimilar” ones when meaningful structure is known in advance.

A second route makes the latent representation structural. In Structural Entropy Guided Probabilistic Coding, structural entropy is defined over a graph of latent variables, and the total loss is

$A$0

with

$A$1

That work explicitly states that the “structural cross-entropy ratio” is not a literal formula, but a conceptual balance between pointwise predictive accuracy and structural quality in latent space (Huang et al., 2024).

A third route couples structural entropy minimization with cross-entropy-like local consistency. In DeSE for unsupervised graph clustering, the overall loss is

$A$2

where $A$3 is differentiable soft-assignment structural entropy and $A$4 is an edge-based cross-entropy over connected and disconnected node pairs (2505.14040). That paper states that there is no explicit “structural cross-entropy ratio”; the interaction is realized instead through weighted joint optimization (2505.14040).

5. Ratios, boundedness, and normalization

A central motivation for ratio-based constructions is interpretability under bounded scales. In the cost-benefit ratio of Chen and Golan, the distortion term originally involved KL divergence, but the unboundedness of KL was identified as problematic because cross-entropy-type terms and entropy reductions are bounded on finite alphabets whereas KL can grow arbitrarily large (Chen et al., 2019). To address this, a new bounded divergence

$A$5

was proposed, together with generalized and commutative variants, all lying in $A$6 (Chen et al., 2019). The same discussion states that bounded measures, especially when scaled by $A$7, are strong candidates as foundations for structural variants such as the Structural Cross-Entropy Ratio (Chen et al., 2019).

A distinct but related normalization appears in algebraic work on common meadows. There, cross-entropy is written as

$A$8

with totalized logarithm and left-sequential multiplication handling zero-probability edge cases without case distinctions. The paper explicitly presents the ratio

$A$9

as a total term in the algebra (Bergstra et al., 11 Feb 2025). Although this is not itself a structural construction, it supplies a rigorous template for ratio-valued entropy expressions in settings where zero probabilities or undefined logarithms would otherwise require conventions.

6. Distinctions, misconceptions, and adjacent notions

Structural Cross-Entropy Ratio is not interchangeable with Jensen-Shannon divergence. JSD is symmetric and bounded, while $B$0 is directional and is intended to expose cases in which $B$1 misses high-probability patterns present in $B$2 (Song et al., 19 Aug 2025). Nor is it interchangeable with pass@k, BLEU, CodeBLEU, or TSED. Those metrics are reference-based, execution-based, or edit-based in ways that target correctness or reference similarity rather than intrinsic structural stability across generations (Song et al., 19 Aug 2025).

It is also not the same as structural entropy. Structural entropy measures uncertainty of hierarchical graph partitions or related structural organizations, as in encoding-tree formulations for graphs, latent-variable communities, stochastic block models, or multi-relational graphs (Huang et al., 2024, Han et al., 2021, Cao et al., 2024). Multi-relational Structural Entropy, for example, introduces the entropy-reduction ratio

$B$3

which quantifies how much structural information is decoded by a community structure, but this is an entropy-reduction ratio rather than a cross-entropy ratio (Cao et al., 2024).

Finally, Structural Cross-Entropy Ratio should be distinguished from structure-aware divergences defined by Bregman divergences of structure-aware entropies over a similarity matrix $B$4. Those constructions make divergences sensitive to graded similarity between domain elements and recover Shannon-based quantities when $B$5, but they are framed as divergences rather than ratios (Sahasrabuddhe et al., 23 Mar 2026). This suggests that the present term occupies a narrower niche: it is most precise when it names a normalized cross-entropy statistic over explicitly structured representations, and more ambiguous when used as shorthand for a balance between predictive error and structural organization.

In current usage, then, Structural Cross-Entropy Ratio is best understood as a developing concept with one explicit, computationally well-specified form in AST-based stability analysis and several broader, context-dependent analogues in structured learning, graph modeling, and information-theoretic cost-benefit analysis (Song et al., 19 Aug 2025).