Formal-AST: Structured Representation

• Formal-AST is a structure-aware representation that uses abstract syntax trees to capture hierarchical, syntactic, and semantic details of formal languages.
• It drives improvements in code generation, verification, and autoformalization by integrating compiler feedback, AST-aware preprocessing, and attention mechanisms.
• Contemporary research shows that aligning AST structure with task-specific constraints is key to overcoming challenges in search efficiency and semantic preservation.

Searching arXiv for papers on Formal-AST and AST-based formal representations. In the cited literature, “Formal-AST” denotes the use of abstract syntax trees, AST-derived graphs, or AST-like structured formal objects as an intermediate or direct representation for generation, understanding, verification, and search. The shared premise is that many targets of interest—source code, SQL, SVRF, Lean 4 statements, Alloy predicates, and textual DSL models—are not adequately described as flat token sequences, because their correctness and utility depend on hierarchical structure, typed relations, and grammar-constrained composition. As a result, contemporary systems either operate directly on ASTs, enrich them with control- and data-flow information, serialize them for standard Transformers, or use compiler-validated refinement to recover a well-formed formal object (Gong et al., 2024). This suggests that Formal-AST is best understood not as a single architecture, but as a family of structure-aware representation and search strategies spanning program analysis, code generation, and autoformalization (Sun et al., 2023).

1. Formal-AST as a representation principle

A central motivation for Formal-AST methods is the claim that code and other formal languages are not merely token streams. AST-T5 states that code has a hierarchical, syntactic structure, and argues that sequence-only modeling can arbitrarily break structural boundaries, miss higher-level units such as expressions, statements, branches, function bodies, and class definitions, and fail to encode long-range dependencies governed by syntax rather than lexical proximity (Gong et al., 2024). AstBERT makes the same point in a domain-specific setting, describing AST as a “tree structure description of code semantics” that helps the model “understand the semantic and syntactic information from the data from different perspectives” (Liang et al., 2022).

This representation principle is formalized explicitly in the comparative survey on AST-based code representation, which adopts the standard definition of an AST as

$\langle N, T, X, s, \delta, \phi \rangle,$

where $N$ are non-terminal nodes, $T$ terminal nodes, $X$ terminal values, $s$ the root, $\delta$ the child-function, and $\phi$ the terminal-value mapping (Sun et al., 2023). In that framing, AST is “formal” because it is induced by the programming language grammar, encodes syntactic structure, and is intended to preserve semantics more faithfully than raw token sequences.

Several papers extend this idea beyond conventional code understanding. In text-to-SQL, ASTormer treats decoding as structured AST generation rather than flat sequence generation, with frontier-node expansion constrained by grammar actions (Cao et al., 2023). In textual DSL engineering, the AST is an intermediate artifact between text and the target meta-model, with text-to-AST handling syntax-related concerns and AST-to-model handling semantic analysis (0801.1219). In autoformalization, recent Lean 4 systems do not always expose an explicit neural AST module, but they operate on formal statements as structured formal objects whose syntax can be compiled, repaired, and semantically checked; ATF explicitly characterizes this as “tool-guided structured generation,” while also noting that it is not an explicit AST parser/generator architecture (Guo et al., 8 Oct 2025).

2. Formal mechanisms for preserving and exploiting structure

Formal-AST systems differ primarily in how they inject structure into preprocessing, objectives, attention, or search. AST-T5 is representative of preprocessing-based structuralization. Its AST-Aware Segmentation chooses chunk boundaries that minimize broken subtrees by solving

$$dp[k, i] = cost[i] + \min_{i - max\_len \le j < i} dp[k - 1, j],$$

where $cost[i]$ counts AST-structure breaks if partitioning happens right after token $i$ (Gong et al., 2024). The same paper replaces random span masking with AST-Aware Span Corruption, masking subtree-aligned spans via

$N$0

This preserves meaningful syntactic units during denoising pretraining.

Other systems modify the encoder or decoder more directly. AstBERT introduces AST-specific embeddings—token embedding, AST-segment embedding, AST-position embedding, and segment embedding—and then constrains attention with AST-Mask-Self-Attention so that AST tokens only influence structurally related tokens on the same branch (Liang et al., 2022). AST-Transformer instead keeps a Transformer encoder but precomputes two relation matrices over AST nodes: an ancestor-descendant matrix $N$1 encoding shortest-path distance in the tree, and a sibling matrix $N$2 encoding horizontal sibling distance (Tang et al., 2021). ASTormer brings a similar idea to grammar-based decoding by combining absolute structural features—node type, parent rule, and depth—with relative LCA-based relations

$N$3

so that self-attention operates over explicit tree relations rather than generic sequence positions (Cao et al., 2023).

A third class of mechanisms treats structure as a symbolic search space rather than only a neural prior. FormalEvolve introduces EvolAST, a call-free diversity fallback that applies symbolic rewrite operations directly to the Lean theorem AST. The rewrite surface includes conservative hypothesis reordering, commutativity, associativity, distributivity for conjunction and disjunction, symmetry swaps such as $N$4, and dual relations such as $N$5 (Lu et al., 20 Mar 2026). This preserves compilation-gated structure while injecting diversity into test-time autoformalization search.

3. Program representation, generation, and synthesis systems

The most extensive Formal-AST work appears in code representation learning and structured generation. The systems below illustrate the main design patterns.

Paper	Task domain	Structural mechanism
“AST-T5” (Gong et al., 2024)	Code generation, transpilation, understanding	AST-aware segmentation and AST-aware span corruption
“AstBERT” (Liang et al., 2022)	Financial code understanding	Pruned AST input, AST-specific embeddings, AST-Mask-Self-Attention
“AST-Transformer” (Tang et al., 2021)	Code summarization	Ancestor-descendant and sibling relation matrices
“ASTormer” (Cao et al., 2023)	Text-to-SQL	AST-aware Transformer decoder with absolute and relative tree positions
“An AST-guided LLM Approach for SVRF Code Synthesis” (Abdelmalak et al., 1 Jul 2025)	SVRF synthesis	ANTLR AST construction, AST serialization, AST-weighted supervision

AST-T5 is deliberately architecture-light: it uses a standard T5-base-like encoder-decoder Transformer, adds no special AST encoder, no new heads, no graph module, and no architectural change at all; the only differences are in data preprocessing and corruption strategy (Gong et al., 2024). The reported downstream gains are strongest in code-to-code settings: AST-T5 surpasses CodeT5 by 2 points in exact match score for Bugs2Fix and by 3 points in exact match score for Java-C# Transpilation in CodeXGLUE.

AstBERT adopts a different strategy by replacing raw code input with pruned AST information and continuing pre-training from CodeBERT. On code question answering, its reported accuracy is 0.588 on CoSQA and 0.571 on AliCoQA, compared with 0.532 and 0.526 for CodeBERT. On BigCloneBench clone detection, AstBERT reaches an F1 score of 0.973, compared with 0.971 for GraphCodeBERT and 0.965 for CodeBERT (Liang et al., 2022).

AST-Transformer addresses the computational burden of long linearized ASTs by restricting attention to the structural relations that matter most. It reports Java code summarization scores of BLEU 46.64, METEOR 28.49, and ROUGE-L 55.21, and Python scores of BLEU 33.82, METEOR 21.96, and ROUGE-L 47.14. The abstract emphasizes that the method can reduce $N$6 of the computational complexity in the encoding process (Tang et al., 2021).

ASTormer applies Formal-AST ideas to grammar-based semantic parsing. Relative to RATSQL+LSTM, ASTormer improves Spider development performance from 70.1 EM / 67.2 EX to 71.7 EM / 70.4 EX with BERT, and from 72.9 / 71.4 to 74.6 / 73.2 with ELECTRA. It also reduces training time per 100 iterations from 191.4s to 45.8s on Spider with ELECTRA-small (Cao et al., 2023).

In domain-specific code synthesis, the SVRF system combines ANTLR-based AST construction, AST serialization by depth-first traversal, retrieval-augmented generation, and an AST-weighted scoring function

$N$7

The reported benchmark grows from 400 initial paired examples to 741 after LLM-based augmentation, and the paper states that the AST approach yields around 40% enhancement in code generation compared with standard text-based fine-tuning (Abdelmalak et al., 1 Jul 2025).

4. Hybrid graphs, compact alternatives, and transformation frameworks

A persistent question in Formal-AST research is whether the AST should remain the primary representation or be replaced, compressed, or hybridized. The 2025 clone-detection study on hybrid graph representations treats AST as the foundational representation, then augments it with CFG, DFG, or FA-AST edges. Its main conclusion is architecture-dependent: GMN already reaches Precision 0.986, Recall 0.922, and F1 0.953 on AST alone, while its best FA-enhanced variant improves F1 only to 0.954; by contrast, GCN and GAT benefit most from AST + CFG + DFG, reaching F1 0.900 and 0.901 respectively, whereas FA-AST frequently introduces structural complexity that harms performance (Zhang et al., 17 Jun 2025). The same paper shows the cost side of enrichment: AST storage cost is 527.54 MB, while AST + FA rises to 1152.00 MB; average density increases from 0.0072 to 0.0202; and GMN inference time rises from 705.680 s to 748.402 s.

AlloyASG argues that plain ASTs can suffer from “exponential growth in their data size” because identical nodes are separately listed in the tree. It therefore defines a Complex-valued Structurally Balanced Abstract Semantic Graph, or CSBASG,

$N$8

with complex-weighted directed edges and structural balance constraint

$N$9

On 6,307 Alloy4Fun models, the paper reports an overall node reduction of 27.25% relative to the AST, and for predicate comparability it reports that 60.74% of oracle edges also appear in the student predicate, while 77.37% of the student predicate’s edges also appear in the oracle (Wu et al., 2024).

Older formal-language work makes the transformation viewpoint explicit. In textual DSL development, the target meta-model is the primary artifact, and semantic analysis is expressed as a sequence of AST transformations: text-to-AST handles syntax, and AST-to-model handles semantic analysis (0801.1219). In tree adjoining grammar, the derivation-tree literature introduces feature-based regular tree grammars so that feature-based TAG derivations can be represented while preserving derivation structure and accounting for feature unification (0804.4584). These papers indicate that Formal-AST is not limited to neural architectures; it also includes explicit meta-model transformations and formal grammar translations.

5. Autoformalization and AST-like formal objects in Lean

Autoformalization extends Formal-AST beyond source code into theorem-prover languages. “A New Approach Towards Autoformalization” decomposes research-level formalization into unlinked formalization, entity linking, and type correction, arguing that an initial formal skeleton can be useful even when names and types are unresolved (Patel et al., 2023). This decomposition is explicitly compatible with an AST-like view: unlinked formalization provides a structured theorem form, entity linking resolves symbolic references, and type correction makes the structure acceptable to the prover.

ATF operationalizes this with tool feedback. It uses Lean 4 compilers for syntax corrections and a multi-LLMs-as-judge setup for consistency validation, and the paper states: “During inference, ATF actively utilizes the developed tools to generate Lean 4 statements, iteratively modifying the output based on feedback until it passes both syntax and consistency checks” (Guo et al., 8 Oct 2025). The syntax tool groups statements based on import libraries, reducing average processing time per statement by 87%, from 6.2s to 0.8s. The consistency tool reports FPR 0.0829 for QWQ-32B, 0.0950 for Qwen3-32B, and 0.0579 for the ensemble vote. On CombiBench Pass@1 consistency, Full ATF reaches 65.38, compared with 41.68 for syntax check only and 23.69 for no tools.

StepFun-Formalizer reframes correctness as requiring both syntactic validity and semantic equivalence, evaluated using BEq. Its 32B model reaches BEq@1 scores of 40.5% on FormalMATH-Lite and 26.7% on ProverBench, while the 7B model reaches 38.3% and 25.1% (Wu et al., 6 Aug 2025). The paper explicitly argues that effective autoformalization requires both formal-language domain knowledge and informal-to-formal reasoning, which it interprets as constructing the right formal tree structure with the right objects, types, dependencies, and logical relations.

FormalEvolve further separates semantic consistency from prover effectiveness. Under a strict generator-call budget of $T$0, it reaches SH@100 of 0.580 on CombiBench and 0.849 on ProofNet, and improves theorem-complete@64 on CombiBench to 13/100, compared with 8/100 for Sample and 8/100 for Compile+Semantic Repair (Lu et al., 20 Mar 2026). Its symbolic AST rewrites therefore serve not only as a syntax-preserving perturbation mechanism, but as a search operator over prover-relevant formal statement variants.

FormalScience adds a human-in-the-loop perspective for scientific domains. Applied to physics, it produces FormalPhysics, a dataset of 200 university-level physics problems and solutions with Lean 4 formal representations. The paper reports perfect formal validity, average statement complexity of about 6.41 objects and 6.22 formulae per statement, and identifies semantic drift categories including notational collapse, abstraction elevation, proof strategy substitution, and implicit premise selection (Meadows et al., 24 Apr 2026). This shows that a compiled Lean object can still diverge substantially from the source scientific meaning.

6. Limits, misconceptions, and open problems

A recurrent misconception is that AST-based representation is uniformly superior to token-based representation. The most systematic empirical study directly challenges that claim: across code clone detection, code search, and code summarization, models trained with AST-based code representation consistently perform worse overall than models trained with token-based code representation (Sun et al., 2023). The same study reports that AST-based models are nevertheless better on important subsets: for Transformer-based clone detection, 1,165 clone pairs (2.14%) are uniquely detected by SBT, while 1,862 (3.42%) are uniquely detected by Token, and AST advantages become more visible when Jaccard token similarity is low.

A second misconception is that richer structural augmentation is automatically better. The hybrid-graph clone-detection study shows that this is false in a model-dependent way: AST + CFG + DFG consistently enhances GCN and GAT, but FA-AST often degrades performance, and GMN remains strongest even with standard AST representations (Zhang et al., 17 Jun 2025). This suggests that the utility of semantic augmentation depends on whether the downstream architecture can exploit the added relational structure without being overloaded by density and redundancy.

A third limitation concerns parseability and structural overhead. AST-T5 assumes the code is syntactically valid enough to parse with Tree-sitter (Gong et al., 2024). AstBERT prunes ASTs because they can be much longer than the original source code, with metadata-like nodes such as lineno and endlineno creating a token explosion problem (Liang et al., 2022). AST-Transformer is motivated by the same issue: ASTs are usually much longer than source code, and naïve linearization makes Transformer attention expensive (Tang et al., 2021).

Finally, formal validity is not equivalent to semantic faithfulness. FormalScience states that alignment scores and validity are nearly uncorrelated, with Spearman and Pearson coefficients near zero and $T$1 in the primary analysis (Meadows et al., 24 Apr 2026). ATF likewise shows that syntax checking alone helps, but semantic consistency checking is essential for faithful formalization (Guo et al., 8 Oct 2025). FormalEvolve sharpens the distinction further by arguing that semantic consistency does not imply prover effectiveness (Lu et al., 20 Mar 2026). Taken together, these results indicate that Formal-AST systems must be evaluated not only for grammatical well-formedness, but also for semantic preservation, search efficiency, and downstream utility.

7. Synthesis and trajectory of the field

Across these papers, Formal-AST emerges as a layered design space rather than a monolithic method. At one end are systems that keep standard sequence-to-sequence architectures and alter only preprocessing or objectives, as in AST-T5 (Gong et al., 2024). At another are encoder or decoder modifications that make attention tree-aware, as in AstBERT, AST-Transformer, and ASTormer (Liang et al., 2022). A third line replaces or augments the AST with alternative graph formalisms, including AST + CFG + DFG hybrids, CSBASG for Alloy, and feature-based regular tree grammars for TAG (Zhang et al., 17 Jun 2025). A fourth line treats formal languages such as Lean 4 as structured objects to be compiled, repaired, and searched over, even when the system does not expose an explicit AST decoder (Guo et al., 8 Oct 2025).

This suggests two durable conclusions. First, the strongest results arise when structural information is aligned with the task’s actual constraints: transpilation benefits from subtree-aware corruption, text-to-SQL benefits from grammar-constrained tree decoding, clone detection benefits from structure most clearly when lexical overlap is weak, and autoformalization benefits from compiler and semantic feedback because the target object is executable formal syntax. Second, structure alone is not sufficient. The literature repeatedly shows that parser choice, preprocessing, encoding strategy, semantic checking, and downstream search objectives determine whether Formal-AST yields a useful inductive bias or an additional source of complexity (Sun et al., 2023).