SyntaxEval: Syntax & Semantic Evaluation

• SyntaxEval is a methodology that systematically assesses models' syntactic reconstruction via AST-based masking and causal inference.
• It quantifies the impact of syntax masking on model performance using metrics like Jaccard similarity for detailed evaluation.
• Beyond model evaluation, SyntaxEval serves as a formal schema for specifying and reasoning about syntax-driven symbolic algorithms.

SyntaxEval is an umbrella term for a family of methodologies and frameworks that systematically evaluate, specify, or reason about algorithms that manipulate, analyze, or reconstruct syntax—particularly in settings where both syntactic and semantic correctness are critical. In contemporary research, SyntaxEval serves both as a technique for diagnosing the linguistic competence of models (e.g., masked LLMs on code) and as a foundational schema in symbolic computation for rigorously linking syntactic manipulations with their intended semantics.

1. SyntaxEval in Model Evaluation: Formal Framework

The SyntaxEval methodology introduced by "Which Syntactic Capabilities Are Statistically Learned by Masked LLMs for Code?" provides a two-part framework aimed at quantifying (a) the fine-grained ability of Masked LLMs (MLMs) to reconstruct masked nodes of Abstract Syntax Trees (ASTs), and (b) the causal effect of such masking on model performance (Velasco et al., 2024).

Let $s$ be a code snippet with AST $\mathrm{AST}(s)$, and $C$ the set of node types from the Python context-free grammar. SyntaxEval defines a masking indicator sequence $M = (M_1, \dots, M_n)$—with $M_j = 1$ if token $s_j$ belongs to a node of type $C$, else $0$—so the masked input is

$$\tilde{s}_j = \begin{cases} \langle \text{mask} \rangle & \text{if } M_j = 1 \ s_j & \text{otherwise} \end{cases}$$

The MLM is then tasked to predict tokens at each $\langle \text{mask} \rangle$. Crucially, the evaluation leverages causal inference, defining treatment $T=1$ (AST-based masking) versus $T=0$ (random masking of equal size), and computes an Average Treatment Effect (ATE)

$\tau = \mathbb{E}[Y_{T=1}] - \mathbb{E}[Y_{T=0}]$

where $Y$ is a metric such as Jaccard similarity between inorder traversals of $\mathrm{AST}(s)$ and $\mathrm{AST}(\hat{s})$. This design distinguishes genuine syntactic competence from surface-level pattern completion.

2. AST-Driven Masking and Causal Evaluation Protocol

The AST-based masking core of SyntaxEval is realized by traversing $\mathrm{AST}(s)$ to locate all node instances of a target type $c \in C$, and then masking all tokens covered by those nodes. Formally,

$$M_j = \mathbf{1} \left( \exists n \in \mathrm{AST}(s): \mathrm{type}(n) = c \wedge j \in \mathrm{span}(n) \right)$$

The resulting masked input, $\tilde{s}$, is then subjected to MLM reconstruction. This infrastructure allows granular probing of, for instance, "for_statement" or "comparison_operator" capabilities.

To isolate the causal effect of masking syntactic constructs, SyntaxEval builds on explicit causal modeling. The evaluation conditions on confounders $Z$ (e.g., AST depth, code length, cyclomatic complexity) and exploits propensity-score weighting to correct for imbalances between treated and control groups. The approach ensures that performance decrements under AST-based masking cannot be ascribed to mere snippet complexity or length, but are attributable to the model's handling of the specified node type.

3. Empirical Findings: MLM Syntactic Competence

Empirical evaluation involved $\approx 50,000$ Python snippets sampled from high-profile GitHub repositories and two pretrained models: CodeBERTa-small-v1 (84M params) and codebert-base-mlm (125M params), both pretrained on CodeSearchNet (Velasco et al., 2024). The primary observation is that, while both models reach high median similarities ($> 0.8$) under random masking, their performance under AST-based masking is consistently worse.

Key findings for node-wise ATEs ($\tau$) are as follows:

Node Type	$\tau_\text{Jaccard}$ (M₁)	$\tau_\text{Levenshtein}$ (M₁)	$\tau_\text{Sørensen\text{–}Dice}$ (M₁)
comparison_operator	−0.186	−0.179	−0.126
for_statement	−0.269	−0.193	−0.243
boolean_operator	−0.083	−0.069	−0.048
identifier	+0.016	+0.001	+0.010

Negative ATEs ($\tau$) for almost all node types suggest that the MLMs do not statistically learn AST-defined constructs; in fact, masking these positions causes greater degradation than random masking. For the "for_statement" node, the effect is particularly pronounced ($\tau \approx -0.27$). "Identifier" emerges as the only node type where $\tau$ is (slightly) positive.

Cumulative distribution analyses further reveal that high reconstruction similarity (e.g., 90% Jaccard) is attained in less than 50% of cases for complex node types like "for_statement," underscoring the structural limitations of current MLMs.

4. Implications for Model Design and Evaluation

The SyntaxEval results directly challenge the claim—proposed in prior probing/explainability research—that MLMs reliably encode syntactic structure in their learned representations. The dominant MLM pretraining objective (15% random mask rate, as in CodeSearchNet) does not induce robust syntactic discrimination at the level of AST node types. Consequently, SyntaxEval identifies the need for inductive biases and objectives that target grammar-based, structured masking, rather than token-level randomness.

A plausible implication is that future MLMs for code should leverage pretraining protocols that exploit or mirror AST-level regularities, and that structured evaluation frameworks like SyntaxEval are essential for accurate measurement of model progress. The methodology also provides a template for extending similar causal-inference based analyses to domains beyond syntax, such as data-flow or control-flow semantics (Velasco et al., 2024).

5. SyntaxEval as a Symbolic-Algorithm Specification Schema

In the context of formal theories and computer algebra, SyntaxEval refers to a logic-based apparatus for specifying and analyzing syntax-based mathematical algorithms (SBMAs) (Carette et al., 2019, Farmer, 2013). In Church’s type theory extended with quotation and partial evaluation, a dedicated "syntax" type $\varepsilon$ is introduced, along with quotation operator $\lceil\cdot\rceil$ and typed evaluation maps $$\operatorname{eval}_\alpha: \varepsilon \rightharpoonup \alpha$$. The critical axioms are:

• Quotation Axiom: For any $e:\alpha$, $\operatorname{isExpr}^{\alpha}(\lceil e \rceil)$.
• Disquotation/Evaluation: For any $e:\alpha$, $\operatorname{eval}_\alpha(\lceil e \rceil) = e$, where defined.
• Definedness: $\operatorname{eval}_\alpha(s)$ is defined iff $s$ encodes a well-typed term of type $\alpha$.

This formal foundation allows the direct specification of SBMAs, such as symbolic differentiation,

$$\forall s:\varepsilon.\; \text{if } \operatorname{isDiffExpr}(s) \text{ then } \left(\operatorname{isDiffExpr}(\operatorname{diff}(s)) \wedge \forall a:\mathbb{R}. (\operatorname{eval}_\rightarrow(s)(a))\downarrow \implies \operatorname{deriv}(\operatorname{eval}_\rightarrow(s),a) = \operatorname{eval}_\rightarrow(\operatorname{diff}(s))(a) \right)$$

with partiality to handle ill-formed or non-differentiable cases (Carette et al., 2019, Farmer, 2013).

6. Syntax Framework Architecture: Local vs. Global Quotation/Evaluation

SyntaxEval systems can be instantiated either as "local" or "global" syntax frameworks (Farmer, 2013).

• Local approach: Embeds a fragment-specific inductive type (e.g., for polynomials) and meta-level quote/eval functions. This design is minimal and avoids logical paradoxes but cannot prove the law of disquotation internally; each syntax domain requires bespoke infrastructure.
• Global approach: Integrates quote and eval as first-class (object-level) operators in the underlying logic (as in Chiron or CTT$_{\mathrm{uqe}}$), enabling reflective reasoning about syntax for all expressions within the theory, at the cost of increased logical complexity and the necessity to restrict eval's domain to remain consistent.

A two-tiered, practical recommendation is to use local deep embeddings where feasible, resorting to global reflection only for designated fragments and with restriction to prevent paradoxes (Farmer, 2013).

7. Conceptual and Practical Significance

SyntaxEval methodologies unify the specification, mechanized reasoning, and rigorous evaluation of algorithms or models that fundamentally interact with syntax. In model evaluation, SyntaxEval’s causal-inference-based protocol exposes systematic deficiencies in syntactic learning of current MLMs for code, establishing a higher standard for syntactic competence assessment (Velasco et al., 2024). In formal methods and symbolic computation, SyntaxEval-style frameworks provide transparent logic-based mechanisms for capturing the interplay between symbolic manipulation and semantic meaning, supporting the correctness analysis of computer algebra systems and similar applications (Carette et al., 2019, Farmer, 2013).

The general principle underlying SyntaxEval in both contexts is the explicit, formal linkage between syntax-level manipulations (actual parse trees, node types, or quoted/eval’d structures) and the corresponding, meaning-level outcomes—ensuring that reasoning, automation, and evaluation are all subject to precise, verifiable criteria.