Logical Form Accuracy (LFAcc)

• Logical Form Accuracy (LFAcc) is an exact-match evaluation criterion that compares predicted logical forms to gold references after canonicalization.
• It is applied across various semantic parsing frameworks, including typed slot grammar, probabilistic CCG, and neural encoder-decoder models.
• LFAcc’s sensitivity to representational variations highlights the need for equivalence-aware metrics when multiple semantically equivalent forms exist.

Logical Form Accuracy (LFAcc) is an exact-match evaluation criterion for semantic parsing and related structured prediction tasks: an instance is counted as correct only when the predicted logical form matches the gold reference under the task’s stated equality relation. In "Statistical Parsing for Logical Information Retrieval" (Coppola, 12 Feb 2026), LFAcc is instantiated as exact syntactic equality over a typed, role-labeled logical language with identity canonicalization; closely related exact-match regimes appear in probabilistic CCG semantic parsing (Zettlemoyer et al., 2012), neural encoder-decoder parsing (Dong et al., 2016), and logical form generation (Liu et al., 2021), while ReCOGS shows that such metrics can be strongly affected by semantically irrelevant representational details (Wu et al., 2023).

1. Formal definition and scope

In the 2026 parsing framework, LFAcc for a set of $N$ sentences is defined as

$$\mathrm{LFAcc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\big[\mathsf{canon}(\hat{\mathrm{LF}}_i) \equiv \mathsf{canon}(\mathrm{LF}_i)\big],$$

where $\hat{\mathrm{LF}}_i$ is the logical form produced by the typed slot grammar, $\mathrm{LF}_i$ is the gold logical form, $\mathsf{canon}(\cdot)$ is the identity mapping in that work, and $\equiv$ is exact syntactic equality of the role-labeled, typed representation (Coppola, 12 Feb 2026). The correctness criterion is therefore exact textual equality after trivial whitespace normalization, not semantic equivalence.

This definition has clear antecedents in earlier semantic parsing work. "Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars" evaluates whether the top predicted logical form is a “completely correct semantics,” with normalization effectively treated as the identity and no described alpha-renaming, beta-reduction, or denotation-based equivalence (Zettlemoyer et al., 2012). "Language to Logical Form with Neural Attention" uses “Accuracy” on Jobs, Geo, and ATIS as the proportion of utterances correctly parsed to their gold logical forms, again by exact match after the paper’s preprocessing and re-lexicalization steps (Dong et al., 2016). "Improving Logical-Level Natural Language Generation with Topic-Conditioned Data Augmentation and Logical Form Generation" defines LF Acc. as exact string match between generated and gold logical form strings (Liu et al., 2021).

This suggests a stable operational core for LFAcc across multiple paradigms: the metric is primarily a measure of serialized structural identity, not of truth-conditional equivalence. What changes across papers is the representation being serialized, the preprocessing applied before comparison, and the extent to which the representation itself is canonical.

2. LFAcc in typed slot grammar parsing

In "Statistical Parsing for Logical Information Retrieval," LFAcc is evaluated in Experiment 2 on a typed slot grammar that deterministically compiles sentences into a logical language with role-labeled predicates, modal quantifiers, and typed variables (Coppola, 12 Feb 2026). The grammar test suite contains 12 test suites spanning 10 reasoning categories and totals 33 sentences. Of the 33 derived logical forms, 20 are ground facts and 13 are universally quantified rules. Example suites include and_gates/01_allies with 3 sentences and 3 facts, depth_chains/01_socrates with 2 sentences and 2 facts, and transitivity/01_taller with 2 sentences and 2 facts.

Ground-truth logical forms are provided as gold-standard annotations authored via LLM-assisted “vibe annotation” and manual quality gating. The grammar is deterministic and type-driven; ambiguity is resolved before grammar application by lexicon lookup on pre-disambiguated inputs, and function words are skipped automatically during matching. Because each sentence matches exactly one grammar rule and yields one logical form, the paper reports zero ambiguous parses. The resulting parsing outcome is “33 out of 33 sentences parsed. 33 out of 33 gold facts derived. Zero ambiguous parses. Zero extra facts,” which yields $\mathrm{LFAcc} = 33/33 = 100\%$ on that set (Coppola, 12 Feb 2026).

The exactness condition is stringent. For ground facts, correctness requires exact predicate name, exact typed role names, exact entity bindings, and exact negation flag. For quantified rules, correctness requires exact quantifier word and variable list, exact premise and conclusion predicates with their role labels, and consistent variable binding. The paper’s examples illustrate the point directly:

• Ground fact: “Alice is taller than Bob” maps to taller_than(theme: alice, reference: bob). Reversing the roles is incorrect.
• Negated fact: “Zeus is not mortal” maps to not mortal(theme: zeus). Omitting negation is incorrect.
• Universal rule: “All men are mortal” maps to always [x:e]: man(theme: x) -> mortal(theme: x). The quantifier, variable name, and role labels must match exactly.
• Reciprocal rule: “If two people trust each other, they are allies” maps to always [x:e, y:e]: trust(agent: x, patient: y) and trust(agent: y, patient: x) -> allies(agent: x, patient: y).

The architecture surrounding this evaluation is explicitly layered: LLM preprocesses, grammar parses, LLM reranks, and QBBN infers. A separate ambiguity experiment shows GPT-4 achieving 19/20 correct on genuinely ambiguous PP-attachment sentences, while direct zero-shot LLM dependency parsing yields UAS $= 12.4\%$ and LAS $= 7.9\%$, far below the Stanford Parser’s UAS $= 91.3\%$ and LAS $$\mathrm{LFAcc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\big[\mathsf{canon}(\hat{\mathrm{LF}}_i) \equiv \mathsf{canon}(\mathrm{LF}_i)\big],$$0 (Coppola, 12 Feb 2026). The significance of these numbers is architectural rather than merely comparative: high LFAcc is obtained not because unconstrained neural systems emit precise structures reliably, but because disambiguation and deterministic compilation are separated.

3. Canonicalization, equivalence, and what counts as “the same” logical form

In the 2026 system, the logical language is fully typed and role-labeled, and the grammar emits a single canonical textual representation per sentence (Coppola, 12 Feb 2026). The paper states that there is no alpha-equivalence, no beta- or eta-reduction, no commutativity or associativity normalization for connectives, no scope canonicalization, and no role-label invariance beyond the language’s own labels. Role names such as agent, patient, theme, and reference are named and their values are not interchangeable. Variable renaming is neither needed nor permitted in evaluation because the parser uses fixed variable names per rule template. Modal quantifiers such as always, usually, sometimes, and never map internally to noisy-OR weights for inference, but LFAcc is computed on the surface logical syntax rather than on those numeric values.

ReCOGS argues that this kind of exact-match LFAcc is often too syntactic. It formalizes an equivalence-aware alternative in which a predicted LF is correct if it is semantically exact under a bijection between variables and set equality of conjuncts, thereby capturing alpha-equivalence and conjunction commutativity (Wu et al., 2023). The paper’s SEM criterion requires that there exist a bijection $$\mathrm{LFAcc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\big[\mathsf{canon}(\hat{\mathrm{LF}}_i) \equiv \mathsf{canon}(\mathrm{LF}_i)\big],$$1 such that, after applying $$\mathrm{LFAcc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\big[\mathsf{canon}(\hat{\mathrm{LF}}_i) \equiv \mathsf{canon}(\mathrm{LF}_i)\big],$$2, the predicted and gold logical forms have exactly the same set of conjuncts. This treats table(46) AND sturdy(46) and table(1) AND sturdy(1) as equivalent, and it abstracts away from conjunct order when conjunction is semantically commutative.

The contrast is substantive. In the typed slot grammar setting, exact structural equality is appropriate because the system is designed to emit one canonical form for each sentence. In ReCOGS, by contrast, the benchmark’s original logical forms tie variable names to word positions and impose semantically irrelevant linearization conventions, so string-match LFAcc confounds semantic interpretation with representational accident (Wu et al., 2023). A plausible implication is that LFAcc is most defensible when the logical language and compiler jointly eliminate arbitrary representational variation; when they do not, equivalence-aware scoring becomes methodologically preferable.

4. LFAcc and neighboring evaluation metrics

Across the literature, LFAcc coexists with several distinct evaluation notions that should not be conflated. The major differences concern what is being compared: logical-form strings, executable outcomes, syntactic trees, or equivalence classes of logical forms.

Setting	Primary correctness notion	Distinct companion metric
Typed slot grammar parsing	Exact textual equality of typed, role-labeled LF	UAS/LAS and PP-attachment accuracy
PCCG semantic parsing	Exact logical-form match	Precision and recall over parsed vs. correct
Neural attention semantic parsing	Exact logical-form match on Jobs/Geo/ATIS	Answer accuracy for some baselines
Logical form generation	Exact string match of serialized programs	Execution accuracy
ReCOGS	Equivalence-aware semantic exact match	String-match LFAcc as a foil

In the 2026 parsing system, UAS and LAS are explicitly different from LFAcc: UAS/LAS measure token-level tree attachment quality, whereas LFAcc measures exact correctness of a typed, role-labeled semantic representation (Coppola, 12 Feb 2026). In the PCCG framework, precision is defined as the number of correct predictions divided by the number of parsed sentences, and recall as the number of correct predictions divided by the total number of examples; correctness still means exact logical-form match (Zettlemoyer et al., 2012). In the neural attention work, “Accuracy” on Jobs, Geo, and ATIS is exact-match LFAcc, but some compared systems report denotation accuracy because they output answers directly rather than logical forms (Dong et al., 2016). In the logic-to-text dual task, Exec. Acc. counts a logical form as correct if it can be successfully executed on the table, a criterion the paper itself notes can reward spurious outputs that are executable but not semantically aligned with the input text (Liu et al., 2021).

The central methodological point is that these metrics answer different questions. LFAcc asks whether the predicted structure is exactly the intended logical form. Denotation or execution measures ask whether the structure, correct or not, yields an acceptable outcome under some executor. UAS and LAS ask whether the syntactic tree is locally correct. Equivalence-aware metrics such as SEM ask whether two distinct strings encode the same meaning. Comparisons across papers are therefore only informative when the equality relation and task objective are aligned.

5. Modeling frameworks in which LFAcc is optimized

The probabilistic CCG framework of 2012 couples LFAcc to a conditional log-linear model over syntax-semantics derivations (Zettlemoyer et al., 2012). The model defines

$$\mathrm{LFAcc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\big[\mathsf{canon}(\hat{\mathrm{LF}}_i) \equiv \mathsf{canon}(\mathrm{LF}_i)\big],$$3

with lexical features counting lexical-entry usage in derivations. Prediction marginalizes over derivations yielding the same logical form and selects $$\mathrm{LFAcc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\big[\mathsf{canon}(\hat{\mathrm{LF}}_i) \equiv \mathsf{canon}(\mathrm{LF}_i)\big],$$4. Training maximizes $$\mathrm{LFAcc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\big[\mathsf{canon}(\hat{\mathrm{LF}}_i) \equiv \mathsf{canon}(\mathrm{LF}_i)\big],$$5, and lexicon structure is induced by GENLEX plus pruning. Evaluation remains exact-match: on Geo880 the paper reports $$\mathrm{LFAcc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\big[\mathsf{canon}(\hat{\mathrm{LF}}_i) \equiv \mathsf{canon}(\mathrm{LF}_i)\big],$$6 and $$\mathrm{LFAcc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\big[\mathsf{canon}(\hat{\mathrm{LF}}_i) \equiv \mathsf{canon}(\mathrm{LF}_i)\big],$$7, and on Jobs640 it reports $$\mathrm{LFAcc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\big[\mathsf{canon}(\hat{\mathrm{LF}}_i) \equiv \mathsf{canon}(\mathrm{LF}_i)\big],$$8 and $$\mathrm{LFAcc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\big[\mathsf{canon}(\hat{\mathrm{LF}}_i) \equiv \mathsf{canon}(\mathrm{LF}_i)\big],$$9, with recall limitations attributed primarily to GENLEX coverage.

The neural encoder-decoder framework of 2016 retains exact-match LFAcc while replacing grammar induction with sequence or tree generation (Dong et al., 2016). On Jobs, Geo, and ATIS, the reported “Accuracy” is the proportion of utterances whose predicted logical form exactly matches the gold after delexicalization and re-lexicalization. Seq2Seq and Seq2Tree models are evaluated under greedy decoding with no explicit grammar or type constraints; attention and argument identification materially affect LFAcc. On Geo, for example, Seq2Seq reaches 84.6 and Seq2Tree 87.1, while removing attention reduces performance substantially. The paper therefore associates LFAcc improvements with learned alignment and hierarchical output structure rather than with explicit symbolic compilation.

The 2021 logical form generation framework uses LFAcc in the reverse direction, from text and table to serialized logical forms (Liu et al., 2021). Here LFAcc is exact string match, beam search of size 3 is used in decoding, and model selection for LG is based on validation LFAcc. The paper reports GPT-2 at 67.03 LFAcc and its proposed method at 67.95 in the full-data setting, with larger gains in the 1k few-shot setting, 41.30 to 45.51. Ablations show that removing back-translation causes the largest LFAcc drop, from 67.95 to 64.56. Within that framework, LFAcc becomes a measure of whether the model can reproduce the exact serialized program, not merely a valid or executable one.

Taken together, these frameworks show that LFAcc is model-agnostic in one narrow sense: it can supervise symbolic grammars, probabilistic grammar induction, neural sequence transduction, and dual generation settings alike. At the same time, the source of error differs by framework—coverage in GENLEX, alignment and rare-word failures in neural decoders, noise in pseudo-labeled augmentation, or upstream disambiguation in pipeline systems.

6. Limitations, controversies, and benchmark design implications

The strongest limitation of LFAcc is brittleness under representational variation. ReCOGS demonstrates that exact string-match LFAcc can change dramatically when semantically inert tokens are removed, when conjunction order is treated as irrelevant, or when variable names are decoupled from word positions (Wu et al., 2023). The paper reports large score changes under meaning-preserving transformations, including lexical-split gains such as Transformer performance moving from 62.4% to 81.1% after removing {x, _} in one setting, and structural-split changes such as Object PP $\hat{\mathrm{LF}}_i$0 Subject PP moving from 0.0% to 82.7% under a participial verb phrase modification. Its conclusion is not that LF evaluation is unnecessary, but that naïve string-match LFAcc can overweight punctuation, variable notation, and linearization conventions.

The typed slot grammar results present a different limitation profile (Coppola, 12 Feb 2026). The reported LFAcc of 100% is obtained on a small, curated, pre-disambiguated set of 33 sentences, and the grammar covers 12 of the 22 reasoning categories used in the inference tests. The paper explicitly identifies upstream disambiguation errors, coverage gaps, and unreported end-to-end performance on open-domain text as remaining issues. It also notes that PP attachment errors upstream would lower LFAcc because role bindings in the emitted logical form would then be wrong. Perfect LFAcc in that experiment therefore demonstrates exactness under controlled conditions, not unrestricted semantic parsing competence.

A common misconception is that logical inference machinery validates parsing correctness. In the 2026 architecture, QBBN is downstream of parsing and is not used to judge LF equivalence for LFAcc (Coppola, 12 Feb 2026). The NEG factor identity $\hat{\mathrm{LF}}_i$1 supports probabilistic inference and backward reasoning through $\hat{\mathrm{LF}}_i$2-messages, but it plays no role in determining whether a parsed logical form matches the gold reference. The verification step for LFAcc is syntactic exact match, not semantic theorem proving.

The broader methodological implication is twofold. First, exact-match LFAcc is appropriate when a representation is intentionally canonical, deterministic, and free of arbitrary naming variation. Second, when logical forms admit multiple semantically equivalent serializations, equivalence-aware metrics such as SEM, or denotation-based checks where feasible, are better aligned with the target notion of semantic correctness (Wu et al., 2023). This suggests that benchmark design and LFAcc computation cannot be separated: the validity of the metric depends on the invariances encoded by the logical form language itself.