ConNF Benchmark Overview

• ConNF Benchmark is an out-of-distribution evaluation suite that tests retrieval-augmented autoformalization on advanced, domain-specific mathematical statements.
• It leverages the DRIFT framework to decompose, retrieve, illustrate, and formalize proofs, ensuring precise dependency retrieval and context provisioning.
• Empirical results show marked improvements in semantic equivalence and type-check pass rates, highlighting its capacity to nearly sextuple baseline performance.

ConNF Benchmark is an out-of-distribution (OOD) evaluation suite designed to assess the performance of retrieval-augmented automatic formalization systems when tested on mathematical statements drawn from a domain not previously encountered by either the retrievers or the formalizers. The ConNF benchmark is based on the con‑nf library, which is specifically curated to formalize the consistency proof for Quine’s New Foundations set theory. Its unique structure and content rigorously challenge LLMs in the autoformalization task by necessitating cross-domain generalization and precise dependency retrieval.

1. Benchmark Structure and Domain

The ConNF benchmark draws on the con‑nf library, containing 1,348 formal objects, each codified for the purposes of formalizing advanced set-theoretic results in Lean. Unlike conventional benchmarks such as ProofNet and MiniF2F-test, ConNF's items are sourced from a specialized mathematical subfield—Quine’s New Foundations—which is less represented in widely used libraries like Mathlib. This results in an OOD scenario: during evaluation, neither retrieval models nor formalizers have previous exposure to the formal objects present in the test split.

This suggests the benchmark serves as a stress test for generalization and adaptive retrieval, prioritizing the evaluation of algorithms on unfamiliar mathematical territories over in-distribution accuracy.

2. DRIFT Framework and Decomposition for ConNF

DRIFT (Decompose, Retrieve, Illustrate, then Formalize) is a retrieval-augmented framework engineered to enhance premise selection and context provisioning for autoformalization tasks on challenging benchmarks such as ConNF. The framework operates in multiple stages:

• Decomposition: The informal mathematical statement (IF) is split into atomic sub-queries, each denoted as a pair $(q_i, \hat{l}_i)$, where $q_i$ is a natural language excerpt and $\hat{l}_i$ a predicted formal representation. This mapping is formalized as $$\text{Decomposer}(IF) \rightarrow Q = \{(q_i, \hat{l}_i)\}_{1}^n$$.
• Retrieval: Each $q_i$ is embedded with a dense passage retriever (BGE‑M3 encoder) and matched to a formal object $p_i$ in the con‑nf library using cosine similarity:

$$p_i = \arg\max_{p \in \mathbb{D}} \phi(q_i, p), \quad \phi(q_i, p) = \frac{q_i \cdot p}{\|q_i\|\|p\|}$$

• Illustration: The Illustrate module employs a greedy algorithm to select a set of illustrative theorems that maximize premise coverage, scaffolding the demonstration of retrieved objects in real-world proof contexts.

3. Performance Metrics and Empirical Findings

The ConNF benchmark emphasizes two principal metrics:

• BEq+@10: Measures semantic equivalence between LLM-produced formal statements and ground-truth, within the top-10 sampled formalizations.
• Typecheck Pass Rate: Assesses syntactic validity by verifying Lean type-checking of outputs.

Empirically, DRIFT achieves a BEq+@10 improvement of 37.14% with GPT‑4.1 and 42.25% with DeepSeek‑3.1 over zero-shot baselines, where traditional approaches yield BEq+ scores often below 10%. In several configurations, DRIFT's multi-stage retrieval and illustrative prompting push pass@10 BEq+ values above 60%. A plausible implication is that precise decomposition and targeted context supplementation can nearly sextuple performance in extreme OOD settings.

4. Comparative Analysis with Other Benchmarks

ConNF presents a notably higher bar for generalization than in-distribution evaluations:

• ProofNet: Contains undergraduate-level theorems with strong overlap to Mathlib, enabling smoother premise identification and formalization.
• MiniF2F-test: Consists of self-contained problems with minimal dependency (average 0.43 per problem), yielding little marginal benefit from external retrieval. DRIFT produces nuanced improvements on ProofNet (doubling F1 versus DPR baselines) and limited value on MiniF2F-test due to the latter’s intrinsic simplicity, but its mechanism is most consequential on ConNF, where the OOD nature and research-level content demand highly adaptive retrieval.

5. Model-Specific Adaptation and Knowledge Boundaries

DRIFT explicitly aligns retrieval strategies to the “knowledge boundaries” of each LLM formalizer. While models such as DeepSeek‑3.1 excel at parametric zero-shot formalization, GPT-4.1 surpasses them when enhanced via external retrieval in OOD scenarios such as ConNF. The predicted formal representations ($\hat{l}_i$) appended to each sub-query facilitate accurate passage retrieval even if the internal representations of the LLM are imperfect. The context-rich prompts assembled by DRIFT (including $\mathcal{J}$, $\mathcal{R}_{DRIFT}$, $\mathcal{T}$, and $IF$) operationally tailor formalization strategies to model-specific strengths, improving both syntactic and semantic fidelity.

6. Illustrative Theorems and Demonstrative Scaffolding

Standard premise retrieval, which delivers formal definitions or declarations, is insufficient for robust autoformalization because such objects do not expose usage patterns within proof environments (e.g., variable instantiation from type declarations). DRIFT’s Illustrate module overcomes this by selecting small sets of theorems that exemplify contextual usage of retrieved objects. For instance, the retrieval of the ZMod type is paired with a theorem that demonstrates its integration within a formal proof. This pairing of retrieval and illustration mitigates common formalization errors, such as mismatched syntax and omitted dependencies.

7. Tooling, Libraries, and Methodological Implications

DRIFT’s application to ConNF leverages modern embedding-based retrieval: BGE‑M3 for passage embedding, cosine similarity for match scoring, and structured mathematical libraries for context grounding. For in-distribution tasks Mathlib serves as the underlying corpus; for ConNF, the dedicated con‑nf library enables isolated OOD evaluation. The comprehensive prompt structure $$(\mathcal{J} \oplus \mathcal{R}_{DRIFT} \oplus \mathcal{T} \oplus IF)$$ integrates instruction, evidence, illustration, and original statement, optimizing the conditioning of the LLM for high-fidelity formal output.

In sum, the ConNF benchmark functions as an exacting testbed for evaluating autoformalization frameworks against advanced, domain-specific mathematics, demonstrating the necessity of decomposition, adaptive retrieval, and contextual illustration, as realized by the DRIFT framework (Zhang et al., 12 Oct 2025).