CoqGym Dataset for Automated Theorem Proving

• CoqGym is a structured dataset containing real and synthetic proofs from 123 Coq projects, serving as a benchmark for tactic-generation models.
• It provides detailed, stepwise proof-state representations including kernel-level ASTs and Lisp-style serializations, facilitating rigorous model training.
• The platform supports both supervised and reinforcement learning approaches, evidenced by improvements in models like ASTactic and NeuroTactic.

CoqGym is a large-scale, structured dataset and interactive environment for machine learning research in automated theorem proving with the Coq proof assistant. Constructed from 123 real-world Coq projects and designed explicitly for end-to-end training and evaluation of tactic-generation models, CoqGym exposes detailed, stepwise representations of proofs and proof states suitable for both supervised and reinforcement learning. By incorporating both human-written and synthetic proofs, as well as complete kernel-level abstract syntax trees (ASTs) and serializations, CoqGym has become a central benchmark and resource for learning-based research on interactive theorem proving in higher-order logic (Yang et al., 2019). Its infrastructure and statistical depth have enabled advances in tactic prediction and premise selection, as demonstrated by models such as ASTactic and NeuroTactic (Li et al., 2021).

1. Dataset Composition and Construction

CoqGym’s data collection begins with the Coq standard library and all packages indexed in the official Coq Package Index. This results in a target corpus of 123 projects, totaling approximately 3,061 .v source files. Only those projects compiling under Coq 8.9 or 8.10 without intervention are included; sources dependent on unavailable or specialized libraries are excluded. To prevent contamination, projects are partitioned at the project level into disjoint training, validation, and test splits. Disjointness at the project—rather than file or lemma—level ensures that neither cross-lemma nor cross-tactic leakage can occur between splits (Yang et al., 2019).

Each proof in the dataset is executed using a patched version of Coq together with SerAPI, enabling the extraction of kernel-level ASTs for all terms, tactics, and full proof-state snapshots at every step. These are serialized as Lisp-style s-expressions directly from the OCaml ASTs, ensuring perfect fidelity with Coq’s internal representation and bypassing the need for third-party parsers. Proofs employing goal-selector syntax (where a tactic is applied simultaneously to multiple goals, making parent-child goal relationships ambiguous) are excluded, amounting to less than 1% of the total. Trivial or automatically generated proofs, such as those containing zero tactics, are also filtered out.

In addition to human-written proofs, CoqGym introduces synthetic proof augmentation: every intermediate goal within a human proof is treated as a candidate mini-theorem, extracting “short” proofs of exactly 1, 2, 3, or 4 steps. This is performed by reusing local hypotheses at that point and completing the proof with Coq’s auto tactic. The result is an augmented dataset containing 159,761 one-step, 109,602 two-step, 79,967 three-step, and 61,126 four-step synthetic proofs, providing a curriculum for models sensitive to proof length.

2. Statistical Characteristics

Across its 123 projects, CoqGym contains 70,856 human-written proofs involving approximately 645,000 Ltac tactics, for an average of 9.1 tactics per proof (standard deviation ≈ 7.3). The distribution of proof length is heavy-tailed: approximately 45% of proofs have length $\ell \leq 5$, while around 20% have $\ell > 15$. Additional key statistics include:

• Average intermediate goals per proof: 8.7 (std 6.1)
• Mean environment size (global premises): 10,350.3 (std ≈ 8,200)
• Average local context size: 5.6 (std 4.2)
• Tactic AST: mean 2.0 tokens per tactic, depth 1.9; 53% of tactics accept at least one argument (identifier, integer, hint database, etc.)

Dataset splits comprise 43,844 training proofs, 13,875 validation proofs, and 13,137 test proofs. The synthetic augmentation adds 410,456 “short” proofs in total.

Split	Projects	Proofs	Tactic Steps
Train	70	≈50,000	—
Validation	26	≈8,000	—
Test	27	≈13,000	—
Total	123	70,856	645,000

3. Data Format and Environment

CoqGym’s content resides in a structured directory under coqgym_data/, with key components including:

• src/: Original Coq .v source files
• sexp/terms/: Serialized Coq term ASTs (one per .v file)
• sexp/tactics/: Serialized ASTs of Ltac tactics
• sexp/states/: Proof-state dumps (environment, goal list, and context)
• structure/proofs.json: Structured proof trees with goal IDs, tactic applications, and parent/child relationships

All serializations use Lisp-style s-expressions generated via SerAPI from Coq’s kernel-level ASTs, ensuring no semantic information loss.

A Python module provides parsing of these s-expressions into in-memory AST objects, supporting direct interaction and loading of the dataset for machine learning pipelines. The dataset also specifies a context-free grammar (CFG in extended BNF) for tactics as used by ASTactic, with non-terminals such as tactic_expr, intro, apply, and restricted terminal symbols (identifiers, small integers {1,2,3,4}, hint-db names, and common modifiers). Each atomic Ltac instruction constitutes a single model “action”; compound tactics (e.g., tac1; tac2) are not generated, focusing the prediction space.

Sample interface code for dataset loading and proof inspection is provided:

from coqgym.dataset import CoqDataset dataset = CoqDataset(root_dir=COQGYM_ROOT) train_proofs = dataset.get_split('train') for proof in train_proofs[:2]: print("Proof ID:", proof.uid) print(" Environment premises:", len(proof.global_premises)) print(" Local context size:", len(proof.local_hypotheses)) print(" Steps:", len(proof.tactics)) # each step: proof.steps[i].goal_ast, proof.steps[i].tactic_ast

The environment enables end-to-end agent interaction by wrapping a Coq process in a standard RL interface:

from coqgym import CoqEnv env = CoqEnv(project_path='/path/to/coq/project') state = env.reset(theorem_name='add_assoc') while not state.is_done(): action = agent.predict(state) # an Ltac AST state, reward, done, info = env.step(action)

4. Benchmarking and Research Applications

CoqGym provides a comprehensive benchmark for tactic-generation models in interactive theorem proving. Notable models trained and evaluated on CoqGym include ASTactic, which generates tactics as programs in the form of abstract syntax trees (Yang et al., 2019), and NeuroTactic, which leverages a GNN-based encoder with graph contrastive pre-training (Li et al., 2021). ASTactic achieves a test set top-1 tactic prediction accuracy of 18.20%, while NeuroTactic improves this to 21.15%, amounting to a 16.2% relative increase.

NeuroTactic’s usage of CoqGym highlights several properties:

• ASTs extracted from Coq via SerAPI and represented as undirected graphs enable GNNs to embed theorem and premise semantics effectively.
• Graph contrastive pre-training uses premise selection as a pretext task, with the InfoNCE objective enforcing alignment between theorem and premise embeddings.
• Fine-tuning these GNNs for tactic prediction results in measurable performance improvements.

Premise selection—selecting the most relevant lemma or hypothesis from an average environment of 10,350 global premises—remains a challenging high-level reasoning task that CoqGym’s structure directly supports. The synthetic proof augmentation acts as a natural curriculum for agents by exposing them to ever longer proof sequences.

Potential further uses include transfer learning, development of premise-selection algorithms, benchmarking RL-based proof search, and comparative studies against other proof assistants by transforming CoqGym data into alternate formal systems.

5. Challenges, Limitations, and Future Directions

Several limitations are inherent to CoqGym. The action space is constrained to a fixed subset of core Ltac tactics to maintain tractable search and learning; custom tactics and combinators are excluded, limiting expressivity. Compound tactic composition is not part of the dataset’s grammar, and integer arguments to tactics are capped at {1,2,3,4}. The burden of selecting relevant premises—out of thousands in scope at each step—is shifted onto the learner.

Synthetic curriculum proofs omit many technique-rich or domain-specialized tactics. AST-level representations preclude random graph augmentations, as such perturbations would typically invalidate kernel-level grammar. Present evaluation metrics do not include statistical significance testing, and only one premise per tactic step is considered in contrastive learning in NeuroTactic, whereas many real proofs involve multiple premises.

Suggested directions include integration of richer semantic annotations (types and dependency graphs), simplification of ASTs to control fan-out and facilitate GNN message passing, and cross-system benchmarking by mapping proof scripts to alternative assistants. CoqGym’s flexibility and scale also underwrite new research in premise selection, tactic generation, transfer learning, and cross-domain generalization in formal reasoning (Yang et al., 2019, Li et al., 2021).

6. Summary and Availability

CoqGym establishes itself as the foundational benchmark and environment for machine learning on interactive theorem proving with Coq. By combining source-level representations, serialized kernel ASTs, structured proof trees, and a Python interface for data and environment manipulation, CoqGym enables the training and evaluation of both supervised and RL models at scale. The dataset’s rigorous filtering, augmentation, and documentation standards ensure consistent and meaningful evaluation. Source code and data are openly available at https://github.com/princeton-vl/CoqGym, providing a foundation for ongoing advances in automated reasoning and program synthesis with proof assistants (Yang et al., 2019, Li et al., 2021).