---
title: AutoOverlap in Lean 4 Autoformalization
url: https://www.emergentmind.com/topics/autooverlap
type: topic
---

# AutoOverlap in Lean 4 Autoformalization

AutoOverlap, as introduced in the context of Lean 4 autoformalization, is the data-overlap fraction between the prompt set used in supervised fine-tuning (SFT) and the prompt set used in Group Relative Policy Optimization (GRPO). In "SFT-GRPO Data Overlap as a Post-Training Hyperparameter for Autoformalization" [2604.13515], it is treated as an explicit post-training hyperparameter for Qwen3-8B with “thinking” disabled, under a controlled ablation in which six training conditions differ solely in training recipe. The central empirical result is that keeping SFT and GRPO data disjoint consistently outperforms full overlap at zero additional compute cost, and that lower overlap is monotonically associated with higher compilation and semantic accuracy on Gaokao-Formal and PutnamBench [2604.13515].

## 1. Definition and formalization

Let $S_{\rm sft}$ denote the set of prompts used in the SFT stage, and $S_{\rm rl}$ the set of prompts used in the GRPO stage. AutoOverlap, denoted by the data-overlap fraction $\alpha$, is defined as

$$
\alpha = \frac{|S_{\rm sft} \cap S_{\rm rl}|}{|S_{\rm rl}|} \in [0,1].
$$

Equivalently, if $|S_{\rm rl}| = N_l$ and $N_{\rm overlap} = |S_{\rm sft} \cap S_{\rm rl}|$, then

$$
\alpha = \frac{N_{\rm overlap}}{N_l}.
$$

The study examines three discrete values of $\alpha$: $0\%$ $(\alpha=0)$, $30\%$ $(\alpha \approx 0.30)$, and $100\%$ $(\alpha=1.0)$ [2604.13515].

Under this formalization, AutoOverlap is not a property of model architecture or optimizer choice. It is a property of set-membership between post-training stages. This framing makes overlap a controllable experimental variable rather than an incidental feature of dataset construction. The paper identifies this as, to its knowledge, the first controlled investigation of SFT-GRPO data overlap as a post-training hyperparameter [2604.13515].

## 2. Controlled training conditions

All runs use the same base model, identical SFT and GRPO hyperparameters, and identical total data volumes. The six conditions are constructed so that overlap is the only manipulated variable in the mixed SFT+GRPO settings. For each desired $\alpha$, the GRPO prompt set is cross-deduplicated against the SFT corpus so that exactly $\alpha \cdot 16{,}000$ of its prompts appear in $S_{\rm sft}$ [2604.13515].

| Condition | Training recipe |
|---|---|
| Base model | No post-training |
| SFT-only | 20 000 $(\mathrm{NL}, \mathrm{Lean}\ 4)$ pairs |
| GRPO-only | GRPO on 16 000 prompts |
| SFT + GRPO-0% | SFT checkpoint, then GRPO on 16 000 non-overlapping prompts |
| SFT + GRPO-30% | $\sim 4{,}800$ prompts from SFT pool and 11 200 fresh ones |
| SFT + GRPO-100% | GRPO prompts fully drawn from the SFT pool |

This design isolates a specific question: whether GRPO contributes additional value when it is applied to prompts already seen during SFT. The reported behavior indicates that the answer depends strongly on $\alpha$. When overlap is complete, the GRPO stage becomes effectively redundant relative to SFT; when overlap is zero, GRPO provides the largest gains [2604.13515].

## 3. Evaluation protocol and metrics

Evaluation is conducted on Gaokao-Formal, comprising 495 problems, and PutnamBench, comprising 672 problems. For each benchmark problem, the system generates $n=8$ rollouts and measures two complementary pass@k metrics [2604.13515].

Compile pass@k, denoted $C@k$, is the fraction of problems for which at least one of $k \le 8$ rollouts typechecks under Mathlib v4.27.0. Semantic pass@k, denoted $S@k$, is the fraction for which at least one rollout both compiles and receives a judge score $\ge 0.7$. For a dataset $D$ of size $M$, with $\mathrm{Rollouts}(x)=\{y_1 \ldots y_8\}$ per $x \in D$,

$$
C@k = \frac{1}{M}\sum_{x \in D} \mathbf{1}[\exists\, i \le k : \mathrm{Compile}(y_i)=\mathrm{True}]
$$

and

$$
S@k = \frac{1}{M}\sum_{x \in D} \mathbf{1}[\exists\, i \le k : \mathrm{Compile}(y_i)=\mathrm{True} \wedge \mathrm{JudgeScore}(y_i)\ge 0.7].
$$

The compile–semantic gap is defined at $k=1$ by

$$
\Delta = C@1 - S@1.
$$

The semantic judge is a frozen LLM, Gemini Flash 3, that returns a continuous score $s_{\rm sem}(\hat y) \in [0,1]$ for each compilable output $\hat y$ relative to ground truth $y^*$. Non-compiling outputs receive $s_{\rm sem}=0$ [2604.13515].

The metric construction is consequential. It distinguishes syntactic validity from semantic correctness, making it possible to detect models that typecheck frequently but solve the wrong formalization problem.

## 4. Quantitative findings

At pass@1, the best mixed setting is always the non-overlapping one. Across both benchmarks, performance degrades monotonically as overlap $\alpha$ increases: $\alpha=0$ gives the highest $C@k$ and $S@k$, $\alpha \approx 0.30$ is slightly lower, and $\alpha=1.0$ collapses back to SFT-only levels [2604.13515].

| Condition | Gaokao-Formal $(C@1 / S@1)$ | PutnamBench $(C@1 / S@1)$ |
|---|---:|---:|
| Base | 19.9 / 10.2 | 11.3 / 3.3 |
| SFT | 61.8 / 41.0 | 28.5 / 14.3 |
| GRPO-only | 50.9 / 28.1 | 36.1 / 11.9 |
| SFT+GRPO-0% | 77.6 / 51.4 | 47.9 / 23.6 |
| SFT+GRPO-30% | 76.4 / 48.6 | 46.4 / 22.9 |
| SFT+GRPO-100% | 62.9 / 40.6 | 29.1 / 14.7 |

On Gaokao-Formal, SFT+GRPO-0% reaches $C@1=77.6\%$ and $S@1=51.4\%$, yielding a semantic gain over SFT of $+10.4$ percentage points. SFT+GRPO-30% is slightly worse at $76.4\%$ and $48.6\%$. SFT+GRPO-100% reaches $62.9\%$ and $40.6\%$, with both metrics essentially flat versus SFT [2604.13515].

On PutnamBench, SFT+GRPO-0% reaches $C@1=47.9\%$ and $S@1=23.6\%$, a semantic gain of $+9.3$ percentage points over SFT. SFT+GRPO-30% reaches $46.4\%$ and $22.9\%$. SFT+GRPO-100% reaches $29.1\%$ and $14.7\%$, again near SFT-only behavior [2604.13515].

The paper’s practical interpretation is direct: non-overlapping GRPO data is essential for RL to provide fresh learning signals, whereas full overlap renders the GRPO stage effectively redundant [2604.13515].

## 5. Compile–semantic gap and benchmark interpretation

A central contribution of the AutoOverlap study is methodological rather than purely comparative. Dual-metric evaluation reveals compile–semantic gaps exceeding 30 percentage points for the highest compiling models, and this disparity is invisible under compile-only benchmarking [2604.13515].

The reported $\Delta$ values at pass@1 already show substantial divergence within the six-condition ablation. On Gaokao-Formal, the gap rises from $9.7$ percentage points for the base model to $20.8$ for SFT, $22.8$ for GRPO-only, $26.2$ for SFT+GRPO-0%, $27.8$ for SFT+GRPO-30%, and $22.3$ for SFT+GRPO-100%. On PutnamBench, the corresponding values are $8.0$, $14.2$, $24.2$, $24.3$, $23.5$, and $14.4$ percentage points [2604.13515].

The same issue appears in external comparisons. For the highest-compiling external baselines, exemplified by Kimina-7B with $C@1 \approx 84\%$, the compile–semantic gap exceeds 35 percentage points [2604.13515]. Models with the highest $C@1$ often have the largest $\Delta$, meaning that many outputs are syntactically valid but semantically incorrect.

This result constrains how autoformalization progress should be interpreted. Compile-only metrics can overestimate end-to-end formalization quality because they collapse type correctness and proof-content correctness into a single observable. The paper therefore recommends dual-metric evaluation as standard practice [2604.13515].

## 6. Practical implications and scope of the term

The immediate recommendation is to treat the SFT↔GRPO overlap $\alpha$ as a tunable hyperparameter and, whenever corpus volume permits, to allocate disjoint data pools $(\alpha \approx 0)$ in order to maximize both compilation success and semantic fidelity [2604.13515]. Because the experiments hold total data volume and post-training hyperparameters fixed, the reported gains occur at zero extra compute cost. In this setting, changing only set overlap is sufficient to alter whether GRPO contributes meaningful additional learning.

A plausible implication is that AutoOverlap functions as a control knob for the novelty of the reinforcement-learning stage relative to the supervised stage. That implication is consistent with the paper’s conclusion that lower overlap provides fresh learning signals, whereas full overlap causes both compile and semantic accuracy to flatline relative to SFT [2604.13515].

The label “AutoOverlap” is not unique to this autoformalization setting. In unrelated work, "AutoOverlap: Enabling Fine-Grained Overlap of Computation and Communication with Chunk-Based Scheduling" denotes a compiler and runtime for intra-kernel overlap on multi-GPU workloads [2601.20595]. In 3D LiDAR localization, an overlap-based observation model has also been described under the AutoOverlap label in technical summaries [2105.11717]. The term therefore has field-specific meanings. Within Lean 4 autoformalization and SFT→GRPO post-training, however, AutoOverlap specifically denotes the overlap fraction $\alpha$ between the SFT and GRPO prompt sets [2604.13515].

Source: https://www.emergentmind.com/topics/autooverlap