dLLM-Corrector-7B: Local Proof Correction

Updated 4 July 2026

dLLM-Corrector-7B is a diffusion language model that locally repairs failing subgoal proofs in Lean by denoising a 512-token block.
It employs bidirectional attention within the block to condition corrections on both preceding and succeeding context for coherent in-filling.
Integrated in the Diffusion-Proof framework, it enhances theorem-proving accuracy on benchmarks like MiniF2F-Test by addressing four additional problems.

dLLM-Corrector-7B is a 7B-parameter large block discrete diffusion LLM for local proof correction in Lean within the Diffusion-Proof framework. It is not a whole-proof generator; rather, it is invoked after dLLM-Prover-7B has produced a proof whose top-level subgoal structure is correct but whose local subgoal proofs fail verification. The model repairs such failures by denoising a 512-token block that spans the faulty subgoal and surrounding context, thereby exploiting bidirectional information inside the block while retaining causal structure across blocks. In Diffusion-Proof, this specialization is the mechanism for local in-filling repair, and it contributes additional solved theorems on MiniF2F-Test beyond the Prover alone (Wang et al., 17 Jun 2026).

1. Position within the Diffusion-Proof architecture

Diffusion-Proof contains two 7B-parameter models derived from the same base, Fast-dLLM-V2-7B: dLLM-Prover-7B for whole-proof writing and dLLM-Corrector-7B for local proof correction. The pipeline is sequential. First, dLLM-Prover-7B generates a Lean proof. Lean then checks the proof. If the proof fails but its “skeleton” remains correct—specifically, if the top-level subgoal statements and their usages are correct—dLLM-Corrector-7B is used to repair the failing subgoal via local in-filling (Wang et al., 17 Jun 2026).

Aspect	dLLM-Prover-7B	dLLM-Corrector-7B
Primary role	Whole-proof generation	Local proof correction
Diffusion block size	32	512
Optimization target	Global proof planning	Local span rewriting

This division of labor is central to the framework. The Prover is optimized for long-range tactic usage and full proof construction, whereas the Corrector is optimized for rewriting a localized proof block conditioned on both prefix and suffix context. The paper reports that a unified use of dLLM-Corrector-7B for both generation and correction yields 40.98% on MiniF2F-Test, substantially below both Prover-only and Prover-plus-Corrector configurations, indicating that correction-oriented large-block training harms global proof generation when used as a single-model replacement (Wang et al., 17 Jun 2026).

Relative to autoregressive provers or correctors, the distinction is architectural rather than merely prompt-based. Autoregressive models generate left-to-right under conditional $P_\theta(x_i \mid x_{<i})$ and cannot natively perform true bidirectional in-filling of a missing middle span conditioned on both left and right context. dLLM-Corrector-7B instead denoises a masked proof block with bidirectional attention inside that block, so later uses of a local lemma can directly constrain the reconstruction of the lemma’s proof.

2. Diffusion substrate and block-level conditioning

The backbone is Fast-dLLM-V2-7B, a block-diffusion LLM built on top of Qwen-2.5-Instruct-7B. In the block dLLM framework, the sequence is segmented into contiguous non-overlapping blocks of size $B$ , attention is bidirectional within each block and causal across blocks, and generation proceeds by sequentially denoising one block at a time. For Diffusion-Proof, the Prover uses $B=32$ , while the Corrector increases the block size to $512$ to cover a full subgoal proof plus neighboring context (Wang et al., 17 Jun 2026).

The underlying diffusion process is discrete masking diffusion over tokens rather than continuous score-based noise. For a token sequence $\mathbf{x}=\{x_1,\dots,x_L\}$ , a noisy sequence $\mathbf{x}^{(t)}$ is formed by masking each token independently with probability $t$ , and the model learns to reconstruct original tokens from the masked sequence. The training loss used for dLLMs in the paper is

$\mathcal{L}(\theta) = - \mathbb{E}_{t, \mathbf{x}^{(0)}, \mathbf{x}^{(t)}} \Bigg[ \sum_{i=1}^L \mathbb{I}[x_i^{(t)} = \text{[MASK]}] \cdot \log P_\omega(x_i^{(0)} \mid \mathbf{x}^{(t)}) \Bigg].$

This is cross-entropy over masked positions. The model predicts over all positions in the block, but only masked positions contribute to the denoising objective.

For correction, the important consequence of the 512-token block is not only increased receptive field size but also a different conditioning regime. The corrected subgoal is generated while attending to tokens before the subgoal and tokens after it. In Lean proofs this matters because later uses of a lemma constrain the type and content that the local proof must establish. The paper describes this as “local proof correction using bi-directional information,” and the examples show that suffix context can influence earlier tactic choices inside the same block (Wang et al., 17 Jun 2026).

3. Training data and the large-block correction objective

The training corpus is derived from a cleaned, standardized Lean dataset of about 5.6M theorem–proof pairs assembled from prior Lean corpora including Lean-GitHub, STP, and DeepSeek-Prover. From this pool, the authors sample 300k examples to form the main supervised fine-tuning dataset for dLLM-Prover-7B. For dLLM-Corrector-7B, they further sample proofs from these 300k that have top-level subgoal decomposition introduced by have, yielding a 128k-record training set for correction (Wang et al., 17 Jun 2026).

Correction examples are built by isolating a top-level subgoal proof and turning it into an in-filling problem. The theorem statement, natural-language comments if present, surrounding proof context, and the chosen have line are retained, but the subgoal proof body is replaced inside a 256-token region with placeholder structure. The top-level subgoal proof is organized into token blocks of size 256 by filling the proof with a placeholder <|fim_middle|> token for training-time annotation. For corrector training, the diffusion block size is then extended to 512, and the model is trained to fill the subgoal proof while padding the unused tail of the block with placeholder tokens (Wang et al., 17 Jun 2026).

Only one subgoal per proof is converted into a correction task in each training example. This matters because it makes the objective explicitly local: the model learns to reconstruct a missing proof span while keeping the rest of the proof fixed. The supervision therefore differs from ordinary supervised fine-tuning on full proofs. The model is trained not to regenerate the theorem solution from scratch, but to align a missing local derivation with already-correct context.

The paper states that the same dLLM loss is reused, but with special attention to correction: loss is applied to both the subgoal proof block and the placeholder token. In the paper’s notation, this is conceptually

$\mathcal{L}_{\text{corrector}} = -\mathbb{E}\sum_{i \in \text{block}} \mathbb{I}[x_i^{(t)} = \text{[MASK]}] \log P_\omega(x_i^{(0)} \mid \mathbf{x}^{(t)}).$

Training hyperparameters for dLLM-Corrector-7B are explicit. The sequence length is 2,048 tokens, the learning rate is $5\times 10^{-6}$ , the scheduler uses 3% warmup, the global batch size is 64, the hardware is 4 × H100, and fine-tuning the Corrector takes about 48 H100 GPU-hours. The paper also notes that there is no additional reinforcement learning or curriculum learning for the Corrector; curriculum is mentioned only for Prover training.

4. Inference-time correction procedure

At inference, dLLM-Corrector-7B is invoked only after Lean verification and a skeleton check. If the initial proof fails but all top-level have subgoal statements and their references are correct, the failing subgoal is selected for local repair. The system then deletes the failing proof body, replaces it with a 256-token sequence of <|MASK|>, and feeds the entire theorem plus the masked subgoal region to dLLM-Corrector-7B (Wang et al., 17 Jun 2026).

The resulting 512-token correction block contains both the prefix before the subgoal and the suffix after it. Because attention inside the block is bidirectional, the model can generate the missing middle span in a way that is constrained by both earlier setup and later usage. Unused block tail positions are filled with placeholder tokens such as <|fim_middle|>. The paper does not present a fine-grained line-level localization algorithm; instead, it replaces the entire failing subgoal body at once.

Decoding follows the standard Fast-dLLM-V2 block-diffusion procedure. The paper gives two inference settings specific to the correction pipeline: temperature is set to 1.2, and “the confidence of one token to be denoised is increased to 0.95.” Each correction call is a single diffusion run rather than a pipeline-level multi-stage iterative rewrite. For pass@32 evaluation, the system allows up to 32 corrections per valid theorem (Wang et al., 17 Jun 2026).

This design makes dLLM-Corrector-7B fundamentally different from autoregressive rewrite correction. An autoregressive system that rewrites from the line before the first error still decodes causally and cannot condition a missing proof span on its right context in a native manner. By contrast, the diffusion corrector performs whole-subgoal regeneration inside a block where prefix and suffix are simultaneously visible.

5. Empirical behavior, benchmarks, and case studies

The headline benchmark numbers in Diffusion-Proof concern the combined framework rather than the Corrector alone. Relative to the autoregressive baseline Qwen-2.5-Lean-SFT-7B trained on the same dataset, Diffusion-Proof achieves 7.53% on ProofNet-Test versus 5.91% for the baseline, an absolute improvement of 1.61%, and 50.00% on MiniF2F-Test versus 43.85%, an absolute improvement of 6.14% (Wang et al., 17 Jun 2026).

The direct contribution of dLLM-Corrector-7B appears in the ablation study. On MiniF2F-Test, dLLM-Prover-7B alone reaches about 48.36%, while dLLM-Prover-7B plus dLLM-Corrector-7B reaches 50.00%. The Corrector therefore solves 4 additional problems, corresponding to a 1.64% absolute gain. The four named problems are imo_1962_p2, amc12b_2002_p7, mathd_algebra_215, and mathd_numbertheory_521. On ProofNet-Test, the Corrector adds no extra solved theorems; the paper attributes this to the dataset’s emphasis on advanced knowledge rather than complex subgoal division or long-range local repair (Wang et al., 17 Jun 2026).

Ablations on correction strategy further isolate the benefit of the large-block approach. AR line-level rewrite correction solves +1 new problem, dLLM in-place correction solves +2 problems, and dLLM top-level correction—the method used by dLLM-Corrector-7B—solves +4 problems. This supports the claim that regenerating an entire top-level subgoal proof in a coherent diffusion block is more effective than either autoregressive rewriting or narrower in-place dLLM correction.

The qualitative examples are consistent with this interpretation. In mathd_algebra_215, the Corrector replaces a failed have h₁ proof with a coherent derivation establishing S = {8, -14} and introduces internal decomposition steps such as h₂ and h₃, using eq_or_eq_neg_of_sq_eq_sq and linarith. In mathd_numbertheory_521, it replaces an unsuccessful tactic sequence with a corrected derivation that uses omega to derive m = n + 2, then resolves the resulting arithmetic constraints to prove m = 18. In imo_1962_p2, the paper attributes the final successful proof specifically to the Corrector and contrasts it with a failure of DeepSeek-Prover-V2-7B with Long CoT. The stated interpretation is that the Corrector’s large diffusion block supports more robust and coherent tactic usage in the relevant local region (Wang et al., 17 Jun 2026).

The conceptual rationale given for dLLM-Corrector-7B is that diffusion-based local repair mitigates several autoregressive pathologies. The paper argues that autoregressive models suffer from long-range coherence difficulties and error compounding over long sequences, whereas dLLMs benefit from long-range coherence and bidirectional awareness. In the correction setting, the relevant unit is the local proof block: rather than extending a flawed prefix, the model can reconsider a whole subgoal proof holistically under bidirectional conditioning (Wang et al., 17 Jun 2026).

An indirect piece of evidence comes from the validation-loss study reported in Diffusion-Proof. When evaluated under a purely causal mask with block size set to 1, the fine-tuned dLLM and the fine-tuned autoregressive model have very similar cross-entropy loss, with Pearson correlation about 0.9846, yet the dLLM-based framework achieves better theorem-proving accuracy. The authors interpret this as evidence that the gain arises from the generation paradigm—iterative block diffusion and bidirectional refinement—rather than from a simple language-model likelihood advantage.

Related diffusion-LM research sharpens the context. “Corrective Diffusion LLMs” argues that standard masked diffusion LLM training does not reliably induce correction-oriented behavior because visible incorrect tokens receive no explicit supervision, and proposes an absorbing–uniform mixture post-training principle to make token-level confidence correlate with correctness (Zhang et al., 17 Dec 2025). This suggests that dLLM-Corrector-7B’s success should not be read as a generic consequence of non-causal denoising alone; rather, its task design, block structure, and verification-mediated deployment are likely essential.

Two adjacent lines of work indicate plausible future extensions. “DLLM-JEPA” proposes a latent-space predictive objective for masked diffusion LLMs and explicitly describes its relevance to a dLLM-based corrector as “latent correction training,” where differently corrupted views of the same content are aligned while denoising tokens (Nam, 24 May 2026). “LoPA” presents a training-free, plug-and-play lookahead parallel decoding method for 7B-scale dLLMs and reports substantial gains in tokens per forward pass and multi-device throughput (Xu et al., 18 Dec 2025). Neither method is part of Diffusion-Proof, but together they suggest a plausible research trajectory in which future variants of dLLM-Corrector-7B combine stronger correction-oriented representation learning with more aggressive diffusion-side inference acceleration.

The limitations stated for the Corrector are correspondingly specific. It is local-only: if the proof skeleton itself is wrong, the model is not invoked. It adds no benefit on ProofNet-Test, where failures are more knowledge-intensive and less amenable to local proof repair. The framework is restricted to Lean4, does not explore Long CoT reasoning with diffusion models, and does not provide a theoretical analysis of dLLMs. The future directions implied in the paper include scaling the Prover and Corrector, integrating reinforcement learning with verifiable rewards and search, improving span-selection heuristics, extending the approach to other proof assistants, and studying Long CoT with diffusion LLMs (Wang et al., 17 Jun 2026).