dLLM-Prover-7B: Lean4 Diffusion Prover

• dLLM-Prover-7B is a 7B-parameter diffusion LLM designed for Lean4 formal theorem proving, using blockwise diffusion for holistic proof generation.
• It partitions proofs into fixed 32-token blocks, enabling bidirectional within-block revision while maintaining causal inter-block progression for global coherence.
• Integrated with dLLM-Corrector-7B, it boosts long-proof accuracy by 4.51–6.14 percentage points over auto-regressive baselines on key benchmarks.

dLLM-Prover-7B is a 7B-parameter diffusion LLM for Lean4 formal theorem proving, introduced as the primary prover in the Diffusion-Proof framework. It replaces auto-regressive next-token generation with blockwise diffusion denoising, with the explicit goal of improving whole-proof generation through stronger long-range coherent tactic usage and iterative within-block revision. The model is fine-tuned from Fast-dLLM-V2-7B and is paired, within the broader Diffusion-Proof system, with a second model, dLLM-Corrector-7B, for local proof repair (Wang et al., 17 Jun 2026).

1. Position within diffusion-based formal theorem proving

Diffusion-Proof presents dLLM-Prover-7B as the core generative engine for formal theorem proving in Lean4. The framework is motivated by the claim that essentially all prior Lean4 provers are auto-regressive transformers trained by next-token prediction, and that such models face persistent difficulties on long formal proofs because they condition only on left context, accumulate early errors over long sequences, and do not naturally support in-filling or middle-of-proof correction (Wang et al., 17 Jun 2026).

Within this framework, dLLM-Prover-7B is the whole-proof writer, while dLLM-Corrector-7B is a second-stage diffusion correction model used when verification fails but the proof skeleton is judged correct. The division of labor is explicit: the Prover plans and writes complete Lean proofs, and the Corrector repairs local segments using bidirectional information. This places dLLM-Prover-7B at the center of Diffusion-Proof’s claim that diffusion LLMs are a viable alternative generation paradigm for formal proof synthesis, rather than merely an auxiliary refinement mechanism (Wang et al., 17 Jun 2026).

The model is notable in the literature because the paper describes Diffusion-Proof as, to the best of its knowledge, the first framework to train and apply diffusion LLMs to formal theorem proving. That claim situates dLLM-Prover-7B not simply as another 7B Lean prover, but as an attempt to transplant discrete diffusion generation into a domain where long-range coherence and verifier sensitivity are unusually stringent (Wang et al., 17 Jun 2026).

2. Diffusion formulation and blockwise architecture

dLLM-Prover-7B is obtained by fine-tuning Fast-dLLM-V2-7B, itself a 7B diffusion LLM based originally on Qwen-2.5-Instruct-7B. No new architectural layers are introduced specifically for Lean4; the adaptation is primarily through fine-tuning and task formatting. The defining architectural feature is block diffusion: the sequence is partitioned into contiguous blocks, with bidirectional attention inside each block and causal attention between blocks (Wang et al., 17 Jun 2026).

For the prover, the block size is fixed at $B=32$, and the context length is 2048 tokens. The sequence includes Lean imports, theorem statement, optional natural-language description or comments, and the proof. Bi-directional attention within a 32-token block allows the model to coordinate several consecutive tactic lines jointly, while causal structure across blocks preserves an overall left-to-right proof-writing order. The paper describes this as a hybrid of auto-regressive global progression and masked-language-model-style local bidirectionality (Wang et al., 17 Jun 2026).

The diffusion process is the standard discrete masking formulation for text. Given a clean sequence $\bm{x}^{(0)}$, each token is independently replaced by $[\text{MASK}]$ with probability $t$, yielding $\bm{x}^{(t)}$. The forward process is:

$$q_t(\bm{x}^{(t)} \mid \bm{x}^{(0)}) = \prod_{i=1}^L \left[(1-t)\,\mathbb{I}(x_i^{(t)} = x_i^{(0)}) + t\,\mathbb{I}(x_i^{(t)} = \text{[MASK]})\right].$$

The learned reverse process predicts original tokens from the noisy sequence, and training minimizes masked-position cross-entropy:

$$\mathcal{L}(\omega) = - \mathbb{E}_{t, \bm{x}^{(0)}, \bm{x}^{(t)}} \left[ \sum_{i=1}^L \mathbb{I}\left[x_i^{(t)} = \text{[MASK]}\right]\cdot \log \mathbb{P}_\omega(x_i^{(0)} \mid \bm{x}^{(t)}) \right].$$

Only masked positions contribute to the loss, and inference proceeds by iterative denoising within each block (Wang et al., 17 Jun 2026).

The paper attributes two main theorem-proving advantages to this setup. First, tokens inside a block attend to both left and right context, which promotes global consistency of tactics and subgoals across that block. Second, generation is iterative, so low-confidence tokens can be revised inside the block rather than being irreversibly committed at first emission. This suggests that dLLM-Prover-7B is best understood as a semi-causal proof writer with local bidirectional revision capacity, rather than as a fully non-causal denoiser.

Attribute	dLLM-Prover-7B
Backbone	Fast-dLLM-V2-7B
Parameter scale	7B
Context length	2048
Block size	32
Within-block attention	Bi-directional
Between-block attention	Causal
Training objective	Pure diffusion masking loss

3. Training data, representations, and optimization regime

The training corpus aggregates Lean4 theorem-proving data from sources including Lean Workbook, Lean-GitHub, STP, and TheoremLlama, totaling 5,595,798 proof records. Because the raw material is heterogeneous, the authors apply rule-based parsing to separate natural-language statements and comments, Lean statements, and Lean proofs. From this larger pool, they randomly sample a 300k-record supervised fine-tuning subset for dLLM-Prover-7B, with approximately a 1:2 ratio between pure Lean proofs and Lean proofs with natural-language annotations or comments (Wang et al., 17 Jun 2026).

The model is trained in a code-completion format following DeepSeek-Prover’s convention rather than in a chat format. Inputs include import lines, theorem signature, hypotheses, and sometimes comment-based explanations; outputs are Lean by proof blocks. The full concatenated sequence is then subjected to the diffusion denoising objective. There is no auxiliary auto-regressive cross-entropy term and no hybrid objective; the training loss is the pure masked denoising loss over the combined theorem-and-proof sequence (Wang et al., 17 Jun 2026).

The optimization regime includes curriculum data sorting: proofs are ordered by length, used as a proxy for complexity, so that training proceeds from shorter and simpler proofs to longer and more complex ones. The hyperparameters reported for the prover are a learning rate of $1\times10^{-5}$, 3% warmup steps, global batch size 64 with per-GPU batch size 2 and gradient accumulation 8, training on 4×H100 GPUs, and approximately 96 H100 GPU hours. The model keeps the base block size $B=32$ for stability (Wang et al., 17 Jun 2026).

The controlled baseline is Qwen-2.5-Lean-SFT-7B, an auto-regressive model trained on exactly the same 300k dataset in the same text-to-text format, differing only in the generation paradigm and training objective. This matched-data comparison is central to the paper’s interpretation of dLLM-Prover-7B’s gains, because it isolates diffusion versus auto-regressive generation rather than conflating data scale or model size (Wang et al., 17 Jun 2026).

4. Inference workflow and interaction with correction

At inference time, dLLM-Prover-7B receives a prompt matching the training format: Lean imports, theorem signature and hypotheses, optional natural-language description or comments, and a by where the proof should be generated. The proof region is initialized as masked, and Fast-dLLM-V2’s blockwise denoising engine iteratively fills tokens block by block, each block containing 32 tokens. Generation stops when a valid Lean proof terminator is reached or the maximum context length is exhausted (Wang et al., 17 Jun 2026).

The paper does not specify the exact number of denoising iterations for the prover, but it does report a denoising confidence threshold in the underlying process: tokens with confidence greater than 0.95 are considered stable. It also notes that temperature can be used to increase diversity and creativity in tactic selection. Within the block, the model can revise tokens across diffusion steps, so a short local proof plan can be stabilized before the block is finalized (Wang et al., 17 Jun 2026).

Diffusion-Proof evaluates dLLM-Prover-7B as a whole-proof writer rather than as a tree-search or best-first node expander. Pass@32 corresponds to 32 independent full proof attempts. This is important for interpreting the model’s contribution: its improvement is presented as an effect of blockwise diffusion generation itself, not of an external search procedure (Wang et al., 17 Jun 2026).

In the full pipeline, dLLM-Prover-7B is followed by dLLM-Corrector-7B when needed. The Prover produces up to 32 candidate proofs; Lean verifies them; if none succeeds but the proof skeleton is considered correct, the system removes the offending subgoal proof and replaces it with 256 <|MASK|> tokens. The Corrector then uses a larger 512-token diffusion block to infill the damaged region using both prefix and suffix context. dLLM-Prover-7B therefore serves as the planning-and-generation stage in a two-stage architecture where local repair is delegated to a separate model (Wang et al., 17 Jun 2026).

Later work on diffusion LLM inference, such as “$R^2$-dLLM” (Du et al., 21 Apr 2026) and “AdaBlock-dLLM” (Lu et al., 30 Sep 2025), studies training-free and scheduling-based decoding accelerations for diffusion LLMs. These papers do not evaluate dLLM-Prover-7B directly. A plausible implication is that dLLM-Prover-7B could inherit some of the same decoding-efficiency ideas, because it shares the masked diffusion and blockwise generation paradigm.

5. Empirical performance and diagnostic analyses

Evaluation is reported on MiniF2F-Test and ProofNet-Test using pass@32. MiniF2F-Test contains 244 Lean4 formal problems from high-school competitions and related sources, while ProofNet-Test contains 186 Lean4 formalized theorems from undergraduate mathematics. On MiniF2F-Test, the matched auto-regressive baseline Qwen-2.5-Lean-SFT-7B reaches 43.85% pass@32, dLLM-Prover-7B alone reaches 48.36%, and the full Diffusion-Proof system reaches 50.00%. On ProofNet-Test, the baseline reaches 5.91%, while the full system reaches 7.53%, with the gain attributed mostly to the prover rather than the corrector (Wang et al., 17 Jun 2026).

The reported benchmark deltas are therefore +4.51 percentage points for dLLM-Prover-7B alone over the auto-regressive baseline on MiniF2F-Test, and +6.14 points for the full system. The paper also reports category-level improvements on MiniF2F, including IMO problems increasing from 5.00% to 15.00%, algebra from 55.68% to 67.05%, and number theory from 58.82% to 63.24%. In long-proof analysis based on DeepSeek-Prover-V2 proof lengths, Diffusion-Proof solves 26 of the longest 108 problems versus 20 for the baseline, and 5 of the longest 54 versus 4 for the baseline (Wang et al., 17 Jun 2026).

One qualitative result highlighted in the paper is that Diffusion-Proof solves the IMO problem imo_1962_p2, which DeepSeek-Prover-V2-7B with Long CoT reasoning does not solve under pass@32. The authors interpret this as evidence that diffusion generation can yield a distinct error profile and a different strategic coherence than auto-regressive reasoning on some difficult formal problems (Wang et al., 17 Jun 2026).

The paper also performs a causal-attention cross-entropy analysis on 190 correct Lean proofs generated by DeepSeek-Prover-V2. When diffusion and auto-regressive models are both evaluated under pure causal masking, the per-example loss distributions are extremely similar. The Pearson correlation of per-example loss is 0.9846 between the fine-tuned AR and fine-tuned dLLM models, and 0.9838 between the base AR and base dLLM models, both with $\bm{x}^{(0)}$0. The authors therefore conclude that the observed theorem-proving improvement is not due to markedly better causal token prediction, but to the diffusion generation procedure itself: iterative blockwise refinement, bidirectional awareness within blocks, and a different failure mode under long proofs (Wang et al., 17 Jun 2026).

Benchmark	AR baseline	dLLM-Prover-7B	Full Diffusion-Proof
MiniF2F-Test (pass@32)	43.85%	48.36%	50.00%
ProofNet-Test (pass@32)	5.91%	—	7.53%

6. Relation to auto-regressive provers, neighboring systems, and limitations

The paper frames dLLM-Prover-7B against matched-size auto-regressive provers rather than against purely larger models. Architecturally, the contrast is between a standard left-to-right transformer using causal attention across the full sequence and a blockwise diffusion model with causal attention between blocks and bidirectional attention within blocks. Empirically, the authors argue that the auto-regressive baseline is more brittle to early mistakes and less able to revise mid-proof errors, despite having similar token-level modeling capacity under causal evaluation (Wang et al., 17 Jun 2026).

This positioning differs from agentic or search-heavy theorem-proving systems such as Delta Prover (Zhou et al., 21 Jul 2025), which uses a general-purpose LLM plus reflective decomposition, iterative proof repair, and a Lean-embedded DSL rather than a specialized diffusion prover. It also differs from verifier-integrated reinforcement-learning systems such as Leanabell-Prover-V2 (Ji et al., 11 Jul 2025), which post-trains existing 7B Lean provers to use multi-turn verifier feedback in long-CoT trajectories. dLLM-Prover-7B instead concentrates the innovation in the generation substrate itself: diffusion-based whole-proof writing with local bidirectional coherence.

The paper states several limitations. Training is limited to a 300k supervised fine-tuning set and a single 7B model, rather than the larger data and reinforcement learning regimes used by state-of-the-art auto-regressive provers. The current diffusion base does not have strong long chain-of-thought capability, only Lean4 is considered, some failures still come from overconfident use of generic tactics such as linarith and nlinarith, and the work does not yet provide a formal theory for why diffusion architectures are superior on such tasks (Wang et al., 17 Jun 2026).

Implementation-wise, the paper reports that diffusion inference is about 2.54× faster than the auto-regressive baseline decoding by leveraging Fast-dLLM’s KV-cache and parallel decoding, while combined Prover+Corrector evaluation including Lean verification takes about 16 hours on MiniF2F and ProofNet, versus about 24 hours for the auto-regressive baseline without vLLM optimization. The authors state that they plan to release code, models, and datasets, with model names dLLM-Prover-7B and dLLM-Corrector-7B derived from Fast-dLLM-V2-7B (Wang et al., 17 Jun 2026).

Taken together, dLLM-Prover-7B occupies a specific place in the theorem-proving landscape: it is a verifier-facing Lean4 prover whose distinguishing contribution is not tree search, decomposition DSLs, or large-scale RL, but blockwise diffusion denoising as the proof-generation mechanism. The reported evidence suggests that this mechanism yields measurable gains over matched auto-regressive training on formal mathematics benchmarks, particularly on longer proofs, while leaving open broader questions about scaling, theory, and integration with search or verifier-driven RL.