SCAN: Self-Denoising Monte Carlo Annotation

• The paper introduces SCAN as a robust self-denoising MC annotation framework that mitigates noise in low self-confidence samples to enhance PRM training.
• SCAN employs selective sampling, soft target labeling, and confidence reweighting to improve annotation efficiency and boost metrics like best-of-8 accuracy and ProcessBench F1 scores.
• By reducing inference cost by over 90%, SCAN demonstrates that compact synthetic datasets can reach or surpass the performance of larger human-annotated datasets in process reward learning.

Searching arXiv for the SCAN paper and closely related PRM context papers. Self-Denoising Monte Carlo Annotation (SCAN) is a data synthesis and noise-tolerant learning framework for training process reward models (PRMs) under weak supervision from Monte Carlo (MC) annotation rather than large-scale human labeling. It is introduced in "SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning" (Ding et al., 20 Sep 2025). The framework addresses a central difficulty in PRM construction: conventional MC annotation is scalable but noisy, because annotation models can both underestimate and overestimate step correctness. SCAN studies that noise, identifies its concentration in low self-confidence samples, and then combines selective sampling, self-denoising labels, and robust training to improve both annotation efficiency and PRM quality (Ding et al., 20 Sep 2025).

1. Position within process reward learning

Process reward models provide fine-grained, step-level feedback for LLM reasoning, and the paper situates them as especially relevant to complex tasks such as mathematical reasoning (Ding et al., 20 Sep 2025). Their utility depends on access to large-scale, high-quality annotations of intermediate reasoning steps, but the paper treats such supervision as costly and difficult to scale when collected from humans.

Within this setting, conventional Monte Carlo annotation is presented as a synthetic alternative. The standard workflow is to generate multiple continuations, or rollouts, from a partially constructed solution and to infer the correctness of a step from whether at least one continuation reaches the correct final answer (Ding et al., 20 Sep 2025). SCAN is formulated as a response to the failure mode of that workflow: although MC estimation can synthesize supervisory signals, its labels are noisy enough to induce overfitting and poor generalization in PRM training, especially as dataset size grows (Ding et al., 20 Sep 2025).

The paper therefore frames SCAN as a data-centric intervention at the annotation stage and a robust-learning intervention at the training stage. This dual emphasis distinguishes it from approaches that treat synthetic labels as fixed targets. A plausible implication is that the framework is intended not merely to reduce annotation cost, but to make MC-derived supervision viable at scale for PRM training.

2. Noise model in Monte Carlo annotation

The empirical starting point of SCAN is a preliminary study of the noise distribution in MC-generated synthetic data (Ding et al., 20 Sep 2025). The study identifies two systematic error modes in annotation models.

Underestimation occurs when the annotation model fails to reach the correct answer even though the existing steps are correct, causing correct steps to be labeled as errors (Ding et al., 20 Sep 2025). Overestimation occurs when the model self-corrects later in the trajectory, so incorrect intermediate steps are not penalized because subsequent continuations still reach the correct final answer (Ding et al., 20 Sep 2025). In the paper’s interpretation, both failure modes are consequences of the annotation model’s own capabilities rather than purely random label corruption.

To analyze this behavior, the paper introduces a self-confidence score:

$$SC_{\theta}(q) = \mathbb{E}_{r \sim P_\theta(\cdot \mid q)}[\mathcal{J}(r, a^*)].$$

Here, $\mathcal{J}(r, a^*)$ indicates whether response $r$ matches the gold answer $a^*$ (Ding et al., 20 Sep 2025). This quantity measures how likely the annotation model is to solve the question correctly when prompted directly.

The paper reports that noise concentrates in low self-confidence samples: high-confidence samples tend to exhibit less label noise, whereas low-confidence samples are more error-prone and should be treated carefully or excluded (Ding et al., 20 Sep 2025). This observation is foundational for the subsequent SCAN pipeline. Rather than assuming all MC annotations are equally reliable, SCAN conditions annotation and training decisions on estimated model competence at the question level.

A related correctness estimator is also given for individual steps:

$$c_t = \mathbb{E}_{r \sim P_\theta(\cdot \mid q, x_{\leq t})}[\mathcal{J}(r, a^*)].$$

This estimator formalizes step correctness under continuation sampling and serves as the basis for denoised labels (Ding et al., 20 Sep 2025).

3. Self-denoising data synthesis pipeline

SCAN’s synthesis strategy begins with a lightweight LLM, exemplified in the paper by Qwen2.5-Math-1.5B-Instruct, to generate responses for each question (Ding et al., 20 Sep 2025). It then computes self-confidence over multiple responses per question and uses that value to determine which samples should enter the annotation pipeline.

For high-confidence samples, the framework distinguishes between already-clean positives and samples that still require MC step-wise annotation. If a response is fully correct and no errors are detected, it is used directly as a positive example (Ding et al., 20 Sep 2025). If a response is incorrect, MC step-wise annotation is performed only when the sample is high-confidence, thereby reducing unnecessary computation on samples that are likely to be noisy or uninformative (Ding et al., 20 Sep 2025). Low-confidence samples are discarded.

This selective sampling procedure is described as discarding noisy low-confidence data and avoiding annotation of already-clean positives, which cuts inference cost by over 94% compared to vanilla MC (Ding et al., 20 Sep 2025). In the abstract, the paper states the same efficiency result in a different form: lightweight models with 1.5B parameters can produce high-quality annotations through self-denoising, enabling PRMs to achieve superior performance with only 6% of the inference cost required by vanilla MC estimation (Ding et al., 20 Sep 2025).

The design reflects a specific diagnosis of annotation failure. Because noise is concentrated where the annotation model is uncertain, the framework reduces compute on precisely those regions where additional rollouts are least trustworthy. This suggests that SCAN treats annotation budget allocation and label denoising as coupled problems rather than independent optimization targets.

4. Denoised labels and robust learning objective

SCAN’s learning procedure is explicitly noise-tolerant. For each annotated sample $(q, \mathbf{x}, \mathbf{c}, SC_\pi(q))$, the framework constructs soft labels using a tolerance distance $d$ (Ding et al., 20 Sep 2025). For steps within $d$ steps before the predicted error, the denoised target is

$\hat{y}_{t} = \min(c_t / SC_{\pi}(q), 1),$

while for other steps, $\hat{y}_t = 1$ if $\mathcal{J}(r, a^*)$0 (Ding et al., 20 Sep 2025).

The paper describes this as label smoothing via tolerance distance. The purpose is to acknowledge that error localization in noisy MC data is imprecise: steps near the predicted error boundary are treated as uncertain rather than assigned hard correct or incorrect labels (Ding et al., 20 Sep 2025). The reported ablation identifies $\mathcal{J}(r, a^*)$1 as the best tradeoff; smaller values overfit noise, whereas larger values make the training signal too soft and performance lags (Ding et al., 20 Sep 2025).

A second mechanism is confidence-wise reweighting. SCAN rescales correctness by self-confidence using

$\mathcal{J}(r, a^*)$2

The paper states that this de-biases annotations by normalizing the varying capabilities of different completer models, and that it is especially important when merging data from models of varying strengths (Ding et al., 20 Sep 2025). In effect, the framework discounts labels generated under weak annotation competence and amplifies those produced under stronger estimated competence, without requiring human calibration.

The final PRM objective is a modified binary cross-entropy:

$\mathcal{J}(r, a^*)$3

The paper characterizes the net effect of this design as robust learning from weak supervision: selective inclusion of higher-quality data, soft labels near uncertain error boundaries, and self-confidence reweighting together enable PRMs to generalize despite systemic, model-induced label noise (Ding et al., 20 Sep 2025).

5. Empirical performance, scaling, and efficiency

The paper reports results for two SCAN datasets. SCAN-Base uses 101K samples annotated with a 1.5B model, and SCAN-Pro scales to 197K samples by incorporating additional 7B and 3B model data (Ding et al., 20 Sep 2025). The larger dataset is reported to require 374.5 GPU hours on 8 A800 GPUs, corresponding to over 90% cost reduction relative to standard MC annotation (Ding et al., 20 Sep 2025).

The main quantitative outcomes are organized around best-of-8 evaluation, ProcessBench, noise reduction, and inference speed.

Setting	Reported result	Context
Qwen2.5-Math-7B-Scan-Base	69.1 average accuracy	Trained on 101K SCAN data
PRMs trained on 264K PRM800K data	69.3 average accuracy	Human-annotated reference
Qwen2.5-Math-7B-Scan-Pro	70.1 average accuracy	Trained on 197K SCAN data
Qwen2.5-Math-7B-Scan-Base	56.8 F1	ProcessBench average across tasks
MC-only baseline	19.9 F1	ProcessBench reference
Qwen2.5-Math-7B-Scan-Pro	59.1 F1	ProcessBench average across tasks
Human-annotated PRM800K	56.5 F1	ProcessBench reference

In best-of-8 evaluation, Qwen2.5-Math-7B-Scan-Base achieves 69.1 average accuracy, outperforming all MC-only 7B and 8B PRM baselines and remaining comparable to PRMs trained on 264K human-annotated PRM800K data at 69.3 (Ding et al., 20 Sep 2025). With 197K samples, Qwen2.5-Math-7B-Scan-Pro rises to 70.1, matching or surpassing models trained with human and strong knowledge-distillation supervision (Ding et al., 20 Sep 2025).

On ProcessBench, which the paper uses for fine-grained error localization, Qwen2.5-Math-7B-Scan-Base achieves 56.8 F1, representing a 36.9 or 39.2 point gain over the MC-only baseline at 19.9 F1, while Qwen2.5-Math-7B-Scan-Pro reaches 59.1 F1 and surpasses the human-annotated PRM800K result of 56.5 F1 (Ding et al., 20 Sep 2025). The abstract emphasizes the 39.2 F1 improvement from 19.9 to 59.1 on ProcessBench under robust learning from weak supervision (Ding et al., 20 Sep 2025).

The paper also reports direct noise reduction. For Qwen2.5-Math-7B, the noise ratio in synthetic data decreases from 51.8% under vanilla MC to 29.4% with SCAN denoising; for Llama-3.1-8B, it decreases from 56.2% to 19.1% on ProcessBench-like data (Ding et al., 20 Sep 2025). Inference speed for SCAN-trained PRMs is given as 44 samples per second for Qwen2.5-7B, compared with 1.5 samples per second for generative critic models on Qwen2.5-Math-7B-Instruct (Ding et al., 20 Sep 2025).

The ablations reinforce the design claims. Removing self-confidence reweighting or tolerance labeling degrades performance, and scaling up dataset size with SCAN continues to improve results without clear overfitting, unlike conventional MC annotations (Ding et al., 20 Sep 2025). Human evaluation is reported to show that SCAN-trained PRMs can identify subtle errors and correct answers even in challenging best-of-$\mathcal{J}(r, a^*)$4 settings (Ding et al., 20 Sep 2025).

6. Relation to baselines, scope, and limitations

SCAN is evaluated against several classes of baselines: MC-only PRMs, PRMs trained on human-annotated PRM800K, and models trained with strong critic or knowledge-distillation supervision (Ding et al., 20 Sep 2025). The reported comparisons are framed around synthetic-data efficiency rather than architectural novelty alone. The paper’s central empirical claim is that a compact synthetic dataset, when denoised and used in a robust learning framework, can outperform much larger or more expensive supervision sources (Ding et al., 20 Sep 2025).

The framework is also presented as plug-and-play: denoised MC data can be integrated as a component or substitute in any framework using MC estimation, and it can enhance downstream knowledge-distillation-based pipelines (Ding et al., 20 Sep 2025). This suggests that SCAN is intended as a general-purpose annotation and training layer for PRM construction, not solely as a single-model recipe.

The paper also states several limitations. Purely MC-based methods cannot catch "false positive" errors, meaning responses marked correct may still contain hidden step-wise errors (Ding et al., 20 Sep 2025). SCAN-trained PRMs retain limitations in deep semantic error detection and remain vulnerable to reward hacking in certain settings (Ding et al., 20 Sep 2025). The paper therefore indicates that further gains may require hybrid approaches, including generative process reward models with richer supervision (Ding et al., 20 Sep 2025).

These limitations are important for interpretation. SCAN is not presented as eliminating the structural weaknesses of MC supervision; rather, it reduces major error sources sufficiently to make weak self-supervision practical. A plausible implication is that the framework is strongest where step correctness is correlated with final-answer recoverability, and less complete where reasoning defects are semantically subtle yet outcome-preserving.

7. Significance for scalable PRM training

The significance attributed to SCAN is that high-quality reward models can be trained from weak, noisy, self-supervision generated by lightweight models rather than exclusively from large-scale human annotation or large critic models (Ding et al., 20 Sep 2025). The paper reports that even a 1.5B annotation model can support superior PRM performance through self-denoising, and that scaling the synthetic dataset further improves performance (Ding et al., 20 Sep 2025).

The broader implication advanced in the paper is that denoising MC at the source makes MC annotation scalable and cost-efficient without requiring strong critic LLMs, potentially widening access to PRM training across domains and institutions (Ding et al., 20 Sep 2025). Within the paper’s results, this claim is supported by the combination of reduced noise ratios, strong ProcessBench gains, favorable best-of-8 accuracy, and substantial reductions in annotation cost (Ding et al., 20 Sep 2025).

In methodological terms, SCAN combines three elements that are often treated separately: diagnosis of annotation-model-induced noise, selective data synthesis conditioned on self-confidence, and robust PRM optimization with soft reweighted targets. The paper argues that this combination, rather than any single component alone, is what enables scalable and robust process reward learning from synthetic data (Ding et al., 20 Sep 2025).