---
title: 'SCAN: Self-Denoising Monte Carlo Annotation'
url: https://www.emergentmind.com/topics/self-denoising-monte-carlo-annotation-scan
type: topic
---

# SCAN: Self-Denoising Monte Carlo Annotation

Searching arXiv for the SCAN paper and closely related PRM context papers.
Self-Denoising Monte Carlo Annotation (SCAN) is a data synthesis and noise-tolerant learning framework for training process reward models (PRMs) under weak supervision from Monte Carlo (MC) annotation rather than large-scale human labeling. It is introduced in "SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning" [2509.16548]. The framework addresses a central difficulty in PRM construction: conventional MC annotation is scalable but noisy, because annotation models can both underestimate and overestimate step correctness. SCAN studies that noise, identifies its concentration in low self-confidence samples, and then combines selective sampling, self-denoising labels, and robust training to improve both annotation efficiency and PRM quality [2509.16548].

## 1. Position within process reward learning

Process reward models provide fine-grained, step-level feedback for large language model reasoning, and the paper situates them as especially relevant to complex tasks such as mathematical reasoning [2509.16548]. Their utility depends on access to large-scale, high-quality annotations of intermediate reasoning steps, but the paper treats such supervision as costly and difficult to scale when collected from humans.

Within this setting, conventional Monte Carlo annotation is presented as a synthetic alternative. The standard workflow is to generate multiple continuations, or rollouts, from a partially constructed solution and to infer the correctness of a step from whether at least one continuation reaches the correct final answer [2509.16548]. SCAN is formulated as a response to the failure mode of that workflow: although MC estimation can synthesize supervisory signals, its labels are noisy enough to induce overfitting and poor generalization in PRM training, especially as dataset size grows [2509.16548].

The paper therefore frames SCAN as a data-centric intervention at the annotation stage and a robust-learning intervention at the training stage. This dual emphasis distinguishes it from approaches that treat synthetic labels as fixed targets. A plausible implication is that the framework is intended not merely to reduce annotation cost, but to make MC-derived supervision viable at scale for PRM training.

## 2. Noise model in Monte Carlo annotation

The empirical starting point of SCAN is a preliminary study of the noise distribution in MC-generated synthetic data [2509.16548]. The study identifies two systematic error modes in annotation models.

**Underestimation** occurs when the annotation model fails to reach the correct answer even though the existing steps are correct, causing correct steps to be labeled as errors [2509.16548]. **Overestimation** occurs when the model self-corrects later in the trajectory, so incorrect intermediate steps are not penalized because subsequent continuations still reach the correct final answer [2509.16548]. In the paper’s interpretation, both failure modes are consequences of the annotation model’s own capabilities rather than purely random label corruption.

To analyze this behavior, the paper introduces a self-confidence score:
$$
SC_{\theta}(q) = \mathbb{E}_{r \sim P_\theta(\cdot \mid q)}[\mathcal{J}(r, a^*)].
$$
Here, $\mathcal{J}(r, a^*)$ indicates whether response $r$ matches the gold answer $a^*$ [2509.16548]. This quantity measures how likely the annotation model is to solve the question correctly when prompted directly.

The paper reports that noise concentrates in low self-confidence samples: high-confidence samples tend to exhibit less label noise, whereas low-confidence samples are more error-prone and should be treated carefully or excluded [2509.16548]. This observation is foundational for the subsequent SCAN pipeline. Rather than assuming all MC annotations are equally reliable, SCAN conditions annotation and training decisions on estimated model competence at the question level.

A related correctness estimator is also given for individual steps:
$$
c_t = \mathbb{E}_{r \sim P_\theta(\cdot \mid q, x_{\leq t})}[\mathcal{J}(r, a^*)].
$$
This estimator formalizes step correctness under continuation sampling and serves as the basis for denoised labels [2509.16548].

## 3. Self-denoising data synthesis pipeline

SCAN’s synthesis strategy begins with a lightweight language model, exemplified in the paper by Qwen2.5-Math-1.5B-Instruct, to generate responses for each question [2509.16548]. It then computes self-confidence over multiple responses per question and uses that value to determine which samples should enter the annotation pipeline.

For high-confidence samples, the framework distinguishes between already-clean positives and samples that still require MC step-wise annotation. If a response is fully correct and no errors are detected, it is used directly as a positive example [2509.16548]. If a response is incorrect, MC step-wise annotation is performed only when the sample is high-confidence, thereby reducing unnecessary computation on samples that are likely to be noisy or uninformative [2509.16548]. Low-confidence samples are discarded.

This selective sampling procedure is described as discarding noisy low-confidence data and avoiding annotation of already-clean positives, which cuts inference cost by over 94% compared to vanilla MC [2509.16548]. In the abstract, the paper states the same efficiency result in a different form: lightweight models with 1.5B parameters can produce high-quality annotations through self-denoising, enabling PRMs to achieve superior performance with only 6% of the inference cost required by vanilla MC estimation [2509.16548].

The design reflects a specific diagnosis of annotation failure. Because noise is concentrated where the annotation model is uncertain, the framework reduces compute on precisely those regions where additional rollouts are least trustworthy. This suggests that SCAN treats annotation budget allocation and label denoising as coupled problems rather than independent optimization targets.

## 4. Denoised labels and robust learning objective

SCAN’s learning procedure is explicitly noise-tolerant. For each annotated sample $(q, \mathbf{x}, \mathbf{c}, SC_\pi(q))$, the framework constructs soft labels using a tolerance distance $d$ [2509.16548]. For steps within $d$ steps before the predicted error, the denoised target is
$$
\hat{y}_{t} = \min(c_t / SC_{\pi}(q), 1),
$$
while for other steps, $\hat{y}_t = 1$ if $c_t > 0$ [2509.16548].

The paper describes this as **label smoothing via tolerance distance**. The purpose is to acknowledge that error localization in noisy MC data is imprecise: steps near the predicted error boundary are treated as uncertain rather than assigned hard correct or incorrect labels [2509.16548]. The reported ablation identifies $d=2$ as the best tradeoff; smaller values overfit noise, whereas larger values make the training signal too soft and performance lags [2509.16548].

A second mechanism is **confidence-wise reweighting**. SCAN rescales correctness by self-confidence using
$$
\hat{c}_i^* = \min(c_i / SC_{\pi}(q), 1).
$$
The paper states that this de-biases annotations by normalizing the varying capabilities of different completer models, and that it is especially important when merging data from models of varying strengths [2509.16548]. In effect, the framework discounts labels generated under weak annotation competence and amplifies those produced under stronger estimated competence, without requiring human calibration.

The final PRM objective is a modified binary cross-entropy:
$$
\mathcal{L}_{\text{SCAN}} = -\mathbb{E}\big[\hat{y}_t \log P_\theta(y_t \mid q, x_{\leq t}) + (1 - \hat{y}_t)\log(1 - P_\theta(y_t \mid q, x_{\leq t}))\big].
$$
The paper characterizes the net effect of this design as robust learning from weak supervision: selective inclusion of higher-quality data, soft labels near uncertain error boundaries, and self-confidence reweighting together enable PRMs to generalize despite systemic, model-induced label noise [2509.16548].

## 5. Empirical performance, scaling, and efficiency

The paper reports results for two SCAN datasets. **SCAN-Base** uses 101K samples annotated with a 1.5B model, and **SCAN-Pro** scales to 197K samples by incorporating additional 7B and 3B model data [2509.16548]. The larger dataset is reported to require 374.5 GPU hours on 8 A800 GPUs, corresponding to over 90% cost reduction relative to standard MC annotation [2509.16548].

The main quantitative outcomes are organized around best-of-8 evaluation, ProcessBench, noise reduction, and inference speed.

| Setting | Reported result | Context |
|---|---:|---|
| Qwen2.5-Math-7B-Scan-Base | 69.1 average accuracy | Trained on 101K SCAN data |
| PRMs trained on 264K PRM800K data | 69.3 average accuracy | Human-annotated reference |
| Qwen2.5-Math-7B-Scan-Pro | 70.1 average accuracy | Trained on 197K SCAN data |
| Qwen2.5-Math-7B-Scan-Base | 56.8 F1 | ProcessBench average across tasks |
| MC-only baseline | 19.9 F1 | ProcessBench reference |
| Qwen2.5-Math-7B-Scan-Pro | 59.1 F1 | ProcessBench average across tasks |
| Human-annotated PRM800K | 56.5 F1 | ProcessBench reference |

In best-of-8 evaluation, Qwen2.5-Math-7B-Scan-Base achieves 69.1 average accuracy, outperforming all MC-only 7B and 8B PRM baselines and remaining comparable to PRMs trained on 264K human-annotated PRM800K data at 69.3 [2509.16548]. With 197K samples, Qwen2.5-Math-7B-Scan-Pro rises to 70.1, matching or surpassing models trained with human and strong knowledge-distillation supervision [2509.16548].

On ProcessBench, which the paper uses for fine-grained error localization, Qwen2.5-Math-7B-Scan-Base achieves 56.8 F1, representing a 36.9 or 39.2 point gain over the MC-only baseline at 19.9 F1, while Qwen2.5-Math-7B-Scan-Pro reaches 59.1 F1 and surpasses the human-annotated PRM800K result of 56.5 F1 [2509.16548]. The abstract emphasizes the 39.2 F1 improvement from 19.9 to 59.1 on ProcessBench under robust learning from weak supervision [2509.16548].

The paper also reports direct noise reduction. For Qwen2.5-Math-7B, the noise ratio in synthetic data decreases from 51.8% under vanilla MC to 29.4% with SCAN denoising; for Llama-3.1-8B, it decreases from 56.2% to 19.1% on ProcessBench-like data [2509.16548]. Inference speed for SCAN-trained PRMs is given as 44 samples per second for Qwen2.5-7B, compared with 1.5 samples per second for generative critic models on Qwen2.5-Math-7B-Instruct [2509.16548].

The ablations reinforce the design claims. Removing self-confidence reweighting or tolerance labeling degrades performance, and scaling up dataset size with SCAN continues to improve results without clear overfitting, unlike conventional MC annotations [2509.16548]. Human evaluation is reported to show that SCAN-trained PRMs can identify subtle errors and correct answers even in challenging best-of-$n$ settings [2509.16548].

## 6. Relation to baselines, scope, and limitations

SCAN is evaluated against several classes of baselines: MC-only PRMs, PRMs trained on human-annotated PRM800K, and models trained with strong critic or knowledge-distillation supervision [2509.16548]. The reported comparisons are framed around synthetic-data efficiency rather than architectural novelty alone. The paper’s central empirical claim is that a compact synthetic dataset, when denoised and used in a robust learning framework, can outperform much larger or more expensive supervision sources [2509.16548].

The framework is also presented as **plug-and-play**: denoised MC data can be integrated as a component or substitute in any framework using MC estimation, and it can enhance downstream knowledge-distillation-based pipelines [2509.16548]. This suggests that SCAN is intended as a general-purpose annotation and training layer for PRM construction, not solely as a single-model recipe.

The paper also states several limitations. Purely MC-based methods cannot catch "false positive" errors, meaning responses marked correct may still contain hidden step-wise errors [2509.16548]. SCAN-trained PRMs retain limitations in deep semantic error detection and remain vulnerable to reward hacking in certain settings [2509.16548]. The paper therefore indicates that further gains may require hybrid approaches, including generative process reward models with richer supervision [2509.16548].

These limitations are important for interpretation. SCAN is not presented as eliminating the structural weaknesses of MC supervision; rather, it reduces major error sources sufficiently to make weak self-supervision practical. A plausible implication is that the framework is strongest where step correctness is correlated with final-answer recoverability, and less complete where reasoning defects are semantically subtle yet outcome-preserving.

## 7. Significance for scalable PRM training

The significance attributed to SCAN is that high-quality reward models can be trained from weak, noisy, self-supervision generated by lightweight models rather than exclusively from large-scale human annotation or large critic models [2509.16548]. The paper reports that even a 1.5B annotation model can support superior PRM performance through self-denoising, and that scaling the synthetic dataset further improves performance [2509.16548].

The broader implication advanced in the paper is that denoising MC at the source makes MC annotation scalable and cost-efficient without requiring strong critic LLMs, potentially widening access to PRM training across domains and institutions [2509.16548]. Within the paper’s results, this claim is supported by the combination of reduced noise ratios, strong ProcessBench gains, favorable best-of-8 accuracy, and substantial reductions in annotation cost [2509.16548].

In methodological terms, SCAN combines three elements that are often treated separately: diagnosis of annotation-model-induced noise, selective data synthesis conditioned on self-confidence, and robust PRM optimization with soft reweighted targets. The paper argues that this combination, rather than any single component alone, is what enables scalable and robust process reward learning from synthetic data [2509.16548].

Source: https://www.emergentmind.com/topics/self-denoising-monte-carlo-annotation-scan