Test-Time Scaling with Reflective Generative Model (2507.01951v1)

Abstract: We introduce our first reflective generative model MetaStone-S1, which obtains OpenAI o3's performance via the self-supervised process reward model (SPRM). Through sharing the backbone network and using task-specific heads for next token prediction and process scoring respectively, SPRM successfully integrates the policy model and process reward model(PRM) into a unified interface without extra process annotation, reducing over 99% PRM parameters for efficient reasoning. Equipped with SPRM, MetaStone-S1 is naturally suitable for test time scaling (TTS), and we provide three reasoning effort modes (low, medium, and high), based on the controllable thinking length. Moreover, we empirically establish a scaling law that reveals the relationship between total thinking computation and TTS performance. Experiments demonstrate that our MetaStone-S1 achieves comparable performance to OpenAI-o3-mini's series with only 32B parameter size. To support the research community, we have open-sourced MetaStone-S1 at https://github.com/MetaStone-AI/MetaStone-S1.

• The paper introduces MetaStone-S1 that unifies reasoning and evaluation through a self-supervised process reward model (SPRM) to optimize test-time scaling.
• It achieves parameter efficiency by reducing annotation requirements and PRM overhead by over 99% while maintaining competitive accuracy across benchmarks.
• Empirical results show significant gains on complex mathematical reasoning tasks and robust zero-shot generalization to out-of-domain challenges.

Test-Time Scaling with Reflective Generative Model: A Technical Overview

The paper "Test-Time Scaling with Reflective Generative Model" (2507.01951) introduces MetaStone-S1, a LLM architecture that unifies reasoning and process evaluation through a self-supervised process reward model (SPRM). This approach is designed to address the computational and annotation inefficiencies of traditional process reward models (PRMs) in test-time scaling (TTS) scenarios, particularly for mathematical and reasoning-intensive tasks.

Motivation and Context

Recent advances in LLMs have demonstrated that test-time scaling—allocating additional compute at inference via strategies such as massive sampling, candidate scoring, and search over reasoning paths—can substantially improve performance on complex reasoning benchmarks. However, these gains often come at the cost of significant computational overhead, especially when separate, large-scale PRMs are used for candidate evaluation. Moreover, existing PRMs typically require process-level annotations, which are expensive and difficult to obtain at scale.

The paper identifies two main TTS paradigms:

• Internal TTS: Extending the model's reasoning process (e.g., via long chain-of-thought) during inference.
• External TTS: Generating multiple candidate solutions and selecting the best via a reward model.

The authors focus on external TTS and propose a reflective generative model that integrates the policy and reward models, aiming to reduce both parameter count and annotation requirements.

Methodology

Reflective Generative Form and SPRM

The core innovation is the Self-supervised Process Reward Model (SPRM), which shares the backbone with the policy model and introduces two lightweight, task-specific heads:

• Token Prediction Head: For next-token generation.
• Process Scoring Head: For step-level evaluation of reasoning trajectories.

This unified architecture enables simultaneous training of both reasoning and evaluation capabilities, with the following key properties:

• Parameter Efficiency: The SPRM head introduces only ~26M additional parameters (for a 7B model), reducing PRM overhead by over 99% compared to traditional approaches.
• On-Policy Optimization: Both reasoning and reward heads are trained end-to-end, leveraging shared representations and enabling on-policy learning.
• Self-supervised Process Reward Loss (SPR Loss): Instead of requiring process-level annotations, the model uses only final answer correctness as supervision. A dynamic weighting scheme filters out noisy step-level labels, focusing optimization on steps consistent with the final outcome.

Inference and Test-Time Scaling

At inference, the model supports multiple reasoning effort modes (low, medium, high), corresponding to different numbers of sampled candidate solutions ($k=2,8,32$). For each candidate, the SPRM head evaluates the step-level reasoning trajectory, and the candidate with the highest geometric mean score is selected as the final answer.

The model also supports integration with search-based TTS methods such as Monte Carlo Tree Search (MCTS), where SPRM provides step-level value estimates for efficient node evaluation.

Empirical Results

The paper presents comprehensive evaluations on mathematical (AIME24, AIME25) and out-of-distribution (LiveCodeBench, C-EVAL) benchmarks. Key findings include:

• Consistent Performance Gains: Across all model sizes (1.5B, 7B, 32B), MetaStone-S1 with SPRM outperforms baselines lacking SPRM, with gains up to +18.6 points on AIME24 for the 1.5B model.
• Parameter Efficiency: SPRM-equipped models match or exceed the performance of models using separate 72B PRMs, despite the SPRM head being orders of magnitude smaller.
• Competitiveness with Closed-Source Models: MetaStone-S1-32B-high achieves 85.2% on AIME24 and 73.6% on AIME25, comparable to OpenAI o3-mini-medium (79.6% and 74.8%, respectively), and surpasses other open-source models of similar or larger size.
• Scaling Law: The authors empirically establish that performance improves logarithmically with the total computation budget (model size × reasoning length), with diminishing returns beyond Best-of-32 sampling.
• Generalization: SPRM demonstrates strong zero-shot generalization to out-of-domain tasks (e.g., code generation), without task-specific fine-tuning.

Ablation and Analysis

• SPRM vs. Traditional PRMs: SPRM consistently outperforms both outcome reward models (ORMs) and traditional PRMs, even when the latter are much larger.
• Self-supervised Optimization: The SPR Loss yields a larger score gap between correct and incorrect solutions compared to standard BCELoss, indicating improved discriminative capability and robustness to label noise.
• Aha Moment: Training dynamics reveal a distinct "aha moment" where the model begins to sharply distinguish between correct and incorrect reasoning trajectories, supporting the efficacy of the self-supervised approach.

Implications and Future Directions

The reflective generative model paradigm, as instantiated by MetaStone-S1, offers several practical and theoretical implications:

• Resource-Efficient TTS: By unifying reasoning and evaluation, the approach enables high-quality TTS with minimal additional computational and parameter overhead, making advanced inference strategies more accessible for smaller-scale deployments.
• Annotation Efficiency: The self-supervised SPR Loss eliminates the need for process-level annotations, facilitating scalable training on large, weakly-labeled datasets.
• On-Policy Learning: Joint optimization of reasoning and reward heads may lead to more robust and aligned models, particularly in domains where off-policy PRM training is suboptimal.
• Integration with Search: The step-level evaluation capability of SPRM is well-suited for integration with advanced search-based TTS methods, such as MCTS, potentially enabling further gains in complex reasoning tasks.

Potential future research directions include:

• Extending the reflective generative model framework to other domains (e.g., scientific reasoning, multi-modal tasks).
• Investigating the interplay between internal and external TTS strategies within the unified architecture.
• Exploring real-time, adaptive reasoning enhancement during inference, leveraging the on-policy, step-level feedback provided by SPRM.

Conclusion

"Test-Time Scaling with Reflective Generative Model" presents a technically sound and empirically validated approach to efficient, scalable TTS in LLMs. By unifying reasoning and process evaluation within a single, self-supervised architecture, the work demonstrates that high-quality step-level guidance and competitive benchmark performance can be achieved with minimal additional resources. The open-sourcing of MetaStone-S1 further supports reproducibility and future research in this area.