DataPRM: Environment-Aware Reward Model

• DataPRM is an environment-aware generative process reward model that verifies intermediate analysis steps using tool-assisted, multi-turn interactions.
• It effectively addresses silent errors and recoverable grounding errors by inspecting execution states and enabling reflection-aware feedback.
• DataPRM enhances downstream policy performance with improved scaling, achieving benchmark gains up to 11.28% on interactive data analysis tasks.

Searching arXiv for DataPRM and closely related PRM literature to ground the article with current citations. DataPRM is an environment-aware generative process reward model for agentic data analysis. It was introduced to address a mismatch between conventional Process Reward Models (PRMs), which had achieved strong results in static domains such as mathematics, and the requirements of interactive data-analysis workflows, where correctness depends on latent execution state, tool use, and iterative exploration rather than on textual plausibility alone. In this formulation, DataPRM serves both as a verifier and as a reward model: it can actively interact with the environment to inspect intermediate states, detect silent errors, and assign reflection-aware step rewards that distinguish correct progress, recoverable grounding errors, and irrecoverable mistakes (Qiu et al., 27 Apr 2026).

1. Position within process reward modeling

DataPRM belongs to the PRM line of work, in which a model evaluates intermediate reasoning steps rather than only final outcomes. Earlier PRM work emphasized static reasoning settings, especially mathematics. A related later study, VersaPRM, documented that open-source math PRMs such as Math-Shepherd and Qwen-2.5-Math-PRM improve mainly on math or math-adjacent tasks and yield only marginal gains in domains such as Law, Philosophy, History, Biology, and Psychology; it summarized this as “Finding 1: Math PRMs struggle to generalize to broader domains” (Zeng et al., 10 Feb 2025).

DataPRM addresses a different but structurally related gap. Rather than extending PRMs from mathematics to broader static academic domains, it targets dynamic data-analysis tasks in which the agent interacts with code, files, tables, plots, and other environment artifacts. The paper frames these tasks as partially observable: the true environment state is latent, and a static verifier relies on an internal prior over that state, which is brittle under out-of-distribution data files and workflows (Qiu et al., 27 Apr 2026).

This positioning is important because it clarifies what DataPRM is not. It is not merely a larger or more general textual verifier, and it is not an outcome-reward model that scores only the final answer. Its central claim is that process supervision for data analysis must be environment-aware and must treat exploratory error recovery differently from genuine logical failure.

2. Failure modes that motivate the model

The motivating empirical study identified two failure modes of general-domain PRMs in agentic data analysis. The first is the silent error: code may execute without exception and still produce a wrong plot, a wrong aggregation, or a logically invalid conclusion. The second is the recoverable grounding error: an agent may make an early mistake involving a wrong column guess, schema mismatch, or failed file path, yet obtain useful feedback that enables subsequent correction (Qiu et al., 27 Apr 2026).

The paper reports two concrete observations. First, PRM-guided search underperforms simple baselines on a DABStep subset; even strong general PRMs can improve over single-path generation but still fail to beat Majority Voting. Second, general PRMs mis-handle environment-dependent errors in opposite directions: they assign low scores to steps that are locally wrong but useful for exploration and later correction, and they assign overly optimistic scores to silent errors because the code “runs” (Qiu et al., 27 Apr 2026).

These observations support a narrower interpretation of what makes data-analysis verification difficult. In static reasoning, local textual coherence may be a useful proxy for correctness. In data analysis, however, correctness often depends on the actual state of the execution environment rather than on syntax or surface plausibility. This suggests that a verifier for data-analysis agents must probe the environment rather than merely read the trace, and must preserve trial-and-error behavior when it produces informative feedback rather than immediately pruning it as failure.

3. Environment-aware generative verifier

DataPRM is explicitly described as a generative, ReAct-style verifier rather than a passive classifier over a trajectory step. Given the agent’s history $h_t$ and current step $\tau_t$, DataPRM takes as input

$$h_{t,0}^{prm} = h_t \oplus \tau_t = h_t \oplus (z_t, a_t, o_t),$$

where $z_t$ is reasoning, $a_t$ is action or code, and $o_t$ is observation. At internal verification step $k$, it generates a tuple

$\kappa_{t,k} = (\hat{z}_k, \hat{a}_k, \hat{o}_k),$

updates its internal context as

$$h_{t,k+1}^{prm} = h_{t,k}^{prm} \oplus \kappa_{t,k},$$

and terminates with

$$(\hat{z}_{K}, r_t, c_t)\sim \rho_{\phi}(\cdot \mid h_{t,K}^{prm}),$$

where $\tau_t$0 is the scalar reward and $\tau_t$1 is a natural-language rationale (Qiu et al., 27 Apr 2026).

A notable design choice is verifier memory across steps. The model retains its own prior verification feedback and appends it to the next step’s context: $\tau_t$2 This is intended to improve consistency across a multi-step analysis session (Qiu et al., 27 Apr 2026).

DataPRM is also tool-augmented. It can call query_document and query_image, connecting the verifier to long documents and images through expert models. The paper states that this matters because many data-analysis tasks depend on notebook cells, manuals, and plots rather than just code text. In operational terms, DataPRM combines intrinsic reasoning learned during training with extrinsic perception through tools (Qiu et al., 27 Apr 2026).

The environment-aware design is presented as the main mechanism for detecting silent errors. Rather than assuming that successful execution implies correctness, the verifier can inspect actual outputs, compare environment keys, or examine visual artifacts. The reported verification behavior—about 2.57 turns and 0.87 tool calls on average—indicates that it functions as an active checker rather than a one-shot scorer (Qiu et al., 27 Apr 2026).

4. Reflection-aware ternary reward and supervision pipeline

DataPRM uses a ternary reward space

$\tau_t$3

The labels are defined precisely. Strictly Correct corresponds to $\tau_t$4, meaning that the step is logically correct and directly advances the solution. Correctable Error / Grounding Error corresponds to $\tau_t$5, meaning that the step contains a minor or recoverable error, such as a wrong file path or a syntax/schema mistake, but produces useful feedback and can be corrected later. Irrecoverable Error corresponds to $\tau_t$6, meaning that the step contains a fundamental logic flaw or hallucination that leads to a dead end (Qiu et al., 27 Apr 2026).

The paper presents this as a “reflection-aware” strategy because it treats some failed steps as informative rather than terminal. The appendix gives a Bayesian interpretation in which reward decomposes into task progress $\tau_t$7 and information gain $\tau_t$8, with $\tau_t$9: $$h_{t,0}^{prm} = h_t \oplus \tau_t = h_t \oplus (z_t, a_t, o_t),$$0 Under this view, $$h_{t,0}^{prm} = h_t \oplus \tau_t = h_t \oplus (z_t, a_t, o_t),$$1 reflects progress plus confirmation of validity, $$h_{t,0}^{prm} = h_t \oplus \tau_t = h_t \oplus (z_t, a_t, o_t),$$2 reflects useful information gain without direct progress, and $$h_{t,0}^{prm} = h_t \oplus \tau_t = h_t \oplus (z_t, a_t, o_t),$$3 reflects neither progress nor useful information (Qiu et al., 27 Apr 2026).

The training data are built through a two-stage pipeline. First, the authors use diversity-driven trajectory generation. They crawl scientific and data-analysis files from GitHub, have human experts revise or extend some sources, generate queries using AutoSDT-style methods and expert LLMs, and then sample $$h_{t,0}^{prm} = h_t \oplus \tau_t = h_t \oplus (z_t, a_t, o_t),$$4 parallel trajectories per validated query using Qwen3-235B-A22B-Instruct as the policy model $$h_{t,0}^{prm} = h_t \oplus \tau_t = h_t \oplus (z_t, a_t, o_t),$$5. A judge model based on DeepSeek-V3.2 checks whether final answers differ, and the pipeline retains only those query instances where not all final answers are identical (Qiu et al., 27 Apr 2026).

Second, they perform knowledge-augmented step-level annotation. Qwen3-235B-A22B-Instruct performs initial step scoring and error attribution; AutoManual merges similar failure types; the merged categories are curated into few-shot examples and injected as external knowledge; and DeepSeek-V3.2 serves as the expert annotator that assigns final ternary rewards. The abstract states that the pipeline constructs over 8K high-quality training instances, while the detailed description states that it yields more than 7K annotated instances for process supervision. In 100 manual spot-checks, the annotation pipeline achieved 86.0% raw accuracy and Quadratic Weighted Cohen’s $$h_{t,0}^{prm} = h_t \oplus \tau_t = h_t \oplus (z_t, a_t, o_t),$$6 (Qiu et al., 27 Apr 2026).

A common misconception is that reward discretization of this kind simply softens binary supervision. In DataPRM, the intermediate value $$h_{t,0}^{prm} = h_t \oplus \tau_t = h_t \oplus (z_t, a_t, o_t),$$7 is not a generic uncertainty score; it has a specific semantic role tied to recoverable grounding errors and information gain. The model’s reward design is therefore bound to the structure of exploratory interaction rather than only to label smoothing.

5. Test-time scaling and benchmark performance

DataPRM is used for Best-of-$$h_{t,0}^{prm} = h_t \oplus \tau_t = h_t \oplus (z_t, a_t, o_t),$$8 inference and also evaluated under Beam Search and Diverse Verifier Tree Search (DVTS). The paper reports that DataPRM improves downstream policy LLMs by 7.21% on ScienceAgentBench and 11.28% on DABStep using Best-of-$$h_{t,0}^{prm} = h_t \oplus \tau_t = h_t \oplus (z_t, a_t, o_t),$$9 inference. It further states that, with only 4B parameters, DataPRM outperforms strong baselines and exhibits robust generalizability across diverse Test-Time Scaling strategies (Qiu et al., 27 Apr 2026).

The detailed benchmark results reported in the paper are summarized below.

Setting	Metric	Result
ScienceAgentBench, Best-of-$z_t$0	Improvement	7.21%
DABStep, Best-of-$z_t$1	Improvement	11.28%
ScienceAgentBench, $z_t$2	SR	24.36 / 25.64 / 25.64
DABStep (Avg.), $z_t$3	Score	37.11 / 39.77 / 40.89

The comparison set includes Majority Vote, LLM-as-a-judge, Self-Rewarding, Math-Shepherd-PRM-7B, Qwen2.5-Math-PRM-7B and 72B, ReasonFlux-PRM-7B, ThinkPRM, and GenPRM. A notable observation is that larger PRMs do not necessarily scale well with larger candidate pools; some even degrade as $z_t$4 increases. DataPRM, by contrast, is reported to exhibit effective scaling, with better Best-of-$z_t$5 performance as the candidate pool expands (Qiu et al., 27 Apr 2026).

The paper also describes an ablation suite over three components: Env for code environment interaction, Multi for multi-turn interaction, and Refl for reflection-aware ternary reward. The variants include CoT, Single-turn Code w/ Env, Multi-turn Code w/o Env, Multi-turn Code w/ Env, and Full DataPRM. The reported pattern is that environment access helps, multi-turn interaction helps more when paired with environment access, and the ternary reward provides the final boost. A separate filtering ablation reports that aggressive filtering is not always best; unfiltered or less aggressively filtered diverse trajectories often yield stronger scaling, which the paper interprets as evidence that trajectory diversity matters more than purity for PRM training in this setting (Qiu et al., 27 Apr 2026).

6. Reinforcement learning integration, limits, and terminological scope

DataPRM is integrated into reinforcement learning through GRPO with clip-higher and token-level loss. The objective is written as

$z_t$6

and the total reward is

$z_t$7

with $z_t$8. The group-normalized advantage is

$z_t$9

The paper also enforces a terminal-step consistency check: if the terminal PRM score conflicts with the outcome reward, the terminal PRM reward is aligned to the outcome reward (Qiu et al., 27 Apr 2026).

The reinforcement-learning results are reported as 78.73% on DABench and 64.84% on TableBench, outperforming both the SFT baseline and outcome-only reward training. The training dynamics reported in the paper are also substantive: outcome-only training leads to entropy collapse, with entropy dropping to about 0.12 after 200 steps and reward plateauing, whereas with process supervision entropy remains around 0.18 and reward keeps rising. The process-reward-trained model also improves pass@3, which the authors interpret as evidence that process supervision helps preserve multiple promising hypotheses rather than collapsing too early (Qiu et al., 27 Apr 2026).

The distinction from outcome-reward baselines follows directly from these results. Outcome-only rewards do not indicate which intermediate steps were useful, cannot distinguish recoverable exploration from dead-end logic, and tend to under-train exploration and environment adaptation. DataPRM instead supplies step-level supervision, environment-aware verification, and partial credit for correctable mistakes (Qiu et al., 27 Apr 2026).

Two boundaries of scope are worth noting. First, DataPRM addresses agentic data analysis specifically; its design assumptions are tied to interactive code environments, silent execution-level failure, and exploratory grounding. Second, the name should not be conflated with the distinct literature on PRAM—the Post-Randomization Method used for privacy-preserving perturbation of categorical data—which concerns transition matrices, estimating equations, differential privacy, and semiparametric efficiency rather than process reward modeling (Tian et al., 2024).