PRM-BiasBench: Bias in PRM Evaluation

• PRM-BiasBench is a benchmark paradigm that evaluates bias-sensitive behavior in language models by scoring outputs for both bias and utility through PRM frameworks.
• It integrates cross-lingual inference-time evaluation and process reward model analysis, measuring metrics such as false-positive rates, composite scores, and precision to assess fairness and reasoning robustness.
• The framework emphasizes a modular, model-agnostic design that separates candidate generation from PRM scoring, offering actionable insights into aggregation methods and calibration trade-offs.

Searching arXiv for papers relevant to “PRM-BiasBench” and the supplied IDs. arxiv_search.query{"search_query":"all:\"PRM-BiasBench\" OR all:\"PRMBench\" OR all:\"Process Reward Models\" bias benchmark", "max_results": 10} “PRM-BiasBench” (Editor's term) denotes a benchmark-oriented family of evaluation frameworks in which a PRM provides the evaluative layer for bias-sensitive model behavior. In the current literature, the term is most directly suggested by a bilingual framework for inference-time bias mitigation in LLMs, where a preference-ranking model scores candidate completions for bias and utility, and by a bias-focused line of work on process reward models, where hidden bias is operationalized through step-level false positives, search failures, and policy misalignment (Khan, 10 Dec 2025, Agrawal et al., 8 Jun 2026). Taken together, these works suggest a benchmark concept centered on how PRM signals are produced, calibrated, and consumed under socially sensitive language generation or multi-step reasoning.

1. Conceptual scope and terminological ambiguity

The term PRM is not used uniformly across the cited literature. In one usage, PRM denotes a preference-ranking model that scores candidate outputs on bias and utility during inference-time mitigation for socially sensitive prompts. In another, PRM denotes a process reward model that evaluates intermediate reasoning steps or trajectories and is later used for Best-of-$N$, guided decoding, or policy optimization (Khan, 10 Dec 2025, Feng et al., 16 May 2025).

This dual usage matters because the associated notion of “bias” also differs. In the language-generation setting, bias refers to stereotypical, harmful, or unfair content, evaluated cross-lingually in English and Urdu. In the reasoning setting, bias refers to systematic misalignment in step scoring, especially the tendency to overcredit plausible but incorrect reasoning steps, producing high false-positive rates and downstream selection errors (Khan, 10 Dec 2025, Agrawal et al., 8 Jun 2026).

PRM usage	Benchmark object	Primary signals
Preference-ranking model	Single-word completions for socially sensitive prompts	bias, utility, composite score
Process reward model	Step-level reasoning traces and their downstream use	FPR, FNR, precision, PRMScore, Best-of-$N$, guided decoding

A plausible implication is that “PRM-BiasBench” is best treated as a benchmark schema rather than a single fixed dataset: a generator or policy proposes candidates, a PRM scores them, and the benchmark measures how those scores interact with fairness, utility, correctness, or robustness.

2. Cross-lingual inference-time bias mitigation

The clearest direct instantiation of the term appears in the English–Urdu study of inference-time bias mitigation. That framework contains 200 English prompts and 200 Urdu translations, for a total of 400 prompt–language pairs, each with a single $[blank]$ to be filled by one word. The covered categories include gender, ethnicity, nationality, religion, age, disability, socioeconomic status, profession, criminality, and appearance / body image (Khan, 10 Dec 2025).

Cross-lingual alignment is handled by translation that preserves semantic structure, the grammatical role of the blank, and sociocultural context. The supplied description states that the Urdu prompts are crafted “to preserve sociocultural and grammatical structure,” and it does not report automated alignment metrics or bilingual lexicons. It also states that the dataset and code are described but not explicitly released with URL or license in the paper text (Khan, 10 Dec 2025).

The benchmarked methods share the same generator–evaluator decomposition. GPT-3.5-turbo serves as the generator, while GPT-4o-mini is used as a zero-shot PRM-based scorer. Three methods are compared: baseline single-word generation, PRM-Select best-of-$N$, and PRM-Sequential refinement guided by PRM critiques. The setup is explicitly model-agnostic, requires no retraining / fine-tuning, and is described as lightweight and modular, since the PRM and generator remain separate components (Khan, 10 Dec 2025).

3. Metrics, workflow, and empirical profile in the language setting

In this instantiation, the PRM assigns two scores to each candidate $c$: $\text{bias}(c)\in[0,1]$ and $\text{utility}(c)\in[0,1]$. These are combined through the composite score

$$S(c) = (1 - \alpha)\cdot \text{bias}(c) + \alpha \cdot \text{utility}(c),$$

with $\alpha = 0.5$, so that

$$S(c) = 0.5 \cdot \text{bias}(c) + 0.5 \cdot \text{utility}(c).$$

No additional fairness index is introduced; mean bias, utility, and composite are the central reporting metrics (Khan, 10 Dec 2025).

The three compared workflows are structurally distinct. The baseline takes the first one-word completion from GPT-3.5-turbo. PRM-Select samples $N$0 candidates, scores each with the PRM, and returns $N$1. PRM-Sequential starts from a baseline completion, requests a PRM critique, and iteratively refines the output for up to 5 refinement steps; the final refined word is taken as output (Khan, 10 Dec 2025).

The reported results establish a pronounced cross-lingual asymmetry at baseline. For bias, the baseline scores are 0.9525 in English and 0.755 in Urdu; for utility, 0.985 in English and 0.85 in Urdu; for the composite score, 0.96875 in English and 0.8025 in Urdu. PRM-Select improves both languages, raising English bias to 0.9765 and Urdu bias to 0.96, while utility reaches 1.0 in English and 0.9825 in Urdu. PRM-Sequential yields the highest bias scores—0.99 for English and 0.975 for Urdu—but Urdu utility drops to 0.8825, which the paper interprets as over-correction toward overly generic or abstract words (Khan, 10 Dec 2025).

The paper also states that no human annotator study was conducted. GPT-4o-mini is the sole judge for both bias and utility, and there are no inter-annotator agreement statistics or human calibration checks. This makes the framework operationally reusable, but it also means that the benchmark measures alignment to the PRM’s judgments rather than to an independently validated human standard (Khan, 10 Dec 2025).

4. Hidden bias in process reward models

A second, technically distinct sense of PRM-BiasBench emerges from work on process reward models for reasoning. Here the benchmark focus is not social bias in generated content but a hidden bias in process supervision itself: PRMs tend to overcredit plausible but incorrect reasoning steps, yielding high step-level false-positive rates. The paper attributes this to severe step-level label imbalance in datasets such as PRM800K, where 73.1% of steps are labeled correct and 26.9% incorrect, even though only 14.2% of full trajectories are correct and 85.8% incorrect (Agrawal et al., 8 Jun 2026).

Standard PRMs are described as being trained with pointwise binary cross-entropy over step labels,

$N$2

The paper argues that this objective amplifies overcredit bias because it fits absolute step labels independently and does not directly enforce that, for a fixed prefix, a correct next step outrank a plausible but incorrect alternative (Agrawal et al., 8 Jun 2026).

The empirical signature of this hidden bias is a high false-positive rate. On PRMBench, the baseline Qwen-PRM-7B records positive step accuracy (TPR) 95.36%, negative step accuracy (TNR) 30.66%, FPR 69.34%, FNR 4.27%, Precision 89.40%, and PRMScore 65.50. The proposed PRISM training framework reduces FPR to 47.13%, raises TNR to 52.86%, increases Precision to 91.93%, and improves PRMScore to 68.00, while increasing FNR to 12.69% (Agrawal et al., 8 Jun 2026).

The benchmark logic is therefore asymmetric. The paper proves that false positives impose a hard ceiling on Best-of-$N$3 performance, whereas false negatives mainly slow convergence. With base correctness rate $N$4, false-positive rate $N$5, and false-negative rate $N$6, if $N$7 then

$N$8

so the asymptotic accuracy ceiling decreases monotonically with $N$9. By contrast, if $[blank]$0 and $[blank]$1, then

$[blank]$2

This establishes a benchmark principle specific to process supervision: precision-first step scoring is often more consequential than maximizing recall (Agrawal et al., 8 Jun 2026).

5. Ranking, aggregation, and solver-as-judge baselines

The process-reward literature further shows that a PRM benchmark cannot stop at step-level classification. It must also evaluate how PRM signals are aggregated at test time. A Bayesian analysis of test-time scaling derives the optimal answer-level decision rule as a weighted majority vote

$[blank]$3

with

$[blank]$4

The first term is the PRM signal term; the second is the LLM signal term. The same work reports that the optimal weighting functions differ significantly across LLM–PRM pairs and often assign substantial negative weights, meaning that low PRM scores are evidence against an answer rather than merely weak evidence for it. In experiments across 5 LLMs and 7 PRMs, the calibration method is reported to surpass vanilla weighted majority voting while using only 21.3% of the computation (Kuang et al., 15 Oct 2025).

A related challenge comes from strong RL-trained solvers that already exhibit emergent PRM capability. On ProcessBench, DeepSeek-R1 and QwQ-32B achieve average F1 scores of 83.5 and 83.7, respectively, exceeding several explicit PRM baselines such as Qwen2.5-Math-PRM-72B at 78.3 and Qwen2.5-Math-PRM-7B at 73.5. The same study reports that external PRM reranking often fails to beat simple majority voting on strong RL models, while Self-PRM can improve accuracy but still exhibits low precision (<10%) on difficult problems (Feng et al., 16 May 2025).

This suggests that a rigorous PRM-BiasBench should include at least four baselines: explicit PRMs, solver-as-judge prompting, Self-PRM reranking, and answer-only majority voting. Otherwise, benchmark conclusions can confuse PRM quality with aggregation quality or ignore the possibility that solver competence itself induces strong evaluative behavior (Feng et al., 16 May 2025, Kuang et al., 15 Oct 2025).

6. Benchmark design principles, limitations, and broader significance

The bias-benchmark literature outside PRMs provides a useful methodological analogue. A bias-injection sandbox for fairness algorithms defines bias(stress)-testing as injecting controlled bias into an otherwise unbiased pipeline and then evaluating interventions against the unbiased ground truth rather than only against biased observational data. Its design includes an explicit generative model, modular bias operators, separate biased and unbiased evaluation, and fidelity to a Bayes-optimal reference classifier (Akpinar et al., 2022). A plausible implication is that a mature PRM-BiasBench should likewise separate observational performance from counterfactual reference performance wherever such a reference can be specified.

Across the cited PRM literature, several limitations recur. The English–Urdu framework is restricted to single-word completions, relies on GPT-4o-mini as the sole judge, and does not include a human annotator study; the paper also notes that Urdu translations “cannot fully capture culturally nuanced stereotypes” (Khan, 10 Dec 2025). The process-reward PRISM framework explicitly trades lower FPR for higher FNR, so its gains come with a nontrivial precision–recall tradeoff (Agrawal et al., 8 Jun 2026). Optimal aggregation results depend strongly on the particular LLM–PRM pair and on calibration quality, with large per-question variance in the ideal weighting function (Kuang et al., 15 Oct 2025). The RL-emergence results are limited to math reasoning, and the mechanism by which PRM capability emerges earlier than final-answer accuracy remains unresolved (Feng et al., 16 May 2025).

In that sense, PRM-BiasBench is best viewed not as a settled benchmark artifact but as a developing benchmark paradigm. In one branch, it formalizes cross-lingual inference-time bias mitigation through PRM-based scoring of bias and utility. In another, it formalizes hidden bias in process supervision through FPR-sensitive evaluation, calibrated aggregation, and downstream search or policy tests. The unifying technical idea is that PRM signals are not neutral metadata: they are intervention points whose bias structure directly shapes selection, decoding, optimization, and ultimately the observed behavior of the system (Khan, 10 Dec 2025, Agrawal et al., 8 Jun 2026).