---
title: Adversarially Trained PRMs (APRM)
url: https://www.emergentmind.com/topics/adversarially-trained-prms-aprm
type: topic
---

# Adversarially Trained PRMs (APRM)

Adversarially Trained PRMs (APRM) are a class of process reward models for large language models (LLMs) in which reward modeling is framed as a two-player adversarial game between a generator that produces plausible but incorrect reasoning steps and a detector (process reward model, PRM) that learns to discriminate between correct and adversarially generated incorrect steps. This framework eliminates the need for expensive human step-level annotation by using algorithmic oracles for correctness, and leverages dynamic, adversarial curricula to drive improved generalization and robustness on complex reasoning tasks [2511.22888].

## 1. Formal Framework: Game-Theoretic Formulation

APRM recasts PRM training as a two-player, general-sum, strongly monotone game:

- **Generator $G_\theta$**: Given a valid reasoning step $s$ from distribution $p(s)$, generates a perturbed step $a \sim \pi_\theta(a|s)$. The generator's objective is to produce incorrect but plausible steps that can deceive the detector.
- **Reward Model $R_\phi$**: Receives the (original step, perturbed step) pair $(s, a)$ and predicts a binary label $y' \in \{0, 1\}$ (correct/incorrect). True correctness $y(a)$ is supplied by a symbolic oracle.
- **Reward Definitions**:
  - $r_R(y, y') = +1$ if $y = y'$, $-1$ otherwise (reward to the PRM for correct detection).
  - $r_G(y, y') = +1$ if $y = 0$ and $y' = 1$ (generator fools the PRM), $0$ if $y = 0$ and $y' = 0$, $-1$ if $y = 1$ (generator fails to corrupt).
- **Utilities**: 
  - $U_G(\pi_\theta, \pi_\phi) = \mathbb{E}[r_G(y(a), y')]$
  - $U_R(\pi_\theta, \pi_\phi) = \mathbb{E}[r_R(y(a), y')]$
  - Regularization: KL-divergence to fixed reference policies and entropy regularizers are added to both objectives for exploration and stability.
- **Optimization**: Both players are trained alternately (using PPO and Optimistic Gradient Descent-Ascent, OGDA) to minimize their respective losses. Training converges to a Nash equilibrium in mixed strategies.

This adversarial setup ensures the PRM is confronted with increasingly subtle errors, challenging the model beyond the limitations of static, hand-annotated datasets [2511.22888].

## 2. Model Architectures and Inputs

Both $G_\theta$ and $R_\phi$ are parameterized as decoder-only transformers, initialized from a pretrained checkpoint (Llama-3.1-8B):

- **Generator Inputs**: Tokenized correct step $s$ plus a “perturb–this–step” prompt. Output is a corrupted or altered reasoning step $a$.
- **Reward Model Inputs**: Full question $Q$, partial chain $S$ up to the current step, and candidate step $a$. Output is a scalar logit for $P(\text{correct}|Q, S)$.
- **No extra heads or adapters are used**; the entire model is fine-tuned end-to-end.

The design allows for direct transfer and compatibility with a wide range of transformer backbone architectures [2511.22888].

## 3. Training Algorithm and Curriculum

APRM employs a continually evolving adversarial curriculum, formalized as follows:

```python
# High-level APRM pseudocode (syntax adapted for clarity)
Initialize Gθ, Rφ ← pretrained Llama-3.1-8B
B_neg = ∅  # Adversarial negative buffer
for iteration in range(T):
    batch = sample_problems()
    for s_i in batch:
        a_i = Gθ.sample(s_i)
        y_i = oracle(s_i, a_i)  # Symbolic correctness label
        B_neg.append((s_i, a_i, y_i))
        y'_i = Rφ.predict(Q_i, S_i || a_i)
        r_G = compute_reward(y_i, y'_i)
    # R-update: mix gold steps and adversarial negatives
    R_batch = build_R_training_batch(batch, B_neg)
    update Rφ by PPO+OGDA
    # G-update: freeze R, train G on –r_G
    G_batch = build_G_training_batch(batch, Rφ)
    update Gθ by PPO+OGDA
```

- **Alternating optimization**: 5 PPO steps for each player alternately, freezing the other. The adversarial negative buffer maintains hard negatives across training and prevents forgetting.
- **Emergent supervision**: The generator produces adversarial steps judged by a symbolic oracle; the PRM is trained using a balanced mix of gold (human or algorithmic) and adversarial data.

This iterative curriculum generates a dynamic set of ever-harder negatives, facilitating robust step-level detection [2511.22888].

## 4. Emergent Step-Level Supervision and Oracle Evaluation

Unlike conventional PRMs that require hand-labeled step-level correctness, APRM relies exclusively on:

- **Algorithmic oracle supervision**: Correctness labels are computed via symbolic equivalence checks, entity matching, or other non-LLM static oracles.
- **Adversarial generator**: The generator $G_\theta$ learns to craft increasingly subtle semantic or contextual errors not present in the pretraining distribution or static curated datasets.
- **Fine-grained detection**: $R_\phi$ must develop rich, contextualized step representations to separate “correct” steps from a growing diversity of “adversarial” negatives.

This process enables robust step-level supervision without manual annotation, and the reward model learns to recognize both syntactic and semantic invalidity [2511.22888].

## 5. Empirical Benchmarks and Performance

APRM demonstrates improved solver and detector performance across multiple domains:

| Setting              | APRM Improvement (pp) | Baseline           | Metric                      |
|----------------------|----------------------|--------------------|-----------------------------|
| Math (avg 5 tasks)   | +3.4                 | ReST-MCTS          | Solver accuracy             |
| OOD: JEEBench        | +5.3                 | ReST-MCTS          | Solver accuracy             |
| SciBench (GPT-OSS-20B)| 63.0 vs 61.8         | ReST-MCTS          | Cross-domain science        |
| RL posttraining      | +6.8                 | ReST-MCTS, Outcome sup. | RL with GRPO         |

- **Ablations** reveal entropy regularization and OGDA are critical for solver accuracy: removing both reduces performance by 6.8 percentage points.
- **Generalization**: APRM-PRMs achieve gains on out-of-distribution (OOD) tasks (e.g., JEEBench, cross-domain SciBench) not seen during training.
- **Adversarial Curriculum**: The generator produces hard negatives including semantically valid but contextually false reasoning steps, far beyond static augmentation [2511.22888].

## 6. Robustness, Generalization, and Theoretical Guarantees

APRM leverages strong monotonicity via regularized game dynamics and OGDA optimization:

- **Nash equilibrium**: Training converges to a stable, non-cycling fixed point in the space of generator-detector policies.
- **Adaptive negative mining**: Generator scale correlates with error diversity; larger models produce subtler errors, further enhancing PRM robustness.
- **Cross-domain transfer**: APRM reward models generalize to identifying non-obvious scientific errors (e.g. physics unit conversion in chemistry) without domain-specific annotation.
- **Theoretical stability**: The Nash equilibrium property provides guarantees of convergence and robustness, in contrast to heuristic or purely supervised PRM pipelines [2511.22888].

## 7. Significance and Extensions

APRM establishes a data-efficient paradigm for robust step-level validation in complex reasoning tasks, removing dependence on human annotation and static datasets. Its key contributions include:

- **Dynamic, adversarially constructed training curricula**, yielding a “moving target” for the reward model.
- **Empirically validated gains** in both mathematical and scientific domains, including OOD robustness.
- **General applicability** to any process reward model context where step-level perturbations can be algorithmically evaluated.
- **Theoretical underpinnings** rooted in monotone games and convergence under OGDA, supporting further research into scalable, stable adversarial training for model-based oversight [2511.22888].

A plausible implication is that adversarial training of PRMs represents a scalable pathway to robust automated oversight of LLM-based reasoners, particularly in domains with limited annotated step-level data and evolving error distributions.

Source: https://www.emergentmind.com/topics/adversarially-trained-prms-aprm