Skywork-Reward-V2-Llama-3.1-8B

• The paper introduces a reward model that adds a simple linear 'reward head' to the Llama-3.1-8B backbone, projecting terminal hidden states into scalar reward scores.
• It utilizes a rigorous two-stage human-AI curation process on a 26-million pair subset from the SynPref-40M dataset to maximize preference realism and data quality.
• Empirical benchmarks show significant performance gains over competitors, though the model remains vulnerable to token-space adversarial attacks requiring further robustness improvements.

Skywork-Reward-V2-Llama-3.1-8B is a state-of-the-art, open-weight reward model targeting reinforcement learning from human feedback (RLHF) use cases in LLMs. Built upon the Meta-Llama-3.1-8B-Instruct transformer backbone, the model features a minimalistic architectural modification: a single linear "reward head" projects the terminal hidden state into a scalar reward score. Training utilizes a rigorously curated subset of 26 million human- and AI-annotated preference pairs from the 40-million-scale SynPref-40M dataset, with a two-stage human–AI annotation workflow focused on maximizing preference realism and dataset coverage. Empirical results establish strong and broad outperformance across industry-standard reward model benchmarks versus both open and closed alternatives, while ablation studies highlight the primacy of careful curation over brute-force data scaling. Despite these strengths, emerging work reveals severe vulnerabilities to adversarial optimization in token space, raising ongoing challenges for robust reward modeling.

1. Model Architecture and Loss Objective

Skywork-Reward-V2-Llama-3.1-8B is anchored in the Llama-3.1-8B-Instruct transformer, retaining all backbone layers and the tokenizer from the original checkpoint. A reward head—a single linear projection over the last token embedding—yields a scalar output:

$$h = \mathrm{LM}_\theta([x;\,y]) \in \mathbb{R}^d, \quad r_\theta(x,y) = w^\top h + b, \quad w \in \mathbb{R}^d, \; b \in \mathbb{R}.$$

No mixture-of-experts or multi-head variants are employed for the 8B model.

The model is trained in a Bradley–Terry pairwise setup. For a triplet $(x, y^+, y^-)$ where $y^+$ is preferred over $y^-$, the objective is

$$L(\theta) = -\mathbb{E}_{(x,y^+,y^-)\sim\mathcal{D}} \left[\log \sigma \left(r_\theta(x,y^+) - r_\theta(x,y^-)\right)\right] + \lambda \|\theta\|_2^2,$$

where $\sigma(\cdot)$ is the logistic sigmoid function and $\lambda$ is a weight decay parameter (Liu et al., 2 Jul 2025, Liu et al., 2024).

2. Dataset Composition and Human-AI Curation Workflow

Skywork-Reward-V2-Llama-3.1-8B leverages SynPref-40M, an integrated collection of 40 million preference pairs aggregated from public and proprietary sources (HelpSteer2, OpenHermes, Stack-Exchange Preferences, and others). For training, a specifically curated 26 million pair subset is selected via an iterative two-stage process:

• Stage 1: Human-in-the-Loop Iteration
  • Seed pool is tagged with LLM-generated preference attribute tuples (task category, objectivity, controversiality, desired attributes, annotation guidelines).
  • Human annotators verify a “gold” seed set, while LLM annotators label a “silver” set.
  • Reward models are trained on silver, evaluated on gold, and ambiguous or error-prone regions are adaptively targeted (using a confidence-weighted KNN retrieval scheme).
  • Labeled pairs are iteratively appended and re-evaluated.
• Stage 2: Automatic Large-Scale Expansion
  • Reward models trained on Stage 1 data are used to identify further preference pairs with high RM agreement.
  • LLMs act as automatic juries for pairs with borderline RM agreement, expanding coverage with minimized further human cost.

For the 40M-variant, an additional 14M “recycled" pairs are generated by flipping previously discarded pairs’ label polarities (Liu et al., 2 Jul 2025).

3. Training and Optimization Regimen

The model is trained with the following core hyperparameters:

• Context length: 16,384 tokens
• Optimizer: AdamW ($\beta = (0.9, 0.999)$), weight decay $\lambda \approx 10^{-2}$
• Batch size: 10,240 (distributed via Tensor-parallel DeepSpeed ZeRO 1, 64×H800 GPUs)
• Learning rate: fixed at $3 \times 10^{-6}$
• Training steps: ~2,600 (1 epoch over 26M pairs)
• No explicit curriculum or staged mixing, except as induced by the two-stage curation (Liu et al., 2 Jul 2025).

4. Empirical Performance and Benchmarking

Skywork-Reward-V2-Llama-3.1-8B establishes a new state-of-the-art among open-weight reward models across seven key benchmarks, including RewardBench v1/v2, PPE Preference/Correctness, RMB, RM-Bench, and JudgeBench. Key metrics include:

Model	RB1	RB2	PPE Pref.	PPE Corr.	RMB	RM-Bench	JudgeBench	Avg.
INF-ORM-Llama3.1-70B	95.1	76.5	64.2	64.4	70.5	73.8	70.2	73.5
Skywork-V2-Llama3.1-8B	96.4	84.1	77.3	83.4	86.4	92.8	80.0	85.7
Skywork-V2-Llama3.1-8B-40M	97.8	86.5	79.8	87.2	89.3	96.0	83.4	88.6

• On RewardBench v1/v2: 96.4%/84.1% (vs. 95.1%/76.5% for INF-ORM-Llama3.1-70B)
• PPE Preference/Correctness: 77.3% / 83.4% (new open RM record)
• Best-of-N (RMB): 86.4% average
• RM-Bench: 97.0% (easy), 95.0% (normal), 86.5% (hard); consistently high across difficulty levels

Ablation studies confirm that gains derive primarily from curated data quality—further increasing data scale through unfiltered or recycled pairs yields only marginal returns (<0.5 percentage points), whereas human-verified curation can provide +2–3 points or more (Liu et al., 2 Jul 2025).

5. Comparative Data-Centric Studies and Insights

A minimalistic approach using only 80K curated preference pairs—Skywork-Reward-Llama-3.1-8B—demonstrates that high RewardBench scores can be achieved with substantially smaller datasets, provided strong decontamination and targeted filtering. This variant, trained identically but with robust data selection, achieves an average RewardBench score of 92.5%. Decontaminating test prompt overlaps yields gains in safety and reasoning by +2.1 and +0.5 percentage points, respectively (Liu et al., 2024). Performance on adversarial and logic-intensive categories (e.g., “Chat Hard”: 87.3%, “Reasoning”: 96.2%) is competitive with much larger models trained on greater data quantities.

Ablations indicate scoring bias corrections, task-specific sample balancing, and careful decontamination have an outsized impact relative to raw dataset volume. The Bradley–Terry pairwise loss remains optimal in this setting. These results reinforce the finding that reward model generalization is bottlenecked primarily by input data signal and annotation rigor.

6. Vulnerabilities to Token-Space Attacks

Despite strong semantic alignment, Skywork-Reward-V2-Llama-3.1-8B exhibits severe vulnerabilities to adversarial optimization operating directly in token space. The Token Mapping Perturbation Attack (TOMPA) framework bypasses natural language generation and instead maximizes RM scores over raw token sequences. Under this black-box policy optimization, TOMPA nearly doubles the average reward for Llama RM compared to GPT-5 reference answers (from 17.48±0.53 to 33.64±0.85, beating GPT-5 on 98% of held-out prompts). The adversarial outputs are nonsensical, with high reward only at maximum sequence length (2048 tokens), revealing representational blind spots and pathological length biases in the RM (Zhang et al., 3 Apr 2026).

Representative TOMPA outputs include repeated special tokens and gibberish:

“<unk> ¤ÍÛäß Ô»≤¶≤ⁿ …𝚌𝚘𝚕𝚕𝚎𝚌𝚝_𝚌𝚑𝚊𝚛𝚜();‹‹‹…”, “«╬╅╬╏╬╅╬╏╍╏╍╬╅╬⌛⌛⌛%%%ˡˡ…” (reward > +34.9).

This exposes a critical misalignment between token-level representation and semantic correctness, as well as a lack of robust out-of-distribution detection.

7. Limitations, Failure Modes, and Future Directions

The model does not fully capture the diversity of human preference, tending to prefer majority-style responses; rare or controversial preferences are underrepresented. Fully automated LLM-only curation is brittle on subjective cases. Edge cases with near-tied preference, or nuanced safety distinctions (hard refusal vs. partial compliance), occasionally remain misranked (Liu et al., 2 Jul 2025).

Token-space adversarial vulnerability exposes the need for next-generation reward models to incorporate adversarial training against synthetic or token-perturbed sequences, enforce semantic or syntactic constraints, and diversify RM ensembles. Length normalization penalties and formal robustness guarantees are proposed avenues for defense (Zhang et al., 3 Apr 2026).

This suggests that, beyond further dataset scaling or incremental model capacity increases, primary research attention should shift to comprehensive data curation protocols, adversarial robustness in input representation, and systematic auditing for benchmark contamination and out-of-distribution detection.