Skywork-Reward-V2-Qwen3-8B Model

• Skywork-Reward-V2-Qwen3-8B is an 8B open-weight reward model designed to evaluate LLM outputs using preference-based scoring.
• It employs a Qwen3-8B decoder-only transformer backbone and is trained on 26 million curated preference pairs from SynPref-40M using the Bradley–Terry objective.
• Empirical benchmarks show strong best-of-N reranking and bias detection performance, while highlighting vulnerabilities to token-space adversarial attacks.

Skywork-Reward-V2-Qwen3-8B is an 8 billion parameter open-weight reward model within the Skywork-Reward-V2 suite, specifically designed to serve as a preference scorer for LLM post-training pipelines, including RLHF and best-of-N candidate reranking. Trained on a subset of the SynPref-40M preference corpus using a human-AI synergistic data curation protocol, this model reflects both architectural advancements and systematic attention to data quality. It features a backbone based on Qwen3-8B and is benchmarked extensively against open and proprietary reward models alike, with detailed empirical, ablation, and robustness analyses.

1. Architectural Foundations and Model Training

Skywork-Reward-V2-Qwen3-8B employs the Qwen3-8B transformer backbone, incorporating a byte-level BPE tokenizer with a vocabulary size around 80k. The architecture consists of a decoder-only transformer with the reward model head—consisting of a single scalar output—applied atop the final hidden state corresponding to the end-of-sequence token. Training is conducted on 26 million curated preference pairs from SynPref-40M using the Bradley–Terry preference objective:

$$L(\theta) = - \mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma(r_\theta(x, y_w) - r_\theta(x, y_l)) \right],$$

where $r_\theta$ is the scalar reward output, and $(x, y_w, y_l)$ denotes prompt, preferred, and rejected responses, respectively. Learning rates (typically $1 \times 10^{-6}$ for the 8B variant), batch sizes, and global training configurations mirror those of the Llama-3.1-8B sibling, with large-batch optimization and minimal regularization (weight decay $\sim$0.1) (Liu et al., 2 Jul 2025).

2. Data Curation Methodology: SynPref-40M and Human-AI Synergy

The SynPref-40M corpus aggregates over 40 million preference pairs from numerous sources. Data curation employs a two-stage process: (1) human-in-the-loop annotation and verification, anchored by gold-labeled seed data with multi-attribute labeling (task category, objectivity, controversiality, desired traits, and annotation guideline); (2) automatic scaling and consistency checks leveraging best reward models as filters. Annotators use LLM-based judges (GPT-4o, Llama-3.1-70B, DeepSeek, Qwen3-32B, etc.) for high-capacity “silver” labeling. Disagreement-driven re-annotation and ensemble validation produce a mixture of approximately 4M gold and 12M silver high-quality pairs (Liu et al., 2 Jul 2025).

In Stage 2, unverified pairs are culled by consensus from two independently trained reward models; only pairs with consistent, if low-confidence, agreement are admitted. This pipeline, uniquely scalable yet rigorous, results in far higher empirical effectiveness than purely synthetic or weakly curated approaches.

3. Empirical Benchmarking and Comparative Performance

Skywork-Reward-V2-Qwen3-8B is evaluated across seven leading reward model benchmarks, reflecting dimensions such as human preference alignment, objective correctness, safety, stylistic bias resistance, and judge agreement.

Model	RB1	RB2	PPE⁺	PPE✓	RMB	RM-B	Judge	Avg
Skywork-Reward-V2-Qwen3-8B	93.7	78.2	70.6	75.1	81.2	82.6	73.4	79.3
Skywork-Reward-V2-Llama-3.1-8B	96.4	84.1	77.3	83.4	86.4	92.8	80.0	85.7
INF-ORM-Llama3.1-70B	95.1	76.5	64.2	64.4	70.5	73.8	70.2	73.5

Performance (average 79.3%) places the Qwen3-8B backbone behind the Llama-3.1-8B variant, highlighting the impact of underlying model capacity and pretraining, yet confirming substantial gains over previous open RMs. Accuracy in best-of-N reranking rises monotonically, and gains persist up to $N=32$ candidates, eliminating the performance plateau observed in earlier models (Liu et al., 2 Jul 2025).

4. Automated Bias Discovery and Failure Modes

Automated interpretability pipelines have identified reward model biases in Skywork-Reward-V2 models, including Qwen3-8B. An LLM-in-the-loop evolutionary schema iteratively proposes, evaluates, and refines candidate biases $A$ by measuring

$$R(A) = \mathbb{E}_x \mathbb{E}_i [R(x, y_1^i) - R(x, y_0^i)], \quad J(A) = \mathbb{E}_x \mathbb{E}_i J(x, y_1^i, y_0^i)$$

with $y_1$ and $y_0$ being counterfactuals toggling attribute $r_\theta$0. A bias is defined as $r_\theta$1 and $r_\theta$2. Evolutionary search, with Pareto selection and mutation, outperforms flat hypothesis search, measured by Diversity–Adjusted Bias Strength (DABS).

Uncovered biases in Skywork-Reward-V2-8B include:

• Redundant-spacing bias (e.g., triple spaces)
• Hallucination bias (unsupported quotations)
• False-confidence statements (continuous learning claims)
• Sycophancy and unwarranted affirmations
• Checklist inclusion for reporting content (unethical behavior)
• Recurrent formatting tropes ("Hope this helps!") (Wang et al., 16 Feb 2026)

Extension to Qwen3-8B is straightforward given black-box access to $r_\theta$3 and suitable editing LLMs, although Qwen3’s tokenizer may induce different low-level artifacts (e.g., stray tabs, non-breaking spaces) and domain-specific deviations due to training data drift.

5. Token-Space Vulnerability: Adversarial Attacks

Skywork-Reward-V2-Qwen3-8B is subject to token-space attacks such as the Token Mapping Perturbation Attack (TOMPA), which optimizes input sequences directly in token space, detaching reward maximization from natural-language plausibility. Given an attack policy $r_\theta$4 and mapping $r_\theta$5, TOMPA maximizes

$r_\theta$6

by optimizing with Group Relative Policy Optimization (GRPO). Empirical results show:

• TOMPA nearly doubles reward scores over GPT-5 baselines for Qwen3-8B RM inputs (mean reward +16.86 for TOMPA vs. +8.12 for GPT-5) and beats gold answers on 98% of prompts.
• Attacks yield degenerate outputs—long, synthetic token sequences with cross-lingual fragments or special symbols—that are non-linguistic in nature but exploit both token and length biases.
• Adversarial reward surges occur near maximal sequence length (e.g., at 2048 tokens).

Architecturally, Qwen3-8B's larger BPE vocabulary and unique embedding distribution make it susceptible to high-dimensional token-pattern exploitation, although increasing $r_\theta$7 and batch size $r_\theta$8 may be required for optimal attack efficiency (Zhang et al., 3 Apr 2026).

6. Ablations, Robustness, and Open Issues

Ablation studies demonstrate that preference curation quality is more impactful than raw dataset scale: models trained on a modest, but carefully curated subset of $r_\theta$9290k samples can outperform previous large-scale RMs. Human-guided annotation with explicit preference attributes yields the highest score improvements in empirical benchmarks. For Skywork-Reward-V2-Qwen3-8B, performance is robust under best-of-N sampling, but the absolute boundaries remain conditioned by the Qwen3-8B backbone’s representational ceiling (Liu et al., 2 Jul 2025).

Bias-discovery pipelines validated with synthetically injected attributes yield high recall ($(x, y_w, y_l)$080%), especially when response presentation is balanced. Evolutionary iterations are crucial for surfacing rare or subtle reward artifacts (Wang et al., 16 Feb 2026).

7. Outlook and Research Implications

Skywork-Reward-V2-Qwen3-8B synthesizes advances in scalable preference curation, transformer reward modeling, and pipeline evaluation, yet open issues remain:

• Can automated, next-generation LLM agents supplant gold-quality human labels?
• How can token-space adversarial vulnerability be systematically mitigated (proposed approaches include adversarial data augmentation, robust length regularization, decodability checks, and ensemble filtering)?
• What mechanisms allow adaptation of reward modeling to personalized or context-dependent objectives at scale?
• Are there alternative loss functions (e.g., order-consistent or uncertainty-aware objectives) that could enhance robustness under RLHF or best-of-N mechanisms?

The evidence indicates that robust preference-based reward modeling for LLMs requires joint advances in data curation, architectural regularity, and adversarial resilience. Skywork-Reward-V2-Qwen3-8B serves as both a benchmark and a case study for future research in scalable, interpretability-driven, and adversarially robust reward models (Liu et al., 2 Jul 2025, Wang et al., 16 Feb 2026, Zhang et al., 3 Apr 2026).