WildReward: In-the-Wild Reward Modeling

• WildReward is a reward model that leverages natural, in-the-wild human interactions instead of traditional preference pairs to train large language models.
• It constructs a four-class pointwise feedback dataset (WildFB) from filtered WildChat logs using methods like feedback mining and refusal validation to ensure high-quality supervision.
• Empirical results indicate that WildReward achieves competitive performance with improved calibration and cross-sample consistency, enhancing online Direct Preference Optimization.

Searching arXiv for the WildReward paper and closely related context. {"query":"arXiv (Peng et al., 9 Feb 2026) WildReward Learning Reward Models from In-the-Wild Human Interactions", "max_results": 5} Searching arXiv for the exact WildReward identifier. to=arxiv_search 天天中彩票谁 天天中json code : {"query":"(Peng et al., 9 Feb 2026)", "max_results": 10} to=arxiv_search 天天种彩票json code : {"query":"(Peng et al., 9 Feb 2026)", "max_results": 10} WildReward is a reward model for LLMs trained directly from in-the-wild human interactions rather than from conventional human-annotated preference pairs. It uses WildChat as an interaction source, extracts reliable feedback from naturally occurring user follow-up messages, and trains an ordinal regression model over ordered satisfaction labels, yielding a four-class pointwise dataset called WildFB with about 186k instances (Peng et al., 9 Feb 2026). In this formulation, reward modeling is grounded in authentic deployment-time behavior: users correct errors, express dissatisfaction, continue a topic because they were helped, or explicitly signal satisfaction. The central claim is that such interaction logs constitute a scalable supervision source, provided that sparsity and noise are handled carefully, and that the resulting reward model can achieve comparable or even superior performance relative to preference-trained reward models while exhibiting improved calibration and cross-sample consistency (Peng et al., 9 Feb 2026).

1. Conceptual setting and motivation

WildReward addresses a structural limitation of standard reward-model pipelines for LLM alignment. Conventional reward models depend on large preference-pair datasets, which are expensive to annotate and difficult to scale. By contrast, deployment logs already contain large volumes of implicit human feedback. WildReward asks whether reward models can be learned directly from that raw interaction stream, without converting supervision into pairwise comparisons first (Peng et al., 9 Feb 2026).

The paper motivates this shift on three grounds. First, real interactions are available at scale through widespread deployment. Second, they capture authentic user preferences rather than preferences elicited under laboratory-style annotation protocols. Third, they potentially support continual improvement: if reward models can be trained from live interactions, then increasing usage can directly enrich the reward signal. This suggests a reward-modeling regime in which data collection is coupled to actual system use rather than to separate annotation campaigns.

The difficulty is that in-the-wild feedback is both sparse and noisy. In a preliminary analysis of 10,000 WildChat samples, about 82% of follow-up queries were neutral or ambiguous, around 17% contained negative feedback, and only about 1% were explicitly positive (Peng et al., 9 Feb 2026). The paper also emphasizes a specific safety-related noise source: some negative user reactions are unjustified, especially when the model correctly refuses harmful requests but the user still complains. WildReward is therefore not simply a change of data source; it is a pipeline for turning low-signal, partially unreliable interaction traces into reward supervision.

2. WildChat-derived supervision and the WildFB dataset

WildReward uses WildChat-4.8M as its interaction source and applies a filtering-and-mining pipeline to construct WildFB, a four-class pointwise feedback dataset of about 186k instances (Peng et al., 9 Feb 2026). Each instance contains conversation history $c$, user query $q$, assistant response $s$, and feedback label $y$.

The initial filtering stage retains only English and Chinese conversations and removes several classes of unsuitable data. These include multimodal queries, tool-dependent queries, trivial identity queries such as “Who are you?”, context-dependent queries requiring external documents or context, very long conversations with more than 20 turns, and very short queries with fewer than 5 words or responses with fewer than 10 words. These filters are implemented with manually written rules and regexes.

Feedback extraction is then performed with gpt-oss-120b, which classifies the user’s follow-up message. The paper first uses a 3-way analysis for exploratory statistics—Negative, Neutral, Positive—and then adopts a 5-way scheme for dataset construction: Explicit Rejection, Error Correction, Neutral Ambiguity, Positive Engagement, and Explicit Satisfaction (Peng et al., 9 Feb 2026). The procedure is deliberately conservative: if the evidence is weak, the default label is Neutral Ambiguity.

Two refinement steps are used to improve label quality. The first is implicit feedback mining. The key idea is that if a user gives positive feedback in nearby turns within a coherent topic, then intermediate turns labeled Neutral Ambiguity may also carry positive feedback. These cases are identified by semantic similarity using cosine similarity of all-MiniLM-L6-v2 sentence embeddings with threshold

$\text{cosine similarity} > 0.6.$

This reclassification yields about 12,310 additional samples and increases the positive subset by about 29% (Peng et al., 9 Feb 2026).

The second refinement step is refusal validation. Another gpt-oss-120b-based classifier determines whether a negative reaction is actually a false-refusal case by distinguishing whether the model refused or answered normally and, if it refused, whether the query was harmful or safe. This identifies about 572 mislabeled refusal cases, which are corrected. Although small in count, the paper reports that these corrections have a very large effect on safety-related evaluation (Peng et al., 9 Feb 2026).

After removing Neutral Ambiguity and applying both refinements, the final WildFB dataset contains four ordered feedback classes. This dataset design is important because WildReward is trained pointwise on absolute feedback labels rather than pairwise on chosen-versus-rejected responses.

3. Ordinal regression formulation

WildReward is trained as an ordinal regression model rather than with Bradley-Terry loss on preference pairs. The ordered labels are:

1. Explicit Rejection
2. Error Correction
3. Positive Engagement
4. Explicit Satisfaction

For each example, the model input is

$x = (c, q, s),$

with ordinal label

$y \in \{1,2,3,4\}.$

The loss is

$$\mathcal{L} = - \sum_{k=1}^{K-1} \big[ \mathbb{I}(y>k) \log P(y>k \mid x; \theta) + (1-\mathbb{I}(y>k)) \log(1-P(y>k \mid x; \theta)) \big],$$

where $K=4$ (Peng et al., 9 Feb 2026). Operationally, the model predicts, for each threshold $k$, whether the true label is above $q$0. The paper characterizes this as a standard cumulative-link-style ordinal objective, emphasizing that it preserves order structure directly rather than treating classes as equally spaced scalar targets.

At inference time, the scalar reward is defined as

$q$1

This score corresponds to the expected ordinal value under the learned cumulative probabilities (Peng et al., 9 Feb 2026). The paper further notes that the probabilistic output can serve as a confidence measure for filtering uncertain predictions.

This formulation differs materially from pairwise reward modeling. A pairwise model learns local preferences between two candidate responses conditioned on a prompt. WildReward instead learns globally ordered pointwise scores from user reactions to single responses. The paper later links this choice to improved cross-sample consistency: absolute feedback labels appear to induce more globally comparable reward values than Bradley-Terry training (Peng et al., 9 Feb 2026). This suggests that pointwise ordinal supervision may be particularly suitable when reward scores are meant to be reused outside static reranking, such as in online optimization.

4. Model instantiation and empirical evaluation

The reported WildReward models are built on Qwen3 4B and Qwen3 8B, trained for one epoch on WildFB with batch size 512, learning rate $q$2, and maximum length 4096 (Peng et al., 9 Feb 2026). The resulting models are WildReward-4B and WildReward-8B.

Evaluation covers RewardBench, RM-Bench in easy, normal, and hard settings, PPE with Human and Correctness subsets, and JudgeBench. These benchmarks span creative writing, instruction following, math, commonsense reasoning, coding, and safety. For WildReward-8B, the reported scores are 86.0 on RewardBench, 83.5 on RM-Bench easy, 78.4 on RM-Bench normal, 69.7 on RM-Bench hard, 62.5 on PPE Human, 65.6 on PPE Correctness, and 66.0 on JudgeBench. For WildReward-4B, the reported scores are 83.6, 82.0, 77.0, 68.6, 61.6, 63.6, and 61.1 respectively (Peng et al., 9 Feb 2026).

The baselines include ArmoRM-Llama3-8B-v0.1, Athene-RM-8B, Llama-3-OffsetBias-RM-8B, Skywork-Reward-Llama-3.1-8B-v0.2, Internlm2-20b-reward, Skywork-Reward-Gemma-2-27B-v0.2, Llama-3.1-Nemotron-70B, and INF-ORM-Llama3.1-70B. The paper’s interpretation is that WildReward is competitive with or better than conventional preference-trained reward models despite not using preference pairs, and that the 8B model surpasses much larger 70B reward models on some settings, especially RM-Bench Hard and PPE Correctness (Peng et al., 9 Feb 2026).

Ablations isolate the effects of feedback mining and refusal validation. On a five-way breakdown reported as Chat, Math, Code, SRF, and SRP, WildReward-4B obtains 79.3, 75.6, 65.8, 90.4, and 72.0. Removing feedback mining lowers several values, while removing refusal validation causes SRF to collapse from 90.4 to 28.5 (Peng et al., 9 Feb 2026). The paper highlights this as evidence that even a small amount of refusal-label noise can heavily distort safety-sensitive reward modeling.

5. Calibration, consistency, and the role of user diversity

A central empirical claim of WildReward is that ordinal regression on in-the-wild feedback yields not only strong ranking performance but also better-calibrated and more globally consistent reward scores. Calibration is evaluated on RM-Bench Normal using the margin between chosen and rejected response scores as confidence, followed by Platt scaling:

$q$3

Measured by Expected Calibration Error, WildReward obtains 2.76%, compared with 8.81% for ArmoRM-Llama3-8B-v0.1 (Peng et al., 9 Feb 2026). The paper also reports that increasing the score-difference threshold improves accuracy; at threshold 0.2, accuracy reaches about 87% while retaining about 50% of predictions.

The paper further argues that pairwise benchmarks are insufficient because they test local comparisons but not whether reward scores are globally comparable across prompts. To probe this, it constructs a pointwise evaluation set from held-out WildChat data: 5,000 instances are initially sampled, then filtered and manually verified, producing a final set of 948 instances with binary positive-versus-negative labels (Peng et al., 9 Feb 2026). On ROC-AUC over this global distribution, WildReward significantly outperforms pairwise-trained baselines. The conclusion drawn in the paper is that ordinal regression on absolute feedback yields stronger cross-sample consistency than Bradley-Terry models.

User diversity is analyzed explicitly. The paper creates two dataset series of equal size, where one series has 10 times more unique users than the other, and evaluates on RM-Bench Normal. Performance improves with more data, but for fixed data size, the datasets with more unique users consistently produce better reward models (Peng et al., 9 Feb 2026). The paper interprets this as direct evidence that diversity, not only scale, matters: more users provide less idiosyncratic and more robust supervision.

6. Use in online DPO, limitations, and implications

WildReward is applied as a scoring model for Direct Preference Optimization. In offline DPO, four responses are sampled per prompt, scored by the reward model, and the highest- and lowest-scoring responses are paired. In online DPO, eight responses per prompt are generated on the fly with the current policy, scored by the reward model, and converted into best-versus-worst preference pairs (Peng et al., 9 Feb 2026). The policy model is Llama-3.1-8B-Instruct, and the prompt set is drawn from Infinity Instruct with 20,000 prompts.

The reported average scores are 58.1 for the original Llama-3.1-8B-Instruct, 58.6 for offline DPO with WildReward, 60.4 for online DPO with ArmoRM, and 62.6 for online DPO with WildReward (Peng et al., 9 Feb 2026). For WildReward online DPO, the detailed metrics are GSM8K 87.9, MATH-500 51.6, MMLU Pro 48.9, IFEval 82.1, Alpaca Eval 2.0 41.3, and Arena Hard 63.5. The paper notes that offline DPO barely helps, whereas online DPO improves substantially, suggesting that a live reward signal is materially more useful than a fixed offline preference construction.

The limitations are explicit. WildFB includes only English and Chinese, which may constrain multilingual generalization. The study does not exhaustively explore alternative backbones or hyperparameters. It does not apply WildReward to full RL training, only to online DPO, because RL is computationally expensive and harder to stabilize (Peng et al., 9 Feb 2026). The paper also stresses that reward models trained from real-world interactions may inherit biases present in those interactions and may still be vulnerable to reward hacking, motivating verification before deployment.

Taken together, WildReward represents a shift from annotation-centric reward modeling to interaction-centric reward modeling. Its defining technical ingredients are a high-precision pipeline for converting WildChat interactions into a four-class feedback dataset and an ordinal regression objective that produces calibrated, globally meaningful scores without preference pairs (Peng et al., 9 Feb 2026). A plausible implication is that future reward-model pipelines may increasingly treat deployment logs not merely as evaluation traces but as the primary substrate for reward learning, especially when online optimization and user-diverse supervision are central design goals.