Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values

Abstract: We propose Reinforcement Learning with Explicit Human Values (RLEV), a method that aligns LLM optimization directly with quantifiable human value signals. While Reinforcement Learning with Verifiable Rewards (RLVR) effectively trains models in objective domains using binary correctness rewards, it overlooks that not all tasks are equally significant. RLEV extends this framework by incorporating human-defined value signals directly into the reward function. Using exam-style data with explicit ground-truth value labels, RLEV consistently outperforms correctness-only baselines across multiple RL algorithms and model scales. Crucially, RLEV policies not only improve value-weighted accuracy but also learn a value-sensitive termination policy: concise for low-value prompts, thorough for high-value ones. We demonstrate this behavior stems from value-weighted gradient amplification on end-of-sequence tokens. Ablation studies confirm the gain is causally linked to value alignment. RLEV remains robust under noisy value signals, such as difficulty-based labels, demonstrating that optimizing for an explicit utility function offers a practical path to aligning LLMs with human priorities.

• The paper presents RLEV, a reinforcement learning framework that integrates human-assigned value signals into the reward function to enhance accuracy and efficiency.
• It employs an additive, clipped reward design and value-sensitive gradient amplification to balance correctness with prompt importance.
• Empirical results show consistent improvements, with 2.0–2.8% gains in value-weighted accuracy and robust generalization across various benchmarks.

Reinforcement Learning with Explicit Human Values (RLEV): A Formal Analysis

Introduction

The paper "Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values" (2510.20187) introduces RLEV, a reinforcement learning paradigm for LLMs that directly incorporates human-assigned value signals into the reward function. This approach addresses a critical limitation of correctness-only RL post-training, which treats all prompts as equally important, by enabling models to optimize for the true utility function defined by human priorities. RLEV is evaluated in exam-style domains with explicit ground-truth value labels, demonstrating consistent improvements in value-weighted accuracy, response efficiency, and strategic termination behavior across multiple RL algorithms and model scales.

Figure 1: RLEV overview. The verifier can be either a reward model or rule-based function.

Human Utility Function and Reward Design

RLEV formalizes the utility of a response as the product of its correctness and the intrinsic value of the prompt: $$U(x, y) = v(x) \cdot \mathbf{1}_{\text{correct}(y)}$$, where $v(x)$ is a normalized human-assigned value and $\mathbf{1}_{\text{correct}(y)}$ is an indicator of correctness. To ensure stable training, the reward function is designed as $$r(x, y) = s(x) \cdot \mathbf{1}_{\text{correct}(y)}$$, with $s(x) = 1 + \min(\alpha \cdot v(x), 1)$, guaranteeing a minimum reward for any correct response and amplifying the reward for high-value prompts.

This additive and clipped scaling is empirically validated to prevent instability from excessively large rewards and to maintain a fine-grained correspondence with human value distributions, which are highly right-skewed in exam datasets.

Policy Optimization and Gradient Analysis

The RL objective is to maximize expected cumulative reward over a dataset, with sparse rewards only at the final step. The policy gradient for a token $k$ at time $t$ is:

$$\frac{\partial J}{\partial z_k} = \pi(k \mid x, y_{<t}) s(x) \cdot \left( p_k - \sum_{v \in \mathcal V} \pi(v \mid x, y_{<t}) p_v \right)$$

where $p_k$ is the probability of final correctness given $y_t = k$. For the EOS token, the gradient is amplified by the value scaling factor $v(x)$0, reinforcing early termination when correctness is likely and the prompt is low-value, and encouraging thoroughness for high-value prompts.

Empirical Results

RLEV is evaluated on a large-scale, multi-subject exam dataset (100k training, 8k test), with models initialized from Qwen2.5-7B and Qwen2.5-32B. Across REINFORCE++, RLOO, and GRPO estimators, RLEV consistently outperforms correctness-only baselines in both standard and value-weighted accuracy, with average H-Acc gains of 2.0–2.8% and substantial reductions in response length (e.g., halving for 32B models).

Figure 2: Distribution of human-defined normalized values $v(x)$1 in training data (100k) with ground-truth values.

Figure 3: Distribution of human-defined normalized values $v(x)$2 in test data (8k) with ground-truth values.

RLEV-trained policies generalize to out-of-distribution English and Chinese benchmarks, outperforming baselines on GPQA Diamond, C-Eval, MMLU-Pro, and SuperGPQA. The method is robust to noisy value signals, such as difficulty-based weak labels and predictor-generated scores, maintaining superior performance even when ground-truth values are unavailable.

Value-Sensitive Termination Policy

A distinctive property of RLEV is its learned value-sensitive termination policy. Token-level analysis reveals that for low-value prompts, the model assigns higher EOS probabilities earlier, producing concise answers. For high-value prompts, EOS probability is suppressed, resulting in more thorough responses. This strategic allocation of token budget is directly attributable to value-weighted gradient amplification.

Figure 4: EOS probability trajectories for RLEV and the baseline, showing different termination policies for high-value (top) and low-value (bottom) prompts.

Ablation and Sensitivity Studies

Ablation studies isolate the impact of human value alignment. Uniform reward scaling and random value shuffling do not yield the same improvements as explicit value alignment, confirming that the gains are causally linked to the correlation between reward magnitude and prompt importance. Sensitivity analysis shows that the choice of scaling hyperparameter $v(x)$3 and the additive/clipped reward form are critical for balancing accuracy and conciseness.

Practical and Theoretical Implications

RLEV provides a principled framework for aligning LLMs with explicit human utility functions in domains where correctness and importance are jointly verifiable. The approach is practical, requiring only per-prompt value labels, and robust to noisy or heuristic value estimates. Theoretically, RLEV demonstrates that direct optimization for non-uniform utility functions yields policies that are both more efficient and more aligned with human priorities, as opposed to implicit value learning or uniform correctness maximization.

The method's reliance on explicit value signals makes it particularly suitable for objective domains such as exams, medical triage, and content moderation. However, its applicability to subjective or multi-dimensional value functions remains an open question.

Future Directions

Potential extensions include dynamic or learned value functions that adapt to user priorities, and hybrid approaches combining RLEV with RLHF to jointly optimize for objective correctness and subjective qualities. Further research may explore multi-dimensional value alignment and real-time adaptation in interactive systems.

Conclusion

RLEV advances the alignment of LLMs by directly optimizing for explicit, human-defined utility functions. The paradigm yields consistent improvements in value-weighted accuracy, response efficiency, and strategic behavior, with gains causally linked to value alignment rather than reward magnitude. RLEV is robust to noisy value signals and generalizes across domains, representing a practical step toward LLMs that judiciously allocate their capabilities according to human priorities. Future work should address the challenges of multi-dimensional and dynamic value functions to further enhance the alignment of AI systems with complex human objectives.