Capability-Adaptive Reward Mechanism

• Capability-adaptive reward mechanisms are dynamic strategies that adjust training signals using model capability, task difficulty, and compute budget.
• They incorporate methods like explicit conditioning, cohort-relative gating, and adaptive curriculum to maintain reward fidelity and discriminability.
• Empirical studies reveal significant gains in accuracy and efficiency while highlighting challenges such as reward hacking and calibration in evolving regimes.

Searching arXiv for papers on capability-adaptive reward mechanisms and closely related adaptive reward designs. Capability-adaptive reward mechanism denotes a class of reward-design and reward-mediated training strategies in which the effective learning signal varies with model capability, task difficulty, rollout regime, or compute budget rather than remaining a fixed scalar function throughout training and inference. Recent work instantiates this idea through explicit conditioning of reward models on capability and reinforcement-learning iteration, cohort-relative reward gating, dynamic hint scaffolding, capability-aware rollout allocation, adaptive milestone shaping, act-adaptive margins, and instruction-specific constraint verification (Liu et al., 12 Jun 2026, Xie et al., 5 Jun 2026, Li et al., 8 Sep 2025, Yao et al., 3 Feb 2026, Zheng et al., 12 Feb 2026, Fang et al., 29 May 2025, Chen et al., 5 Sep 2025). Across these formulations, the central objective is consistent: preserve reward fidelity while maintaining discriminability and useful gradients as the learner, the generator, or the environment changes.

1. Conceptual scope

A capability-adaptive reward mechanism is motivated by the observation that static rewards degrade when the policy or generator moves away from the distribution on which the reward was originally calibrated. In text-to-image alignment, HPSv3++ identifies both capability shift across generators and distribution shift along RL iterations: a reward model trained on older diffusion outputs can over-reward images that are merely good for earlier systems, under-reward frontier-quality images, and suffer score-standard-deviation collapse as RL narrows the rollout distribution (Liu et al., 12 Jun 2026). In deep-research web agents, SlimSearcher makes the same point from a computational-efficiency perspective: a fixed reward that values only correctness or uses absolute penalties fails to distinguish minimal necessary trajectories from brute-force search, so efficiency must be assessed relative to what the current cohort can achieve (Xie et al., 5 Jun 2026).

The same logic appears in reasoning-oriented RL. SEELE shows that RLVR becomes ineffective when problem difficulty is misaligned with model capability, because rollout accuracy near 0 or 1 yields weak advantages; it therefore adapts hint length to keep training in a high-efficiency region around $a \approx 0.5$ (Li et al., 8 Sep 2025). CoBA-RL argues that uniform rollout budgets ignore the model’s evolving capability and reallocates samples to prompts with higher capability-oriented training value instead of treating all tasks as equally informative (Yao et al., 3 Feb 2026). In GUI agents, ADMIRE addresses long-horizon credit assignment by extracting milestones from successful trajectories and then using them asymmetrically for successful and failed episodes, so that partial progress becomes reward-bearing without discarding verifiability (Zheng et al., 12 Feb 2026).

A recurrent distinction in this literature is that capability adaptivity need not mean modifying the scalar reward formula alone. Some methods change the reward function directly; others alter the conditioning variables supplied to the reward, the data on which reward is evaluated, or the amount of computation spent before reward is sampled. This broader usage is explicit in SEELE, CoBA-RL, and Re-FORC, where the underlying verifiable or terminal reward may remain unchanged while difficulty control, rollout allocation, or reward forecasting changes the effective training or inference signal (Li et al., 8 Sep 2025, Yao et al., 3 Feb 2026, Zabounidis et al., 3 Nov 2025).

2. Canonical design patterns

Recent work can be organized by where adaptivity enters the reward pipeline: in the reward function itself, in reward aggregation, in reward exposure through data or budget allocation, or in inference-time control. Representative instances are summarized below.

Mechanism	Adaptive signal	Reward construction
HPSv3++	capability $c$, RL iteration $t$	conditioned reward $r(x,y\mid c,t)$
SlimSearcher	cohort-relative tool and token minima	$$R_{\text{final}} = r_{\text{correct}} r_{\text{tool}} r_{\text{len}}$$
TAROT	model capability via curriculum selection	$R_{\text{TAROT}} = \sum_l \alpha_l w_l r_{i,l}$
ACE-RL	instruction-specific checklist and target length	$R = \tfrac12(R_L + R_C)$
LASER-D	difficulty bucket and dynamic target length	length bonus gated by correctness
ADMIRE	milestone completion and trajectory outcome	asymmetric milestone reward

The most explicit conditional reward among these is HPSv3++, which extends an unconditional text-to-image reward into

$$r_{\Theta}(x,y \mid c,t)=f_{\phi}(E_{\theta}(x,y),c,t),$$

where $c$ is an inferred capability condition and $t$ is normalized RL iteration. Capability and iteration are injected through FiLM layers in the reward head, so the same image-text pair can be scored differently depending on the generator regime and training stage (Liu et al., 12 Jun 2026). SlimSearcher instead keeps reward scalar but makes it cohort-adaptive: $c$0 where tool and token multipliers are defined relative to the best correct trajectory in the current cohort, preventing efficient-but-wrong trajectories from receiving reward (Xie et al., 5 Jun 2026).

TAROT formalizes adaptivity as curriculum-conditioned reward aggregation over test tiers: $c$1 with $c$2 for basic, intermediate, complex, and edge tests. Here $c$3 controls curriculum allocation and $c$4 controls the contribution of each tier to the scalar reward, allowing different reward landscapes for weaker and stronger code models (Park et al., 17 Feb 2026). ACE-RL uses a different decomposition: $c$5 where $c$6 scores length against an instruction-specific target and $c$7 averages verifier judgments over an automatically generated constraint checklist tailored to the instruction (Chen et al., 5 Sep 2025). LASER and LASER-D further show that even simple step-shaped bonuses on output length can act as adaptive reward shaping when the target length is allowed to vary with difficulty and training stage (Liu et al., 21 May 2025).

Another important pattern is adaptive margin design. ChARM keeps the pairwise reward-modeling setting but replaces a global Bradley–Terry margin with an act-adaptive margin

$c$8

so that reward learning pressure depends on how the model’s own generative distribution distinguishes chosen and rejected role-playing acts (Fang et al., 29 May 2025). This shifts adaptivity from explicit environment capability to behavior-specific modeling confidence.

3. Signals used to estimate capability

Capability-adaptive mechanisms differ most sharply in how they estimate the learner’s current regime. In HPSv3++, capability is inferred from image-group features by a trainable Capability Encoder, while RL iteration is passed as an explicit scalar $c$9; the paper additionally uses intra-group score standard deviation as an unsupervised indicator of discriminability and directly optimizes standard-deviation-based objectives to prevent collapse on frontier or post-RL distributions (Liu et al., 12 Jun 2026). AdaGRPO, also for text-to-image RL, uses deterministic ODE rewards on candidate prompts to maintain an EMA capability anchor $t$0 and variance $t$1; prompts whose rewards lie closest to this anchor are treated as lying near the current learning boundary, and advantages are fused from both local and global statistics (Bu et al., 5 Jun 2026).

In reasoning RL, SEELE estimates capability at the level of per-instance rollout accuracy $t$2 and fits an item-response-theory-style function

$t$3

mapping hint fraction $t$4 to expected accuracy. It then inverts this model to choose hint lengths that keep each instance near a target accuracy $t$5, thereby matching effective task difficulty to the evolving model (Li et al., 8 Sep 2025). CoBA-RL instead combines a batch-level capability signal

$t$6

with a Beta-distributed preference density over pass rates and a budget saturation factor

$t$7

yielding a capability-oriented value function for rollout allocation (Yao et al., 3 Feb 2026).

Several methods rely on relative rather than absolute signals. SlimSearcher computes efficiency against cohort minima $t$8 and $t$9 among correct trajectories, so the threshold for “efficient enough” tightens automatically as the policy improves (Xie et al., 5 Jun 2026). Re-FORC moves this idea to inference time by forecasting expected future reward as a function of additional thinking tokens,

$r(x,y\mid c,t)$0

and then stopping or continuing reasoning according to whether forecasted gain exceeds token cost (Zabounidis et al., 3 Nov 2025). ADMIRE uses the opposite strategy: instead of a numeric capability scalar, it derives capability from successful trajectories themselves, distilling milestone sequences that are refined whenever newer successes reveal shorter or better subgoal decompositions (Zheng et al., 12 Feb 2026).

4. Domain-specific realizations

In generative image modeling, capability-adaptive reward mechanisms address non-stationary generator quality. HPSv3++ broadens supervision through the 212K-pair HPDv3++ dataset built from Qwen-Image outputs, preserves legacy human-preference knowledge through data-aware orthogonal gradient descent, and then conditions the reward on both generator capability and RL iteration while using standard-deviation-driven unsupervised objectives on unlabeled rollout groups (Liu et al., 12 Jun 2026). AdaGRPO complements this by adapting not the reward model itself but the RL loop around it: prompts are selected near the model’s current capability boundary, and sample quality is judged against both intra-group and history-calibrated baselines rather than only local group statistics (Bu et al., 5 Jun 2026).

In web agents and long-form reasoning, the emphasis shifts from alignment fidelity to compute efficiency. SlimSearcher’s adaptive reward gating keeps correctness as a hard gate and then rewards relative efficiency in tools and tokens, which is specifically designed to avoid brevity bias and reward hacking in long-horizon information seeking (Xie et al., 5 Jun 2026). LASER-D and LASER-DE use length-based reward shaping to teach reasoning models when to think briefly and when to think at length, with dynamic target lengths inferred separately for easy, medium, and hard problems (Liu et al., 21 May 2025). Re-FORC extends capability adaptivity to test-time control by deciding whether further thinking tokens, additional trajectories, or a larger model are worth their cost under a net-utility criterion $r(x,y\mid c,t)$1 (Zabounidis et al., 3 Nov 2025).

In RL with verifiable rewards for symbolic reasoning and code, the same principle appears as difficulty matching. SEELE adaptively appends solution hints so that rollout accuracy remains in the high-variance regime where RLVR learns most effectively (Li et al., 8 Sep 2025). TAROT constructs four-tier test suites and then selects curriculum and reward weights according to model capability, with weaker models benefiting from easy-to-hard schedules and stronger or code-specialized models often benefiting from complex- or hard-first schemes (Park et al., 17 Feb 2026). CoBA-RL replaces uniform rollout counts by a capability-oriented allocation mechanism that decides which prompts deserve more exploration under a fixed total budget (Yao et al., 3 Feb 2026).

In interactive agents and open-ended systems, capability adaptivity often takes the form of structured intermediate reward. ADMIRE anchors GUI-agent reward to milestones distilled from successful trajectories, then denoises successful runs while scaffolding failed runs with partial-progress credit (Zheng et al., 12 Feb 2026). RULE goes further by making the reward coefficients themselves endogenous: $r(x,y\mid c,t)$2 and updating $r(x,y\mid c,t)$3 intergenerationally according to whether realized lifetime component rewards exceed age-conditioned expectations (Bailey, 2024). ACE-RL adapts long-form generation rewards to instruction-specific constraints, and ChARM adapts pairwise role-playing reward margins to behavior-specific modeling confidence, showing that capability-adaptive reward design also applies to subjective alignment tasks rather than only to verifiable environments (Chen et al., 5 Sep 2025, Fang et al., 29 May 2025).

5. Empirical effects

Reported gains are substantial, though highly domain-specific. HPSv3++ reports state-of-the-art preference prediction, outperforming HPSv3 by 9.8% on HPDv3 and 5.5% on GenAI-Bench, while achieving $r(x,y\mid c,t)$4 on HPDv3++ aesthetic and text-following respectively, and it improves GenEval during RL across SDXL, FLUX.1-dev, and Qwen-Image (Liu et al., 12 Jun 2026). AdaGRPO improves both scalar reward metrics and UniGenBench++ scores across Flow-GRPO, DanceGRPO, and Flow-CPS, with smoother reward curves and higher plateaus under both single-reward and multi-reward training (Bu et al., 5 Jun 2026).

In efficiency-sensitive agents, SlimSearcher reduces average tool-call rounds by $r(x,y\mid c,t)$5 while maintaining or improving accuracy on GAIA, BrowseComp, and XBenchDeepSearch, indicating that adaptive gating can shift the accuracy–cost Pareto frontier rather than merely trading accuracy for brevity (Xie et al., 5 Jun 2026). LASER-D and its variant report a $r(x,y\mid c,t)$6 gain on AIME2024 while reducing token usage by $r(x,y\mid c,t)$7, and on smaller reasoning models they improve average benchmark accuracy while cutting average reasoning length by roughly two-thirds (Liu et al., 21 May 2025). Re-FORC reports $r(x,y\mid c,t)$8 compute reduction at maintained accuracy for early stopping, $r(x,y\mid c,t)$9 higher accuracy at equal compute and $$R_{\text{final}} = r_{\text{correct}} r_{\text{tool}} r_{\text{len}}$$0 less compute at equal accuracy for model selection, and $$R_{\text{final}} = r_{\text{correct}} r_{\text{tool}} r_{\text{len}}$$1 to $$R_{\text{final}} = r_{\text{correct}} r_{\text{tool}} r_{\text{len}}$$2 gains from adaptive test-time scaling (Zabounidis et al., 3 Nov 2025).

On math reasoning, SEELE surpasses GRPO by $$R_{\text{final}} = r_{\text{correct}} r_{\text{tool}} r_{\text{len}}$$3 points and SFT by $$R_{\text{final}} = r_{\text{correct}} r_{\text{tool}} r_{\text{len}}$$4 points on average across six math benchmarks, while also outperforming the best previous supervision-aided method by $$R_{\text{final}} = r_{\text{correct}} r_{\text{tool}} r_{\text{len}}$$5 points; these gains are tied directly to keeping rollout accuracy near the theoretically favored mid-range (Li et al., 8 Sep 2025). CoBA-RL reports consistent benchmark improvements over GRPO and Knapsack-RL across multiple Qwen models, and on Qwen2.5-7B-Instruct attains $$R_{\text{final}} = r_{\text{correct}} r_{\text{tool}} r_{\text{len}}$$6 average versus $$R_{\text{final}} = r_{\text{correct}} r_{\text{tool}} r_{\text{len}}$$7 for GRPO, while also showing that $$R_{\text{final}} = r_{\text{correct}} r_{\text{tool}} r_{\text{len}}$$8 accuracy under half budget can exceed GRPO’s $$R_{\text{final}} = r_{\text{correct}} r_{\text{tool}} r_{\text{len}}$$9 under full budget (Yao et al., 3 Feb 2026). TAROT shows that the optimal curriculum indeed flips with model capability: weaker models gain more from easy-to-hard or basic-focused schedules, whereas stronger and code-specialized models often gain more from complex- or hard-first curricula (Park et al., 17 Feb 2026).

For language-generation alignment, ChARM reports about a $R_{\text{TAROT}} = \sum_l \alpha_l w_l r_{i,l}$0 improvement over conventional Bradley–Terry preference ranking and translates those gains into stronger role-playing agents under DPO (Fang et al., 29 May 2025). ACE-RL reports improvements of $R_{\text{TAROT}} = \sum_l \alpha_l w_l r_{i,l}$1 over SFT and $R_{\text{TAROT}} = \sum_l \alpha_l w_l r_{i,l}$2 over RL baselines on WritingBench, and its top-performing model surpasses GPT-4o by $R_{\text{TAROT}} = \sum_l \alpha_l w_l r_{i,l}$3 in that benchmark (Chen et al., 5 Sep 2025). In GUI and embodied agents, ADMIRE reports over $R_{\text{TAROT}} = \sum_l \alpha_l w_l r_{i,l}$4 absolute improvement in success rate across different base models on AndroidWorld and maintains strong gains under GRPO, RLOO, and DAPO in ALFWorld and WebShop (Zheng et al., 12 Feb 2026).

6. Limitations, ambiguities, and open questions

A first ambiguity is definitional. Several papers explicitly show that capability adaptivity does not always require altering the reward scalar itself: SEELE keeps the binary verifiable reward unchanged and instead alters hint-based task difficulty; CoBA-RL reallocates rollouts; Re-FORC leaves the task reward intact and predicts its future value under extra compute (Li et al., 8 Sep 2025, Yao et al., 3 Feb 2026, Zabounidis et al., 3 Nov 2025). A plausible implication is that the field is better understood as studying adaptive reward systems rather than only adaptive reward functions.

A second recurring issue is reward hacking and calibration failure. SlimSearcher shows that removing the correctness gate causes catastrophic collapse, with GAIA accuracy dropping to $R_{\text{TAROT}} = \sum_l \alpha_l w_l r_{i,l}$5 while tool rounds collapse to $R_{\text{TAROT}} = \sum_l \alpha_l w_l r_{i,l}$6, making explicit that efficiency shaping without a hard correctness constraint is unstable (Xie et al., 5 Jun 2026). HPSv3++ notes that reward hacking and subtle failure modes are mitigated but not eliminated, particularly as RL hyperparameters and generator regimes shift (Liu et al., 12 Jun 2026). ADMIRE depends on the quality of milestone generation and the granularity of milestone updates, so poor abstraction or excessive update frequency can move the target in destabilizing ways (Zheng et al., 12 Feb 2026).

A third limitation is the cost of adaptation itself. SEELE requires multi-round rollout sampling and per-instance curve fitting, which is substantially more expensive than single-round GRPO and depends on access to high-quality stepwise solutions (Li et al., 8 Sep 2025). CoBA-RL requires reliable pass-rate estimates and verifiable binary rewards, which narrows its immediate applicability outside domains such as math or code (Yao et al., 3 Feb 2026). ACE-RL depends on strong LLMs for filtering, checklist construction, and constraint verification; mis-specified or biased constraints can turn instruction-adaptive reward into instruction-specific overfitting (Chen et al., 5 Sep 2025). ChARM’s self-evolution loop improves coverage but can overfit or propagate reward-model bias if pseudo-labeled data accumulate without sufficient control (Fang et al., 29 May 2025).

Open directions recur across the literature. HPSv3++ explicitly suggests extending capability-conditioned reward modeling to more modalities and more reward axes such as safety and style (Liu et al., 12 Jun 2026). AdaGRPO suggests that prompt selection and advantage shaping can be made capability-aware with minimal architectural change, which raises the prospect of similar mechanisms in language, audio, or multimodal RL (Bu et al., 5 Jun 2026). TAROT, ACE-RL, and ADMIRE collectively indicate that future systems may combine curriculum selection, structured intermediate constraints, and instruction- or capability-conditioned aggregation rather than relying on a single adaptive trick (Park et al., 17 Feb 2026, Chen et al., 5 Sep 2025, Zheng et al., 12 Feb 2026).