Adaptive Reward Fusion (ARF)

• Adaptive Reward Fusion (ARF) is a dynamic integration approach that flexibly combines diverse reward and feedback signals on a per-instance basis to enhance policy learning.
• It employs mechanisms like attention-based feature fusion, hybrid reward scheduling, and Bayesian inference to adaptively modulate signal weights according to context.
• Empirical studies show ARF improves sample efficiency, training stability, and robustness in applications such as RL from human feedback and reward modeling.

Adaptive Reward Fusion (ARF) refers to a class of mechanisms for context-dependent, data-driven integration of multiple reward, feedback, or feature signals in reinforcement learning (RL), reward modeling, and RL from human feedback (RLHF). ARF approaches span expressive attention-based feature fusion, dynamic hybridization of disparate rewards, adaptive routing over expert aggregation, and principled Bayesian inference over conflicting reward specifications. Central to ARF is the shift from static or fixed reward combination to adaptive, learnable processes that modulate how different sources of information influence policy learning and value estimation, often on a per-instance or per-state basis.

1. Formal Definitions and Core Mechanisms

Adaptive Reward Fusion arises in settings where an agent must reconcile or synergistically use multiple streams of reward or feedback. The sources may be diverse: learned dynamics representations, explicit reward signals, human preferences (scalar, pairwise, or continuous), or outputs from distinct reward models. Unlike naive averaging or fixed-weight schemes, ARF mechanisms embed flexibility—typically via gating, attention, or scheduling functions—so that the agent dynamically adjusts the weight assigned to each input in response to context.

Representative instantiations include:

• Attention-based Feature Fusion: Self-attention or gating modules fuse high-dimensional feature vectors, such as dynamics-specific embeddings $\phi_{\mathrm{dyn}}(s)$ and reward-specific embeddings $\phi_{\mathrm{rew}}(s)$, into a single representation $\phi_{\mathrm{fused}}(s)$ for policy/critic input (Nath et al., 16 Jun 2026).
• Hybrid Reward Scheduling: Mixtures of sparse (“hard”) and dense (“continuous”) reward components, with a time-varying schedule $\lambda(t)$, enabling smooth curricula from exploration (via shaping) to exploitation (via correctness) (Sahoo, 17 Nov 2025).
• Multi-Perspective Adaptive Pooling: Routing networks that combine outputs of distinct scoring views (last-token, mean, attention) to form a scalar reward, with mixture weights produced per-instance (Miao et al., 13 Jan 2026).
• Feedback Fusion in RLHF: Simultaneous use of numerical rewards and preference-based (pairwise or continuous) feedback in unified loss functions with learnable or scheduled weighting (Khorasani et al., 15 Aug 2025).
• Bayesian Posterior Fusion: Robust computation of posteriors over reward functions from conflicting reward sources, retreating to uncertainty when evidence disagrees and adaptively updating as new data arrives (Krasheninnikov et al., 2021).

2. Mathematical and Algorithmic Formulations

The fusion mechanism is realized in different mathematical frameworks according to task demands and feedback structure:

A. Attention-based Feature Fusion

Given state $s$,

• $$\phi_{\mathrm{dyn}}(s), \phi_{\mathrm{rew}}(s) \in \mathbb{R}^d$$
• Concatenate: $$f_{\mathrm{cat}}(s) = [\phi_{\mathrm{dyn}}(s); \phi_{\mathrm{rew}}(s)] \in \mathbb{R}^{2d}$$
• Compute queries/keys/values: $Q(s) = W^Q f_{\mathrm{cat}}(s)$, $K(s) = W^K f_{\mathrm{cat}}(s)$, $V(s) = W^V f_{\mathrm{cat}}(s)$
• Attention: $\phi_{\mathrm{rew}}(s)$0
• Fused: $\phi_{\mathrm{rew}}(s)$1 (Nath et al., 16 Jun 2026)

A scalar-gate variant uses a learnable $\phi_{\mathrm{rew}}(s)$2 to blend features.

B. Hybrid Reward Schedulers

Given a hard reward $\phi_{\mathrm{rew}}(s)$3 and a dense reward $\phi_{\mathrm{rew}}(s)$4, fuse: $\phi_{\mathrm{rew}}(s)$5 where $\phi_{\mathrm{rew}}(s)$6 is linearly or exponentially scheduled over training steps (Sahoo, 17 Nov 2025).

C. Multi-View Adaptive Pooling

Given three view-extracted features $\phi_{\mathrm{rew}}(s)$7, $\phi_{\mathrm{rew}}(s)$8, $\phi_{\mathrm{rew}}(s)$9 and context $\phi_{\mathrm{fused}}(s)$0:

• Scores: $\phi_{\mathrm{fused}}(s)$1, $\phi_{\mathrm{fused}}(s)$2, $\phi_{\mathrm{fused}}(s)$3
• Routing: $\phi_{\mathrm{fused}}(s)$4
• Reward: $\phi_{\mathrm{fused}}(s)$5 (Miao et al., 13 Jan 2026)

D. Joint Reward–Preference Fusion

In RL, combine SAC policy gradient $\phi_{\mathrm{fused}}(s)$6 and a Bradley–Terry preference loss $\phi_{\mathrm{fused}}(s)$7, total loss: $\phi_{\mathrm{fused}}(s)$8 where $\phi_{\mathrm{fused}}(s)$9 may be a fixed or scheduled hyperparameter (Khorasani et al., 15 Aug 2025).

E. Bayesian Posterior Fusion over Reward Functions

Given $\lambda(t)$0 learned reward parameters $\lambda(t)$1, form

$\lambda(t)$2

Robust fusion (MIRD, MIRD-IF) uses mixtures, maximum-causal-entropy IRL, and support over independently correct features (Krasheninnikov et al., 2021).

3. Application Domains and Empirical Performance

A. Representation Fusion in RL

In "Structured Representation Learning with Locally Linear Embeddings and Adaptive Feature Fusion" (Nath et al., 16 Jun 2026), ARF as self-attention gating between dynamics- and reward-specific representations enables a SAC agent to achieve superior sample efficiency (80% of top performance in $\lambda(t)$3300K steps vs. $\lambda(t)$4450K for vanilla SAC across 8 RoboSuite tasks) and higher final returns than MLP-fusion, scalar-gate, or concat baselines. Attention heatmaps reveal context-dependent shifting between feature reliance throughout episode progression and environment complexity.

B. Reward Schedules for RLHF

In "The Good, The Bad, and The Hybrid: A Reward Structure Showdown in Reasoning Models Training" (Sahoo, 17 Nov 2025), ARF as a mixture of hard and dense reward yields intermediate trade-offs: hybrid schedules produce $\lambda(t)$533% accuracy with improved training stability over purely discrete (40% accuracy, high-variance) or continuous (28% accuracy, stable) reward. The adaptation of $\lambda(t)$6 functions as a curriculum over reward functions.

C. Adaptive Aggregation in Reward Modeling

In "AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling" (Miao et al., 13 Jan 2026), the ARF approach outperforms static pooling (last-token, mean, attention) in pairwise preference accuracy (Qwen-3-4B: 70.8 with AdaJudge vs. 67.6/69.4 with static), and especially on difficult or heterogeneous domains (RM-Bench "hard" subset: 43.7 vs. 35.0/41.4). Ablations confirm the critical role of both gated representation refinement and adaptive pooling.

D. RLHF and Reward Modeling Personalization

"ARF-RLHF: Adaptive Reward-Following for RLHF through Emotion-Driven Self-Supervision and Trace-Biased Dynamic Optimization" (Zhang, 3 Jul 2025) demonstrates ARF's use of emotion-driven continuous feedback, data augmentation, and a preference tracker to surpass both PPO and DPO in LLM finetuning across multiple backbone architectures (Qwen2, Gemma2, LLaMA3.2), with gains of +3.3% over PPO and +7.6% over DPO. The online adaptive mechanism tracks shifting user preferences and supports zero human annotation after static corpus construction.

E. Dual Feedback for Actor Optimization

"Fusing Rewards and Preferences in Reinforcement Learning" (Khorasani et al., 15 Aug 2025) establishes DFA as an ARF instance fusing scalar and preference feedback, matching/exceeding SAC in MuJoCo continuous control and outperforming RLHF baselines in preference-supplied GridWorld. Empirical curves show smoother learning and robustness to feedback noise compared to fixed-reward RL.

F. Robust Bayesian Reward Combination

"Combining Reward Information from Multiple Sources" (Krasheninnikov et al., 2021) introduces MIRD/MIRD-IF, explicitly constructing posteriors that preserve support on independently correct features, maintain behavior-space balance, and adaptively update as new input arrives. Only methods with such support succeed in settings where learned rewards conflict or incompletely specify the true objective.

4. Theoretical Guarantees and Desiderata

The theoretical backbone of ARF is characterized by:

• Behavioral Convexity: Fused representations or reward posteriors yield convex combinations of policies or feature-expectations, ensuring coverage of possible desirable behaviors (Krasheninnikov et al., 2021).
• Strong Informativeness: Where multiple sources agree on observed behavior, the fused construct concentrates posterior mass, optimizing jointly for those aspects.
• Conservatism and Uncertainty Retreat: In case of model conflict or misspecification, robust ARF variants (MIRD-IF) expand posterior support, avoiding unwarranted confidence (Krasheninnikov et al., 2021).
• Theoretical Alignment with RL Objectives: Joint reward–preference losses reduce to entropy-regularized RL (SAC) or actor-critic objectives under specified probabilistic models (Khorasani et al., 15 Aug 2025, Zhang, 3 Jul 2025).

5. Practical Implementations and Hyperparameterization

ARF mechanisms require careful hyperparameterization:

• Schedule Design (Hybrid Schedulers): Linear or exponential ramps for $\lambda(t)$7 (e.g., $\lambda(t)$8 selection) determine the exploration-exploitation trade-off and convergence properties (Sahoo, 17 Nov 2025).
• Mixture and Routing Networks: Gating architectures typically use small MLPs or transformers; softmax ensures mixture weights are normalized.
• Data Augmentation and Replay: RLHF settings benefit from synonym replacement, trace truncation, and periodic buffer re-evaluation to maximally exploit limited feedback (Zhang, 3 Jul 2025).
• Preference and Reward Weights: Weight $\lambda(t)$9 in joint losses (DFA, hybrid scheduling, MIRD prior) tunes the dominance of feedback signals, often requiring ramp-up or annealing (Khorasani et al., 15 Aug 2025, Krasheninnikov et al., 2021).
• Efficiency and Scalability: ARF modules add computation (e.g., attention/fusion forward passes), but can be optimized via batch replay or low-rank adapters for large models (Nath et al., 16 Jun 2026, Zhang, 3 Jul 2025).

6. Extensions, Limitations, and Future Directions

Possible extensions include:

• Meta-learning of Schedulers or Fusion Gates: Adaptive or learned curricula for scheduling fusion weight, possibly via meta-gradient approaches (Sahoo, 17 Nov 2025).
• Expansion to Multi-modal Feedback: Incorporation of multimodal or group-based feedback (multisession preference tracking, voice, facial signals) (Zhang, 3 Jul 2025).
• Automated Discovery of Expert Pooling / Routing: Discovering new expert pooling operations and scaling mixture-of-expert approaches for reward aggregation (Miao et al., 13 Jan 2026).
• Robust Online Bayesian Fusion: Continual update of reward posteriors as new sources arrive, with automated tuning of trust priors and posterior variance (Krasheninnikov et al., 2021).
• Scalability to Large Foundation Models: Empirical validation and architectural compression (distillation, low-rank heads) for ARF at LLM scales (Miao et al., 13 Jan 2026).
• Multi-objective ARF: Simultaneously integrating disparate objectives (e.g., factuality, helpfulness, policy compliance) with conditional fusion networks (Miao et al., 13 Jan 2026).

Limitations identified in the literature include increased parameter count, inference cost, hand-picking of expert views, and validation limited to sub-10B parameter models (Miao et al., 13 Jan 2026). Empirical findings indicate that the expressivity of the adaptive module and the flexibility of weight schedules are central to robust, high-performing ARF systems, but must be balanced against overfitting proxies and computational cost.

7. Comparative Table of ARF Mechanisms

Paper / System	ARF Mechanism	Domain / Application
(Nath et al., 16 Jun 2026) SAC-LLE	Self-attention Feature Fusion	RL, control (robotics)
(Sahoo, 17 Nov 2025) Hybrid Scheduler	Time-varying Reward Mixture	RLHF, reasoning LLMs
(Miao et al., 13 Jan 2026) AdaJudge	Adaptive Multi-View Pooling	Reward modeling, LLMs
(Zhang, 3 Jul 2025) ARF-RLHF	Emotion-based Scorer Fusion	RLHF, dialogue LLMs
(Khorasani et al., 15 Aug 2025) DFA	Reward–Preference Joint Loss	RL, continuous control
(Krasheninnikov et al., 2021) MIRD/MIRD-IF	Bayesian Posterior Fusion	Inverse RL, robust planning

These approaches collectively demonstrate the generality and impact of ARF as a paradigm for robust, context-sensitive integration of multiple reward and feedback streams in modern reinforcement learning and reward modeling frameworks.