Composite Rewards in Sequential Decision-Making

• Composite rewards are reward structures that integrate multiple sub-reward signals across time, features, or objectives to offer rich, multifaceted feedback in decision-making.
• Researchers implement composite rewards using component-based, hierarchical, and bandit-based frameworks to improve credit assignment and policy optimization in complex environments.
• Applications span robotics, finance, and natural language generation, demonstrating enhanced convergence, reduced regret, and improved alignment with human evaluations.

Composite rewards are reward structures in reinforcement learning (RL) and related sequential decision-making domains that are constructed by combining multiple sub-reward components, stages, or signals—either across time, features, modalities, or objectives. Rather than receiving or optimizing for a single scalar reward (as in classical RL), agents utilizing composite rewards receive multifaceted signals, which can be aggregated, temporally distributed, or structurally integrated to guide policy learning and evaluation. Composite rewards address the limitations of single-objective feedback, enabling richer supervision and more nuanced optimization, especially in complex, multi-objective, delayed, or partially observed environments.

1. Formal Definitions and Taxonomy

Composite rewards can be systematically categorized along several dimensions, reflecting their structure, temporal distribution, and the domains they apply to:

• Component-based composition: Rewards are constructed as a linear or nonlinear combination of sub-reward functions $r_k$, each capturing different task objectives or performance metrics (e.g., $R(\mathbf{x}, a) = \sum_k w_k r_k(\mathbf{x}, a)$).
• Spread (temporal) composition: Rewards are fragmented and received over extended intervals after an action, often as a vector of time-indexed components $(r_{t,0}, r_{t,1}, ...)$, with $r_{t,\tau}$ realized $\tau$ steps after taking action at $t$ (1910.01161, 2012.07048, 2305.02527, 2303.13604).
• Anonymous and aggregated feedback: Composite feedback may aggregate reward fragments from distinct past actions, preventing attribution of observed rewards to particular actions (1910.01161, 2012.07048, 2303.13604).
• Multi-objective and multi-metric structures: The agent balances or alternates optimization across diverse performance metrics, often using frameworks like bandit-based metric selection (2011.07635) or reward balancing in policy gradient methods (2305.03286).
• Hierarchical and modular composition: Rewards are structured according to the decomposition of the task into subtasks, often formalized as hierarchies of reward machines or finite-state automata (2205.15752).
• Delayed composite feedback: The agent receives global, potentially non-Markovian, sequence-level rewards calculated from a composite function of the trajectory rather than merely summing stepwise rewards (2410.20176).

This diversity of composite reward forms allows for targeted feedback, improved credit assignment, and compatibility with human assessment or practical constraints on measurement and feedback.

2. Mathematical Frameworks and Modeling Approaches

Composite reward modeling spans a range of inference and optimization structures:

• Bayesian observation model: In independent reward design, each proxy reward specified in an environment is treated as an observation of the underlying true reward parameter vector $\theta$. The posterior over $\theta$ is given by

$$P(\theta | r_{1:N}, M_{1:N}) \propto \prod_{i=1}^N P(r_i | \theta, M_i) P(\theta),$$

where $$P(r_i | \theta, M_i) \propto \exp(\beta R(\xi^*; \theta))$$ parameterizes designer optimality and $\xi^*$ is the optimal proxy path (1806.02501).

• Multi-objective aggregation: Composite rewards often sum weighted sub-rewards:

$$R(\mathbf{x}, a) = \sum_{k=1}^K w_k r_k(\mathbf{x}, a)$$

In domains like financial trading (2506.04358), the reward is

$$\mathcal{R} = w_1R_{ann} - w_2\sigma_{down} + w_3\text{D}_{ret} + w_4\text{T}_{ry}$$

with closed-form terms for annualized return, downside risk, differential return, and Treynor ratio.

• Hierarchical and sequential models: Hierarchies of reward machines and transformers with in-sequence attention implement composite (possibly non-Markovian) reward functions, aggregating instance-level rewards via learned or predefined weights reflecting each step’s contribution to a global reward (2205.15752, 2410.20176).
• Bandit-based selection: The DORB framework uses multi-armed bandit algorithms (Exp3) to dynamically choose which sub-reward to optimize at each round, providing adaptive multi-reward optimization rather than fixed-weighted combination (2011.07635).
• GAN-based decoupled reward: In composite motion learning, composite rewards are provided by multiple discriminators, each producing a partial imitation reward for a distinct motion group, and task-driven rewards, which are normalized and jointly optimized (2305.03286).

The choice of framework reflects both the temporal and hierarchical structure of the learning task, as well as the need for adaptivity to noisy, sparse, or multi-source feedback.

3. Algorithms and Implementation Strategies

Implementing composite rewards entails both reward modeling and appropriate algorithmic integration:

• Independent reward design: Practitioners specify environment-specific rewards in a "divide-and-conquer" fashion; the inference of a common reward is accomplished via Bayesian modeling and posterior sampling, frequently using Monte Carlo approximation of normalizing constants and Metropolis algorithms for posterior exploration (1806.02501).
• Bandit-based reward scheduling: Training loops alternate among reward objectives according to bandit algorithms; metrics are scaled into a uniform range, and the controller dynamically selects the next reward signal based on observed performance gains (2011.07635). This avoids manual tuning of static weights.
• Multi-critic advantage estimation: In deep RL with composite rewards, separate critics (value functions) are trained per reward, and the overall policy gradient loss is aggregated with adaptive weights and normalized advantage streams to prevent overemphasis on any single objective (2305.03286).
• Delayed composite feedback decontamination: Adaptive algorithms (ARS-UCB, ARS-EXP3) tackle the multi-armed bandit problem with composite, temporally spread, and anonymous reward by grouping actions into rounds of increasing length to ensure the error vanishes as rounds lengthen—thus mimicking conventional UCB/EXP3 behavior (2012.07048).
• Reward model pretraining and imputation: Offline RL methods can address missing or composite reward signals by training a supervised reward model (e.g., an MLP) on the available annotated samples and imputing rewards for unlabelled transitions, effectively reconstructing a composite reward vector for each transition (2407.10839).
• Transformer-based modeling of non-Markovian composite rewards: The Composite Delayed Reward Transformer (CoDeTr) leverages causal transformers with in-sequence attention to predict the composite reward as a softmax-weighted sum of sequence-level reward predictions, enabling credit assignment to crucial steps beyond simple temporal summation (2410.20176).

4. Domains and Applications

Composite rewards arise in several real-world and research domains as a direct response to multi-faceted or delayed feedback scenarios:

• Robot planning and motion control: Divide-and-conquer reward design yields more tractable reward engineering, especially in manipulation, navigation, and motion imitation tasks (1806.02501, 2305.03286).
• Natural language generation: Visual storytelling and sequence generation benefit from simultaneously optimizing for human-centric criteria such as relevance, coherence, and expressiveness, often encoded through a tailored composite reward (1909.05316, 2011.07635).
• Multi-agent systems: Difference rewards assign credit more fairly to individual agent contributions within a global outcome, fostering scalable decentralized policy learning (2012.11258).
• Online learning and bandits: Ad delivery, clinical trials, and recommendation systems face delays and aggregation in observed rewards; composite reward algorithms are key for bounding regret in these settings (1910.01161, 2012.07048, 2303.13604, 2305.02527).
• Finance: Robust trading strategies require balancing absolute returns, risk (both total and downside), and market-relative metrics; composite rewards can capture nuanced investor utility profiles (2506.04358).
• RL with human-in-the-loop feedback: Human-generated sequence-level evaluations often reflect composite judgments over key moments; modeling these with attention-based or hierarchical approaches closely matches real evaluation practices (2410.20176).
• Multi-modal model evaluation: Reward models integrating text, images, and videos use composite signals to reflect human preferences across modalities, providing training targets, test-time selection, or data cleaning (2501.12368).

5. Empirical Findings and Theoretical Guarantees

The use of composite rewards frequently results in measurable improvements in practical sample efficiency, credit assignment, generalization, and robustness:

• Performance metrics: Empirical studies in diverse tasks demonstrate lower regret, faster convergence, reduced design time, and improved alignment with human judgments when using composite over single-objective rewards (1806.02501, 1909.05316, 2011.07635, 2305.03286, 2501.12368).
• Regret bounds: In bandit problems with composite feedback, regret bounds of the order $\tilde{O}(T^{2/3} + T^{1/3}\nu)$ (where $\nu$ reflects delay or feedback spread) have been established, and adaptive algorithms show order-optimality (2303.13604, 2012.07048).
• Robust policy learning: Algorithms leveraging hierarchical or adaptive composite rewards are shown to yield superior robustness and transfer in the face of noisy, partial, or delayed reward signals (2205.15752, 2410.09187, 2410.20176).
• Optimization properties: Composite reward functions designed for financial trading are shown to be monotonic, bounded, and modular, ensuring stability for gradient-based optimization (2506.04358).

6. Challenges and Limitations

Despite their broad applicability, composite rewards introduce their own computational and conceptual complexities:

• Weight tuning: Linear combinations require setting (and sometimes adaptively updating) mixture weights; poor tuning may lead to focusing excessively on particular objectives (2305.03286, 2506.04358).
• Estimation under delayed/anonymous feedback: Proper decontamination (or attribution) is required to avoid biased estimates, especially in bandit and MDPs with composite delayed feedback (1910.01161, 2012.07048, 2305.02527, 2303.13604).
• Non-Markovian credit assignment: When rewards depend on the full sequence, standard RL credit assignment breaks down, necessitating more sophisticated sequence modeling (e.g., transformer architectures or in-sequence attention) (2410.20176).
• Computational cost: In Bayesian inference for independent reward design, approximate sampling and planning (e.g., for the normalizing constant) may incur additional computational load (1806.02501).
• Potential for conflicting signals: Aggregating rewards from disparate objectives or timescales can sometimes introduce optimization conflicts, requiring careful algorithmic design (multi-critic value normalization, curriculum learning) (2305.03286, 2011.07635).

7. Future Directions and Broader Implications

Advances in composite reward modeling are influencing several trends and open research topics:

• Automated reward engineering: LLM-driven progress functions and composite reward frameworks are enabling data-driven, task-adaptive reward synthesis without manual design (2410.09187).
• Transfer and generalization: Modular and hierarchical reward decomposition facilitates systematic generalization and policy transfer across tasks and domains (2205.15752).
• Human-aligned RL: Composite and delayed reward models better capture the nuanced, sequence-level feedback provided by human evaluators, which is increasingly important in safety-critical and user-facing applications (2410.20176, 2501.12368).
• Dynamic weighting and adaptivity: Future methods are expected to move toward adaptive, curriculum-based, or meta-learned weighting of composite sub-rewards to reflect nonstationarity or changing priorities during training (2011.07635, 2506.04358).
• Robustness and explainability: The explicit, modular structure of composite rewards enhances the interpretability of agent behavior and reward contribution analyses, which is desirable for deployment in high-stakes settings (2305.03286, 2506.04358).

Composite rewards thus serve as a foundational tool for structuring supervision, credit assignment, and policy optimization in complex, real-world RL and online learning problems, supporting both algorithmic rigor and practical relevance across diverse applications.