Auction-Based Online Policy Adaptation for Evolving Objectives

Abstract: We consider multi-objective reinforcement learning problems where objectives come from an identical family -- such as the class of reachability objectives -- and may appear or disappear at runtime. Our goal is to design adaptive policies that can efficiently adjust their behaviors as the set of active objectives changes. To solve this problem, we propose a modular framework where each objective is supported by a selfish local policy, and coordination is achieved through a novel auction-based mechanism: policies bid for the right to execute their actions, with bids reflecting the urgency of the current state. The highest bidder selects the action, enabling a dynamic and interpretable trade-off among objectives. Going back to the original adaptation problem, when objectives change, the system adapts by simply adding or removing the corresponding policies. Moreover, as objectives arise from the same family, identical copies of a parameterized policy can be deployed, facilitating immediate adaptation at runtime. We show how the selfish local policies can be computed by turning the problem into a general-sum game, where the policies compete against each other to fulfill their own objectives. To succeed, each policy must not only optimize its own objective, but also reason about the presence of other goals and learn to produce calibrated bids that reflect relative priority. In our implementation, the policies are trained concurrently using proximal policy optimization (PPO). We evaluate on Atari Assault and a gridworld-based path-planning task with dynamic targets. Our method achieves substantially better performance than monolithic policies trained with PPO.

• The paper presents a novel auction-based bidding mechanism that enables decentralized, modular policy arbitration in dynamic multi-objective reinforcement learning environments.
• It employs a shared neural network with an attention pooling module and PPO training, achieving significant performance gains in gridworld and Atari benchmarks.
• Experimental results show robust scalability and clear interpretability, with optimized bid penalties preventing overbidding and facilitating swift online adaptation.

Auction-Based Modular Policy Adaptation for Dynamic Multi-Objective Reinforcement Learning

Introduction

This paper addresses the challenge of online adaptation to evolving objective sets in multi-objective reinforcement learning (MORL) environments. Classical MORL methods, relying on scalarization or Pareto optimality, lack flexibility and interpretability, particularly when the objective set is dynamic. The authors propose a decentralized, modular framework wherein each objective is controlled by a selfish local policy. These policies coordinate via a novel auction-based bidding mechanism. This composition enables immediate, interpretable online adaptation, facilitating fast response to the creation or removal of objectives during deployment.

Auction-Based Bidding Mechanism for Policy Composition

The central innovation is the formulation of the multi-objective decision problem as a general-sum Markov game. Each local policy is augmented with a bid, representing the policy’s internal estimate of the urgency or value of executing its associated action in the global context. At each decision point, policies submit their actions and bids; the action of the highest-bidding policy is executed, with ties resolved randomly. This competitive mechanism for policy arbitration is parameterized by three key hyperparameters: the bidding interval $\tau$, the maximum bid $\beta$, and the bid penalty coefficient $\rho$.

To prevent pathological behavior—such as policies always overbidding to suppress competitors—bid penalties proportional to the bid value are imposed, using either a Winner-Pays or All-Pay penalty scheme. The mechanism is decentralized: no global meta-controller evaluates cumulative utility.

Framework Implementation and Optimization

A shared neural network architecture is employed for all policies when objectives are from the same family (e.g., reachability, as in the cat-feeder setting). Policies are trained concurrently using Proximal Policy Optimization (PPO), leveraging principles from recent multi-agent RL optimization studies. Observation spaces for local policies include not only global state and all active objectives but also incorporate a learned attention pooling module. This architecture enables efficient aggregation of an arbitrary number of objectives into a fixed-length embedding, facilitating scalability and handling a dynamically changing objective set. Policies are required to infer the global configuration of objectives for effective, context-aware bidding.

Experimental Evaluation

Comprehensive empirical studies are conducted on two benchmarks: a gridworld-based cat feeder environment and Atari Assault, both reframed to fit the incremental MORL setting.

Substantial performance improvements are observed over strong baselines, including single-policy PPO with scalarized rewards and deep W-learning (DWN), as demonstrated by the learning curves:

Figure 2: Learning dynamics in Cat Feeder and Atari Assault environments, highlighting accelerated convergence and superior final returns for bidding-based policies.

For Cat Feeder, the All-Pay auction mechanism achieves an average performance of 79.46 feedings minus expirations, compared to -2.28 for DWN and 22.17 for best reward-shaped PPO. In Atari Assault, bidding-based policies maintain an average score above 660, outperforming PPO (430.08) and DWN (310.80). Notably, monolithic architectures with scalarized (even reward-shaped) objectives exhibit severe credit assignment limitations, especially under asynchronous objective appearance/disappearance and sparse rewards.

Analysis of Policy Control and Bidding Behavior

The control distribution among policies and their bid profiles are analyzed to investigate the interpretability and internal competitive dynamics of the system:

Figure 1: Distribution of control time across policies in Cat Feeder, validating effective resource allocation and frequent policy handoffs.

Bids are conservatively selected in environments without expiring objectives (as in Atari Assault), while urgent scenarios (Cat Feeder) elicit a broader, differentiated bidding spectrum. The All-Pay penalty drives more frugal bidding, while Winner-Pays supports more aggressive competition. The system yields easily interpretable runtime explanations—at any step, it is possible to audit which policy is in control and why.

Scaling and Ablation Studies

Scalability is tested by increasing the number of simultaneous objectives beyond the training regime:

Figure 3: Test-time scaling in Cat Feeder: policies trained for $m=7$ objectives generalize robustly to $m=14$ at inference.

Performance remains robust, with only mild degradation when doubling the number of concurrent objectives. This outcome is attributed to the attention pooling module, which allows the network to generalize to arbitrary numbers of objectives at deployment.

Ablation studies reveal the necessity of: (i) at least two bid levels for meaningful competition, (ii) balanced bid penalty values for preventing overbidding or complete abstention, and (iii) optimal action window size—too small leads to oscillatory control, too large prevents effective switching.

Theoretical and Practical Implications

The modular, auction-based model presents distinct advantages:

• Immediate objective set adaptation: New objectives can be served instantly by instantiating new copies of the generic policy.
• Interpretability: Real-time policy arbitration and explicit objective prioritization enable auditability and post-hoc failure analysis.
• Unattainable by scalarization: Scalar-reward compression cannot realize the context-dependent trade-offs required by dynamic objective sets or facilitate assignment of objective-specific reward shaping.

The framework effectively resolves multi-objective arbitration without a global orchestrator or reliance on hand-coded priority rules or externally computed Pareto frontiers. Its reliance on standard deep actor-critic methods (PPO) underscores its implementational simplicity.

Future Directions

The current method only handles objectives of the same family, e.g., reachability, due to shared policy architecture and homogeneous observation embeddings. Extension to heterogeneous objectives (e.g., mixing reachability and safety) imposes new challenges: policies must learn to bid contextually against opponents from distinct families, potentially requiring richer auction protocols or override rules (e.g., critical safety always preempts regular objectives).

Enhanced bidding schemes, hierarchically structured auctions, or non-linear value aggregation are candidate extensions, especially as system objectives grow in complexity and heterogeneity.

Conclusion

The auction-based modular adaptation framework provides a principled and effective solution to online control in MORL settings with arbitrary, evolving objective sets. By decentralizing policy trade-off via a learned, penalized bidding game, the method yields interpretable, immediately adaptive, and high-performing controllers. These results highlight its suitability for dynamic multi-agent and robotics applications where resilience to objective change is mandatory.