Learning What Matters Now: A Dual-Critic Context-Aware RL Framework for Priority-Driven Information Gain (2506.06786v1)

Abstract: Autonomous systems operating in high-stakes search-and-rescue (SAR) missions must continuously gather mission-critical information while flexibly adapting to shifting operational priorities. We propose CA-MIQ (Context-Aware Max-Information Q-learning), a lightweight dual-critic reinforcement learning (RL) framework that dynamically adjusts its exploration strategy whenever mission priorities change. CA-MIQ pairs a standard extrinsic critic for task reward with an intrinsic critic that fuses state-novelty, information-location awareness, and real-time priority alignment. A built-in shift detector triggers transient exploration boosts and selective critic resets, allowing the agent to re-focus after a priority revision. In a simulated SAR grid-world, where experiments specifically test adaptation to changes in the priority order of information types the agent is expected to focus on, CA-MIQ achieves nearly four times higher mission-success rates than baselines after a single priority shift and more than three times better performance in multiple-shift scenarios, achieving 100% recovery while baseline methods fail to adapt. These results highlight CA-MIQ's effectiveness in any discrete environment with piecewise-stationary information-value distributions.

• The paper introduces CA-MIQ, a dual-critic RL framework that fuses extrinsic task rewards with intrinsic exploration to drive adaptive behavior in SAR missions.
• CA-MIQ outperformed baseline methods by achieving nearly four times higher success rates and 100% recovery after priority shifts in grid-world evaluations.
• The framework enhances real-time decision-making in dynamic environments, offering promising avenues for future research in hierarchical and meta-RL approaches.

Dual-Critic Context-Aware Reinforcement Learning for Adaptive Information Gain in SAR Missions

In the paper "Learning What Matters Now: A Dual-Critic Context-Aware RL Framework for Priority-Driven Information Gain," Panagopoulos et al. introduce CA-MIQ, a reinforcement learning (RL) framework designed specifically for autonomous systems engaged in high-stakes search-and-rescue (SAR) missions. The framework addresses the need for dynamic adaptation in environments where information priorities can change abruptly due to evolving mission contexts.

Framework Design

CA-MIQ utilizes a dual-critic architecture to navigate the complexity of SAR operations. Unlike traditional RL methods that assume a stationary environment, CA-MIQ employs both extrinsic and intrinsic critics:

1. Extrinsic Critic: This critic learns from task-specific rewards, guiding the agent to perform actions that yield immediate operational benefits.
2. Intrinsic Critic: This critic maximizes information gain by incentivizing exploration based on novelty, information-location awareness, and alignment with current mission priorities. The intrinsic critic helps the agent adapt its strategy swiftly when a priority shift occurs.

The integration of dual critics is specifically tailored to enhance adaptability and efficiency in SAR missions where priorities fluctuate, challenging the assumptions of existing RL models.

Experimental Evaluation

The framework was tested in SAR scenarios modeled as grid-world environments, characterized by discrete state-action spaces typical of high-level decision-making tasks. Results demonstrated remarkable adaptability in the face of priority shifts. CA-MIQ achieved nearly four times higher success rates than baseline methods following a priority change, showcasing a strong capacity for prompt adaptation and realignment with new operational needs.

Notably, CA-MIQ achieved 100% recovery success after priority shifts, a feat that baseline methods failed to replicate, underscoring the effectiveness of the context-aware approach. In single and multiple priority shift scenarios, CA-MIQ maintained superior mission success rates, illustrating its robustness in dynamic conditions.

Implications and Future Directions

This work holds significant implications for the development of autonomous systems in SAR environments and similar contexts that require rapid response to changing information landscapes. The ability to dynamically adjust exploration strategies based on shifting priorities can enhance mission success rates, reduce risk exposure, and optimize time usage, critical factors in disaster response and recovery operations.

Looking forward, integrating CA-MIQ with other learning paradigms, such as hierarchical learning and meta-RL, could further improve adaptability and transfer capabilities across diverse missions and environments. The paper invites future research to explore adaptive weighting mechanisms for intrinsic motivation components, extending the framework's applicability beyond discrete state-action spaces and into domains characterized by partial observability.

By addressing the gap between stationary RL environments and the dynamic demands of real-world missions, CA-MIQ offers a promising step toward more agile and responsive autonomous systems, capable of maintaining operational efficacy in evolving conditions.