Cognition-aware Policy Optimization (CoPO)

• Cognition-aware Policy Optimization (CoPO) is a dynamic reinforcement learning framework that modulates cognitive depth and social coordination based on task complexity.
• It employs a two-stage process combining supervised fine-tuning with modified PPO/GRPO objectives to adapt reasoning strategies and reward shaping.
• Experimental results demonstrate improved success rates and efficiency in both LLM-based agents and multi-agent environments, reducing token usage and enhancing coordination.

Cognition-aware Policy Optimization (CoPO) encompasses a family of reinforcement learning algorithms for aligning cognitive resource allocation with task demands in both LLM agents and multi-agent systems. These approaches are characterized by dynamic adaptation of cognitive depth, explicit control of reasoning strategies, or coordination of social reasoning factors, aimed at optimizing for correctness, efficiency, and adaptive behavior in sequential decision environments.

1. Conceptual and Formal Foundations

Cognition-aware Policy Optimization operationalizes cognitive modulation—explicit control over an agent’s reasoning effort—by incorporating cognitive-level or social modulation variables directly into the policy optimization loop. The approach is applicable in two main domains: LLM-based sequential agents and decentralized multi-agent control.

In LLM settings, CoPO is structured as a Markov Decision Process (MDP) or partially observable MDP, with cognitive depth represented via special tokens or structured annotations. The policy $\pi_\theta$ outputs not only actions but also cognitive-level indicators at each step. For multi-agent systems, as in self-driven particle (SDP) and traffic modeling, cognition is reflected in the local social coordination factor (LCF), which parameterizes the balance between self-interest and group-level cooperation.

Reward shaping, dynamic credit assignment, and meta-gradient adaptation are central. The optimization goal is typically:

$$\theta^* = \arg\max_\theta\,\mathbb{E}_{\tau\sim\pi_\theta}[\mathcal{R}(\tau)]$$

where $\mathcal{R}(\tau)$ is a composite reward incorporating accuracy, length/efficiency, cognitive allocation, or social coordination, as appropriate.

2. Methodological Architecture

CoPO algorithms follow a two-stage learning paradigm:

• Stage 1: Cognition-aware Supervised Fine-Tuning (SFT), where the model is exposed to expert-annotated traces (in LLMs: with different cognitive level tags; in MARL: with various social value parameterizations).
• Stage 2: Policy optimization using a modified reinforcement learning objective that enables efficient, adaptive cognitive allocation.

Step-Level and Dual-Process Instantiations

LLM-based CoPO variants (Yang et al., 13 Feb 2026, Cheng et al., 22 May 2025) adopt either a multi-level cognitive hierarchy or a dual-process (“fast/slow thinking”) paradigm, using explicit annotation in the action space (e.g., <fast_think>...<slow_think> tokens or structured [level, rationale, action] triplets). At each decision point, the policy can select the appropriate cognitive level or reasoning strategy, driven by online estimates of task difficulty or local agent confidence.

In multi-agent CoPO (Peng et al., 2021), each agent samples a local coordination factor $\phi_i$ from a learned distribution, establishing a context-specific weighting between self-reward and the mean neighborhood reward. This enables emergent, adaptive trade-offs between cooperation and competition within dynamic social contexts.

3. Training and Optimization Procedures

The RL stage of CoPO leverages trajectory collection, group-wise normalization, and importance correction, integrated into a clipped PPO or GRPO-style surrogate loss. Key procedural details:

• Advantage Calculation: Step-level or group-level advantages are computed for each cognitive trace, normalized within the batch. In LLM settings, this is further refined by weighting each cognitive-level variant at each step by a confidence score, such as the mean log-probability assigned to the next action conditioned on the rationale variant.
• Reward Composition: Composite rewards balance task correctness (e.g., answer accuracy), cognitive or social efficiency (e.g., token-length budget or neighborhood coordination), and explicit reasoning pattern regularization (encouraging use of appropriate cognitive levels).
• Adaptive Allocation: Online task difficulty is estimated (e.g., by sampling a batch of candidate rollouts and computing success rates), and token budgets or cognitive priors are adjusted accordingly.

A typical CoPO RL iteration involves: collecting a batch of trajectories, expanding successful steps to generate all cognitive-level variants, assigning confidence-weighted advantages, and optimizing the PPO-style loss via Adam. In the MARL setting, meta-gradients are used to update the distribution parameters for the agent-level coordination factor.

4. Experimental Results and Benchmarking

CoPO demonstrates substantial improvements over baselines in both LLM and MARL contexts.

LLM Reasoning Agents (Yang et al., 13 Feb 2026, Cheng et al., 22 May 2025):

• On ALFWorld, Qwen2.5-7B with CoPO achieved a 92.5% success rate versus 83.5% for GRPO, while reducing token usage by ~65%.
• In multi-step math reasoning, CoPO reduces token counts by up to 70% with only minor (<3%) drops in answer accuracy, tripling the accuracy-efficiency measure ACU.
• Cognitive depth allocation is found to be highly adaptive: for easy examples, shallow reasoning dominates; for hard examples, deep or slow reasoning is upweighted.
• Ablations confirm that average log-prob confidence weighting is critical for correct depth assignment and that supervised cold-start is essential to avoid mode collapse.

Multi-Agent Coordination (Peng et al., 2021):

• In traffic simulation, CoPO achieved higher success and safety rates than Independent PPO, Mean-Field PPO, and Curriculum Learning across diverse scenarios.
• Emergent behaviors such as queuing, yielding, negotiation, and lane-switching were observed, linked directly to the adaptation of the social value orientation.
• Quantitatively, CoPO agents reach goals more often, with improved efficiency and fewer collisions in all tested environments.

5. Theoretical Insights and Ablation Studies

The fundamental insight of CoPO is the alignment of cognitive or social resource allocation with real-time task demands through local credit assignment and explicit reward shaping.

• For LLMs, ablative removal of confidence-based step-level reweighting leads to collapse into either shallow or deep reasoning modes, confirming the necessity of step-wise modulation.
• For MARL, the meta-gradient adaptation of the LCF enables the system to discover context-specific social strategies, which are robust to changes in agent density and policy mixture.
• The use of composite and softly enforced efficiency constraints, rather than hard length clipping or static penalties, leads to superior efficiency-accuracy tradeoff and interpretable behavioral traces.

6. Comparative Analysis and Related Approaches

CoPO generalizes previous resource allocation and hybrid policy schemes by unifying reasoning depth, efficiency, and social coordination into the policy objective itself. Standard PPO/GRPO methods either under- or over-allocate cognitive resources and lack adaptation to local difficulty. Fixed-length or static penalty schemes do not accommodate the variability of cognitive needs.

Related algorithms such as GRPO, THINKPRUNE, DAST, and SimPO do not jointly optimize for adaptive cognitive allocation and efficiency, nor do they provide transparent and interpretable traces of reasoning effort.

7. Interpretability and Practical Implications

The explicit reasoning-mode tokens in LLM CoPO agents enable post hoc inspection and attribution of cognitive strategies per example and per step. The fine-grained cognitive trace facilitates debugging, interpretability, and possible integration into systems requiring explainability guarantees.

In multi-agent settings, the emergent behaviors under CoPO suggest utility in domains where dynamic teamwork, negotiation, or social reasoning is critical, such as autonomous driving or swarm robotics.

A plausible implication is that cognition-aware policy optimization will enable future agents to optimize both performance and resource efficiency, adapting reasoning effort on a per-task, per-interaction basis with high transparency and robustness to distributional shift.

---

References:

• "Think Fast and Slow: Step-Level Cognitive Depth Adaptation for LLM Agents" (Yang et al., 13 Feb 2026)
• "Incentivizing Dual Process Thinking for Efficient LLM Reasoning" (Cheng et al., 22 May 2025)
• "Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization" (Peng et al., 2021)