Cooperative Plan Optimization (CaPo)

• Cooperative Plan Optimization (CaPo) is a suite of algorithmic frameworks that coordinates multi-agent systems to optimize interdependent plans under shared constraints.
• The CAPO framework leverages consensus aggregation, decomposing signal from waste to enhance policy optimization and improve scalability across complex tasks.
• Empirical studies in robotics, multi-agent path finding, and autonomous systems demonstrate significant gains in efficiency, safety, and performance over traditional methods.

Cooperative Plan Optimization (CaPo) encompasses a suite of algorithmic frameworks and methodologies for orchestrating coordinated action among multiple agents or plan components, with the aim of globally optimizing coupled objectives under shared constraints. CaPo arises in diverse domains, including multi-agent reinforcement learning, path planning, combined combinatorial optimization, robot-human interaction, multi-module AI pipelines, and coordinated control of cyber-physical systems. Central to CaPo is the simultaneous optimization of interdependent plans—spatial, temporal, or policy-driven—by leveraging joint information, consensus aggregation, and explicit or implicit credit assignment for component contributions. Recent work highlights both theoretical guarantees and scalable empirical performance across these settings.

1. Signal-Waste Decomposition and the CAPO Framework

Traditional on-policy policy optimization methods such as Proximal Policy Optimization (PPO) face an optimization-depth dilemma: running multiple SGD epochs per batch increases first-order (natural gradient) signal improvement only up to a point, after which path-dependent divergence (dubbed "waste"—Fisher-orthogonal to the desired update direction) grows linearly with depth, degrading trust region adherence and ultimately, policy improvement. This decomposition is formalized as: $$\Delta = c\,\hat d + \epsilon, \qquad \hat d = \frac{F^{-1}g}{\|F^{-1}g\|_F}$$ with $c\,\hat d$ the natural-gradient signal and $\epsilon$ the excess waste. Empirically, the signal term saturates early, while the waste term grows unchecked, leading to policy collapse with excessively deep optimization (Su et al., 13 Mar 2026).

The Consensus Aggregation for Policy Optimization (CAPO) approach reallocates computation from depth to width: $K$ independent PPO replicas, each initialized identically but shuffling minibatch order, optimize on the same sampled batch. The resulting $K$ policies are aggregated to yield a consensus. Consensus can be performed either in Euclidean parameter space (CAPO-Avg) or in the natural parameter space of exponential-family policies via the logarithmic opinion pool (LogOP). In natural parameter space, consensus yields provable KL penalties and first-order surrogate improvements superior to the component experts.

Key theoretical results include:

• The consensus preserves the average linearized surrogate, reduces KL cost (Jensen's inequality), and improves the KL-penalized surrogate step,
• Trust region compliance is stricter for the consensus than for the mean expert,
• Waste is canceled efficiently, while first-order signal is preserved.

On continuous-control benchmarks, CAPO (LogOP consensus) achieves up to $8.6\times$ PPO return under fixed sample budgets; waste reduction is up to $46\%$ in high-dimensional tasks (Su et al., 13 Mar 2026).

2. Cooperative Plan Optimization in Multi-Agent Path Finding and Coordination

In multi-agent path finding (MAPF), cooperative plan optimization addresses not just collision avoidance, but explicit coordination for composite team tasks. The Cooperative Multi-Agent Path Finding (Co-MAPF) formalism requires paired agent types (e.g., grasp and transfer units) to rendezvous at jointly selected meeting points and fulfill composite tasks assigned as ordered pairs. The Cooperative Conflict-Based Search (Co-CBS) algorithm explicitly decouples the planning of agent meetings from pathfinding by:

• Enumerating meeting-time/location tuples (the "meetings level"),
• Solving classic CBS on individual path constraints within each meeting schedule,
• Deferring the expansion of roots in the meeting-tree forest until required.

Co-CBS guarantees optimality and completeness for well-formed instances and empirically achieves $>90\%$ success rates on large warehouse benchmarks. The algorithm is extensible to variable agent coalitions, dynamic tasks, heterogeneous agent types, and richer cooperative settings (Greshler et al., 2021).

3. Meta-Planning and Progress-Adaptivity in LLM-Driven Embodied Agents

In embodied multi-agent cooperation with LLM-based agents, CaPo introduces explicit meta-plan generation and progress-adaptive meta-plan execution:

• Meta-plan generation: Agents engage in multi-round discussion to produce a structured, long-horizon plan which partitions the task into subtasks, assigns responsibilities, and optimally orders steps; consensus is reached via iterative LLM-based critique and refinement.
• Progress-adaptive execution: Agents monitor for significant events (e.g., new object discovered, subtask complete) and trigger targeted meta-plan adaptation discussions to coherently update the joint plan.

Each agent's architecture encompasses modules for perception, memory, LLM-wrapped communication, cooperative meta- and replanning, plan parsing, and action execution. Experiments on ThreeDWorld Multi-Agent Transport and Communicative Watch-And-Help show CaPo achieves $84\%$ task completion (vs $72.5\%$ for standard LLM baselines) and reduces required steps by $c\,\hat d$0 (Liu et al., 2024). Ablations confirm that both strategic meta-planning and progress-adaptivity are essential for efficiency. Limitations include reliance on high-quality LLMs and perceptual accuracy.

4. Cooperative Credit Assignment in Sequential Teams

In sequential cooperative teams sharing a common reward, efficient local learning signals depend on decomposing group rewards into optimally "factored" per-agent utilities. CAPO (Counterfactual Advantage Policy Optimization) formalizes this via the Sequential Aristocrat Utility (SeqAU), which maximizes statistical learnability of each agent's contribution under fixed execution order. The per-agent advantage is: $c\,\hat d$1 Practically, CAPO fits an additive reward decomposition from off-policy trajectories and performs fictitious sampling for downstream agent contributions, yielding closed-form, critic-free advantages and policy gradients. Analytical bounds establish flat advantage variance in team size $c\,\hat d$2, and empirical results show superior sample efficiency for large sequential teams and multi-LLM pipelines (Deshmukh et al., 20 Apr 2026).

5. Combinatorial and Hybrid CaPo: The Travelling Thief Problem

The travelling thief problem (TTP) integrates TSP (tour selection) and knapsack (item selection) subproblems with strong interdependencies. Standard solvers alternate optimizing each component with the other fixed, which impedes global optimality. The CaPo framework for TTP coordinates combinatorial plan optimization via:

• An augmented 2-OPT operator (PGCH) that simultaneously reverses tour segments and updates the collection plan, dropping low-profit and adding high-profit items along the reversed segment,
• A boundary-restricted bit-flip search mediating plan adjustment for only profit-boundary items, reducing search overhead and converging to improved composite solutions.

Across three standard benchmark categories, this CaPo solver sets new best gains in up to $c\,\hat d$3 of large instances and outperforms prior work on the Relative Deviation Index (Namazi et al., 2019).

6. Cooperative CaPo in Physical and Cyber-Physical Systems

Cooperative plan optimization extends naturally to multi-robot and multi-vehicle systems. In human-aware robot navigation, joint trajectory optimization over both robot and human paths, with social constraints (proxemics, time-to-collision, directionality), yields legible, safe navigation in real and simulated environments. The optimization is cast as a nonlinear least-squares problem over a trajectory factor graph, solved in real time, with zero collision violations and superior comfort scores over purely reactive methods (Khambhaita et al., 2017).

For connected autonomous vehicles in constrained environments (e.g., roundabouts), CaPo is realized by parallel distributed optimization over convexified, hard-constrained OCPs. Reference waypoint updates determine passing sequence implicitly, while dual-consensus ADMM enables fast, per-agent local solves. This approach achieves real-time operation (sub-10ms for $c\,\hat d$4), strict safety, and major velocity improvements over rule-based and standard nonlinear programming baselines (Huang et al., 2023).

7. Generalizations, Limitations, and Directions

Cooperative Plan Optimization methods share the unifying principle of actively coordinating or aggregating candidate plans under common information to neutralize noise or suboptimality induced by path/agent-dependent variance while preserving essential first-order optimization signal. This paradigm admits further extensions:

• Multi-agent parallelization for broader settings and policy classes,
• Stronger diversity induction and Bayesian aggregation,
• Application to structured policies (discrete/continuous/graphical models),
• Integration with LLM fine-tuning under trust-region and natural-gradient constraints.

Limitations include increased computational load (though often embarrassingly parallel), dependency on accurate world or perceptual models, and, in some cases, the challenge of extending analytic guarantees under non-additive or nonconvex objectives. Future work targets efficient real-time adaptation, learned triggers for replanning, and scalable credit decomposition in complex, long-horizon multi-module pipelines.

---

Principal sources: (Su et al., 13 Mar 2026, Liu et al., 2024, Deshmukh et al., 20 Apr 2026, Greshler et al., 2021, Namazi et al., 2019, Khambhaita et al., 2017, Huang et al., 2023).