CIEN: Scalable Multiagent Coordination

• CIEN is a scalable architecture for multiagent reinforcement learning that estimates collective influence from the task object's state, removing the need for explicit communication.
• It redefines agent coordination by predicting a low-dimensional aggregate influence, decoupling model complexity from the number of agents.
• Empirical results show CIEN achieves near-centralized performance in both simulated and real robotic tasks while offering robust scalability and reduced computational overhead.

The Collective Influence Estimation Network (CIEN) is an architectural and algorithmic approach for scalable multiagent reinforcement learning (MARL) under strict communication limitations. CIEN is designed to estimate, represent, and leverage the aggregate effect of all non-observed teammates on a shared task object, removing the necessity for explicit state or action exchanges and mitigating the unsustainable computational scaling of conventional Action Estimation Networks (AENs). The fundamental innovation is a low-dimensional "collective influence" computed solely from observable object state, permitting agents to infer latent interaction structure and recover near-centralized coordination performance without inter-agent communications (Luo et al., 13 Jan 2026).

1. Motivation and Conceptual Framework

Standard MARL schemes typically rely on frequent communication of other agents’ states or actions to counteract non-stationarity. AENs mitigate limited observability by explicitly predicting the individual actions $a^{-i}$ of all teammates. However, these networks scale poorly: input/output dimensions and parameters increase with the agent population size $N$, producing significant computational and memory burdens that preclude deployment in large-scale or real robotic systems.

CIEN reconceptualizes the estimation objective. Instead of reconstructing each teammate’s action, CIEN directly predicts the "collective influence" $c^{-i}$—the aggregate effect of all other agents as manifest in the task object’s state $s^o$. Statistically, this influence is both sufficient and minimal for local policy computation in many cooperative robotic tasks. This approach ties the relevant dimensionality to the physical subspace of the object (e.g., 2D displacement), decoupling network size from $N$, and enabling scalable, modular team growth.

2. Mathematical Formulation

Let agent $i$ have local proprioceptive state $s^i$, observe the task object state $s^o$, and remain ignorant of other agents' actions $a^{-i}$. CIEN defines:

• Collective Influence: $c^{-i} \triangleq e_\psi(s^o)\in\mathbb{R}^k$ (typically $k=2$ for position).
• Actor: $\pi_\phi(a^i|s^i, s^o, c^{-i})$.
• Critics: $Q_{\theta_1}$, $Q_{\theta_2}$.

The algorithm employs a Soft Actor-Critic (SAC)-inspired objective:

$$y = r + \gamma \min_{j\in\{1,2\}} Q_{\theta'_j}(s^i_{\text{next}}, s^o_{\text{next}}, a^i_{\text{next}}, c^{-i}_{\text{next}}) + \alpha \log \pi_\phi(a^i_{\text{next}}|s^i_{\text{next}}, s^o_{\text{next}}, c^{-i}_{\text{next}})$$

where $$a^i_{\text{next}} = \pi_\phi(s^i_{\text{next}}, s^o_{\text{next}}, c^{-i}_{\text{next}})$$ and $$c^{-i}_{\text{next}} = e_{\psi'}(s^o_{\text{next}})$$. Critic loss is:

$$L(\theta_j) = \mathbb{E}_{\mathcal{B}} \left[ (Q_{\theta_j}(s^i, s^o, a^i, c^{-i}) - y)^2 \right]$$

Actor and CIEN are updated via entropy-augmented objectives analogous to SAC, with soft target network updates. CIEN is structurally identical to a small actor; its output directly enters the critic and actor forward passes, preserving training dynamics.

3. Network Architecture

Each agent maintains three local networks:

Component	Input	Architecture	Output
CIEN $e_\psi(s^o)$	$s^o\in\mathbb{R}^2$	2 FC layers (128 units), ReLU; Linear+$\tanh$	$c^{-i}\in[-1,1]^2$
Actor $\pi_\phi(\cdot)$	$[s^i, s^o, c^{-i}]$ (10-dim)	2 FC layers (256 units), ReLU; heads for $\mu_\phi$, $\log\sigma_\phi$	$a^i\in[-1,1]^3$ (tanh sampling)
Critic $Q_{\theta_j}$	$[s^i, s^o, a^i, c^{-i}]$ (13-dim)	2 FC layers (256 units), ReLU	Scalar Q-value

Target networks ($\theta'_j$, $\psi'$) share architectural design with frozen, soft-updated weights.

All networks operate independently of $N$, requiring neither parameter expansion nor input/output dimension changes as new agents join. A plausible implication is the capacity for "plug-and-play" team resizing without retrofitting existing agent models.

4. Training Algorithm

Training proceeds for each agent as a parallel instance, without centralized critics, shared action predictions, or networked replay buffers:

1. Initialize all networks and targets.
2. For each episode and timestep:
   • Compute $c^{-i}_t = e_\psi(s^o_t)$.
   • Sample $a^i_t \sim \pi_\phi(\cdot|s^i_t, s^o_t, c^{-i}_t)$.
   • Execute action, observe $r_t, s^i_{t+1}, s^o_{t+1}$.
   • Store transition $$(s^i_t, s^o_t, a^i_t, c^{-i}_t, r_t, s^i_{t+1}, s^o_{t+1})$$ in replay buffer.
3. Sample minibatches, compute SAC targets, and update critics by minimizing $L(\theta_j)$.
4. Periodically update actor ($\phi$) and CIEN ($\psi$) via gradient step, and softly update targets.
5. No additional regularization is employed beyond SAC’s entropy term.

Empirically, training is stable, and networks remain lightweight per agent at all scales.

5. Scalability Characteristics

CIEN's architectural and runtime costs are strictly bounded in agent population size $N$. In contrast to AENs, which estimate all $(N-1)$ teammates’ actions and scale as $O(N)$ in parameters and activations, CIEN networks:

• Accept task object state as sole input ($O(1)$).
• Emit collective influence with fixed output dimension $k$ (typ. 2–3).
• Parameter load is $O(k h + h^2)$ (for hidden size $h$), independent of $N$.

At runtime, each agent processes CIEN, actor, and critic—each $O(1)$ forward pass—contributing constant compute/memory per agent. New agents instantiate their own networks without modification of existing agents’ models.

This suggests "strong scalability," enabling large team deployments without bottlenecking resources or requiring architectural redesign.

6. Empirical Validation

CIEN was validated in a three-arm disk-lifting task using Robosuite simulation:

• System: 3 UR5e arms, 120° apart, jointly lift disk from 0.896m to 1.36m.
• State: $s^i\in\mathbb{R}^6$ (joint angles), $s^o\in\mathbb{R}^2$ (height, tilt).
• Actions: $a^i\in\mathbb{R}^3$ (joint velocities), 4Hz control.
• Reward: sum of $r_{\text{height}}$, $r_{\text{angle}}$, 200 steps max.

Baselines:

• Centralized SAC (20-D state, 9-D action).
• Decentralized SAC without CIEN.

Outcomes:

• Centralized SAC: 10/10 runs converge, final lift at ≈1.36m within 40k episodes.
• Decentralized SAC: fails to converge; suboptimal returns.
• CIEN-SAC: 9/10 runs converge by 40k episodes, final lift at ≈1.36m; matches centralized baseline.
• Performance curves demonstrate near-centralized coordination in communication-free regime.

A plausible implication is that collective influence modeling fully substitutes for explicit action exchange in structured cooperative settings.

7. Real-World Deployment and Robustness

Deployment to hardware involved 3 UR5e arms and object tracking via a calibrated RGB camera and AprilTag/QR-marker array. The object state (height, tilt) is extracted via vision using a PnP algorithm.

Noise profiles included ≈1cm height and 1° tilt estimation errors, resulting in occasional action instability. Robustness was achieved by in-simulation fine-tuning under noisy state inputs (target height relaxed to 1.25m, 8k supplementary episodes).

• CIEN-SAC required ∼4k episodes for stable policy convergence in noisy settings.
• Centralized SAC lagged under equivalent noise.
• Real-world execution: both CIEN-SAC and centralized SAC lifted the disk in ≈12s without physical damage.
• CIEN-SAC produced smoother height and tilt time series, with reduced overshoot/oscillation under vision noise.
• Each arm required only individual joint encoder readings and object vision; no inter-arm communication.

This demonstrates high deployment feasibility of CIEN-based MARL under sensor-limited and bandwidth-constrained conditions.

8. Significance and Implications

CIEN provides an operationally efficient, scalable, and modular solution to the coordination problem in cooperative MARL, particularly for robotic teams with restricted inter-agent bandwidth. By reframing inter-agent estimation as collective influence modeling on the object-level subspace, it sidesteps the key limitations of per-agent action prediction and allows seamless team extension.

A plausible implication is that collective influence–based architectures can achieve near-centralized performance in many cooperative control tasks, provided the task object's state encodes sufficient latent information about team interaction. Extensions may include broader classes of manipulation, locomotion, or multi-object tasks with learned or engineered influence encapsulations.

Further investigation is warranted into conditions where object-centric influence modeling becomes insufficient; for example, strategic adversarial multiagent scenarios or tasks with nonlocal topological dependencies. Nonetheless, CIEN exemplifies a new paradigm for scalable MARL in both simulated and real domains (Luo et al., 13 Jan 2026).