Soft-Label Governance for Distributional Safety in Multi-Agent Systems

Abstract: Multi-agent AI systems exhibit emergent risks that no single agent produces in isolation. Existing safety frameworks rely on binary classifications of agent behavior, discarding the uncertainty inherent in proxy-based evaluation. We introduce SWARM (\textbf{S}ystem-\textbf{W}ide \textbf{A}ssessment of \textbf{R}isk in \textbf{M}ulti-agent systems), a simulation framework that replaces binary good/bad labels with \emph{soft probabilistic labels} $p = P(v{=}+1) \in [0,1]$, enabling continuous-valued payoff computation, toxicity measurement, and governance intervention. SWARM implements a modular governance engine with configurable levers (transaction taxes, circuit breakers, reputation decay, and random audits) and quantifies their effects through probabilistic metrics including expected toxicity $\mathbb{E}[1{-}p \mid \text{accepted}]$ and quality gap $\mathbb{E}[p \mid \text{accepted}] - \mathbb{E}[p \mid \text{rejected}]$. Across seven scenarios with five-seed replication, strict governance reduces welfare by over 40\% without improving safety. In parallel, aggressively internalizing system externalities collapses total welfare from a baseline of $+262$ down to $-67$, while toxicity remains invariant. Circuit breakers require careful calibration; overly restrictive thresholds severely diminish system value, whereas an optimal threshold balances moderate welfare with minimized toxicity. Companion experiments show soft metrics detect proxy gaming by self-optimizing agents passing conventional binary evaluations. This basic governance layer applies to live LLM-backed agents (Concordia entities, Claude, GPT-4o Mini) without modification. Results show distributional safety requires \emph{continuous} risk metrics and governance lever calibration involves quantifiable safety-welfare tradeoffs. Source code and project resources are publicly available at https://www.swarm-ai.org/.

• The paper introduces SWARM, a novel framework that uses soft probabilistic labels for nuanced measurement of toxicity and welfare trade-offs in multi-agent systems.
• The methodology replaces binary classifications with continuous expected-value computations and incorporates regulatory interventions like transaction taxes and circuit breakers.
• Experimental results reveal that strict governance significantly reduces welfare without lowering toxicity, highlighting a cost paradox in AI regulation.

Soft-Label Governance for Distributional Safety in Multi-Agent Systems

Introduction

The paper "Soft-Label Governance for Distributional Safety in Multi-Agent Systems" introduces a novel framework, SWARM, which addresses distributional safety in multi-agent environments. Traditional AI safety frameworks often employ binary classifications of agent behavior, overlooking the inherent uncertainties. In contrast, SWARM utilizes soft probabilistic labels to quantify agent interactions, allowing for continuous-valued payoff computation and toxicity measurement. This approach facilitates nuanced governance interventions by leveraging a modular governance engine equipped with configurable levers.

Framework and Methodology

SWARM introduces a comprehensive simulation framework that replaces binary labels with soft probabilistic labels, enabling the computation of expected values for payoffs and toxicity metrics. The framework's governance engine includes various levers such as transaction taxes, circuit breakers, and random audits designed to intervene in multi-agent interactions. By adopting probabilistic metrics like expected toxicity and quality gap, SWARM quantifies the impact of these governance interventions on systemic safety and welfare.

Figure 1: SWARM pipeline. Observable signals are converted to a proxy score $\hat{v}$, then to a soft label $p$ via calibrated sigmoid. The payoff engine computes expected surplus and harm.

The proxy evaluation in SWARM converts observable signals into a proxy score $\hat{v}$, which is then transformed into a probabilistic soft label using a calibrated sigmoid function. This process enables the calculation of expected surplus and harm for each agent, allowing governance levers to systematically adjust the interaction costs and access permissions for agents.

Experimental Results

The evaluation of SWARM involved seven different scenarios with varying configurations of agent types and governance regimes. One notable finding is that strict governance does not significantly reduce toxicity levels but instead results in a considerable welfare loss. For example, strict governance achieves the same toxicity as the baseline while reducing welfare by over 40%. This highlights the potential cost of governance when applied without adaptive strategies.

Figure 2: Toxicity and welfare across seven scenarios (error bars: pm1 std). Strict governance achieves identical toxicity to baseline (0.300 vs. 0.300) but reduces welfare by 40.2%.

In adversarial scenarios, such as the adversarial red team scenario, there is a collapse in systemic interaction after initial exploitation, resulting in significantly reduced cumulative welfare. On the other hand, scenarios like the threshold dancer exploit governance boundaries to achieve high systemic activity and welfare, albeit with elevated toxicity levels.

Figure 3: Risk-Welfare Pareto Frontier. A scatter plot of mean welfare against mean toxicity across simulated governance scenarios in SWARM.

Ablations on externality internalization revealed that while toxicity remains largely invariant, welfare decreases with increasing internalization parameters. This indicates that the redistribution of harm costs alone does not enhance safety unless paired with adaptive mechanisms that can dynamically respond to governance signals.

Discussion

The results suggest several key insights into governing multi-agent systems. First, the governance cost paradox indicates that regulatory interventions can impose welfare costs without corresponding safety improvements. Effective governance requires balancing systemic welfare and distributional safety, exploring optimal Pareto frontiers rather than pursuing rigid thresholds. The study also demonstrates the importance of continuous over binary safety metrics in capturing nuanced strategic interactions like threshold dancing.

Validation experiments extended SWARM's utility beyond scripted agents to complex LLM-backed agents, demonstrating its adaptability to real-world applications. However, limitations exist, such as reliance on simplified proxy evaluations and static agent responses, which could affect real-world applicability.

Conclusion

SWARM presents a significant advancement in AI safety by employing soft-label probabilistic metrics to evaluate distributional risk in multi-agent systems. The framework provides a robust mechanism for understanding the trade-offs between safety and system welfare, offering insights into the effects of governance interventions. Future work will focus on enhancing adaptive agent strategies and refining proxy evaluation models to better simulate real-world conditions.