Safe Multi-Agent Behavior Must Be Maintained, Not Merely Asserted: Constraint Drift in LLM-Based Multi-Agent Systems

Abstract: Modern LLM based agents are no longer passive text generators. They read repositories, call tools, browse the web, execute code, maintain memory, communicate with other agents, and act through long horizon workflows. This shift moves the unit of safety. A system may produce a compliant final answer while leaking private information through an internal message, delegating authority beyond its original scope, calling an external tool with sensitive context, or losing the evidence needed to reconstruct why an action was allowed. We argue that many emerging failures in LLM-based multi-agent systems share a common structure: safety critical constraints do not remain operative throughout the trajectory. We call this phenomenon constraint drift: the loss, distortion, weakening, or relaxation of constraints as they pass through memory, delegation, communication, tool use, audit, and optimization. The position taken here is that safe multi-agent behavior must be maintained, not merely asserted. Prompts, guardrails, tool schemas, access control, and final output checks are necessary, but they are insufficient unless constraints remain fresh, inherited, enforceable, and auditable across execution. We propose Constraint State Governance as a research paradigm for LLM-based multi-agent systems. In this paradigm, safety-critical constraints are maintained as explicit execution state, while constraint-native reinforcement learning improves utility only within maintained safety boundaries. The goal is not to freeze agentic systems under rigid rules, but to make safety operational across the trajectories through which modern agents actually act.

• The paper introduces constraint drift, highlighting how safety constraints degrade over multi-agent trajectories rather than being static output features.
• It presents Constraint State Governance, using cryptographically signed tokens to ensure constraints are maintained throughout execution.
• Empirical results show that constraint-native reinforcement learning achieves up to 98% admissibility, drastically reducing hidden exposure rates.

Maintaining Safety in Multi-Agent LLM Systems: Constraint Drift and the Case for Constraint State Governance

Introduction

LLM-based agents are evolving from passive text generators into systems that operate on external environments. When deployed as multi-agent frameworks, they read and edit codebases, use external tools, and make stateful modifications through long-horizon workflows. This transition shifts the locus of safety from output-level assertions to the full execution trajectory. The paper "Safe Multi-Agent Behavior Must Be Maintained, Not Merely Asserted: Constraint Drift in LLM-Based Multi-Agent Systems" (2605.10481) articulates the need for safety mechanisms that maintain constraints throughout the trajectory, rather than relying solely on final output checks or localized enforcement.

Constraint Drift: A Systems-Level Safety Failure

The paper introduces constraint drift to describe the degradation or loss of operational constraints across LLM-based agent trajectories. Unlike specification gaming or reward hacking—which stem from misaligned objectives or proxies—constraint drift is a preservation failure: safety-critical constraints degrade as they traverse memory, delegation, tool use, communication, optimization, and audit interfaces. The paper delineates five diagnostic drift modes:

• Memory drift: Constraint loss when context summarization or memory retrieval omits or inaccurately transmits constraints.
• Authority drift: Deviation of delegated authority from the originally intended scope.
• Information flow drift: Leakage of sensitive data through internal channels, tool arguments, or shared memory, not captured at output.
• Accountability drift: Irrecoverable loss of evidence necessary for post-hoc justification or audit of actions.
• Utility-induced drift: Systematic constraint weakening when utility optimization treats constraints as soft objectives.

In agentic environments, these drift modes frequently co-occur and evade detection when safety checks are only output-centric.

Figure 1: Trajectory-level constraint preservation replaces final-answer safety checking; constraint drift is diagnosed at interfaces for memory, authority, information flow, accountability, and optimization.

Limitations of Existing Safety Mechanisms

Present approaches—including prompts, guardrails, tool schemas, access controls, and final output filtering—enforce safety at isolated decision points but fail to guarantee that constraints remain operative throughout multi-agent execution. Benchmarks such as AgentLeak (Yagoubi et al., 12 Feb 2026), PAC-Bench (Park et al., 13 Apr 2026), and Colosseum (Nakamura et al., 16 Feb 2026) expose privacy leakages, authority escalation, and audit failures that output-only checks systematically miss. Agent frameworks (OpenAI Agents SDK, Anthropic MCP, Google A2A, LangGraph) facilitate compositionality, but lack substrate for constraint inheritance, auditability, and enforcement across trajectories. The absence of a persistent, checkable representation of constraints creates room for drift, as constraints can be inaccurately remembered, silently omitted, or reinterpreted post hoc.

Constraint State Governance: Maintaining Operational Force

The central proposal is Constraint State Governance (CSG), a paradigm where safety-critical constraints become first-class execution state throughout the agentic trajectory. Constraints are instantiated as cryptographically signed tokens encapsulating their origin, scope, operational predicate, version, and audit history. Each agent and tool operates over a certified constraint state, not informal recollections.

When a governance-critical action is proposed, an admission control procedure checks:

• The action’s compatibility with the freshest version of each active constraint.
• That delegated capabilities (scoped authorities) are valid and unamplified.
• Information flow is consistent with allowed labeling/budgeting.
• Each permitted action is accompanied by reconstructable and tamper-evident audit evidence.

This process reduces multi-step constraint degradation by binding constraint state, decision evidence, and action realization via signed records. The idealized design assumptions include collision-resistant hashing, unforgeable signatures, and conservative effect abstraction to ensure that any realized action can be justified in post-hoc audit.

Constraint-Native Reinforcement Learning

CSG is fundamentally coupled to reinforcement learning (RL). Instead of using constraints as soft penalties, constraint-native RL only optimizes utility within the admissible set—trajectories that have not violated operational constraints at any point. The reward landscape is shaped such that violation results in inadmissibility, removing drifted trajectories from policy optimization.

$$\mathcal{T}_{\mathrm{adm}}(B_0) = \{\tau: m_j(\tau) \geq \epsilon_j, \forall j\}$$

where $m(\tau)$ quantifies preservation of freshness, authority scoping, information flow, and auditability across the trajectory.

Figure 2: CSG turns static rules into governed execution rollouts; constraint-native RL improves utility within the admissible set, excluding trajectories with drift-induced violations.

The RL signal is thus partitioned: admissible trajectories are ranked for utility/policy improvement; inadmissible ones are violation evidence, not low-quality successes. This separation operationalizes the safety-utility tradeoff: agents cannot “over-perform” by quietly violating constraints.

Empirical Validation and Numerical Results

A case study on the AgentLeak benchmark underscores the failures of output-centric defense regimes. When only output-level filtering is applied, visible answers appear compliant, but internal information exposures via inter-agent messages or shared memory persist at rates above 68%. With “CSG Lite” (minimal constraint governance at all observable channels), hidden exposure rates drop by over an order of magnitude, and every transition is auditable with hash-linked event records.

Policy search experiments further validate constraint-native optimization. Baseline policies using only output filtering fail to achieve fully admissible behavior, with forbidden exposure $\approx 33\%$. Strict policies that over-redact achieve admissibility but at negative utility. Optimally balanced CSG policies achieve $\approx 98\%$ admissibility and near-maximal utility, supporting separation between enforceable safety and utility maximization.

Practical and Theoretical Implications

The CSG paradigm offers a concrete systems layer for trajectory-level assurance. Theoretically, it distinguishes constraint drift from broader agentic risk (miscoordination, collusion) and from classic reward misspecification failure modes. Practically, CSG proposes directly actionable instrumentation: constraint tokens, capability delegation, and tamper-evident audit chains can be engineered into agent infrastructure.

Key implications include:

• Agent safety cannot be established post hoc or via local output checks; safety must be persistent, auditable, and enforced at every critical transition.
• Governing constraints as state enables precise measurement and continuous improvement of constraint encoding, delegation, and repair strategies.
• Constraint-native RL supports robust policy improvements without conflating utility and safety, and reveals structural weaknesses in constraint design or enforcement.
• The paradigm is compatible with hierarchical, context-dependent, or even informally specified constraints, provided operational predicates can be extracted for critical transitions.
• CSG is not a proof of total safety—undecidable cases, incomplete effect abstraction, and adversarial environments remain open threats.

Critical Assessment and Future Directions

A significant research agenda emerges:

• Multi-agent safety benchmarks should make constraint drift observable, with trajectory-level logs capturing failures invisible at the output layer.
• Agent frameworks should natively encode constraint state, delegation, admission evidence, and audit chains.
• Constraint extraction, compilation, and repair require further work in translating vague user instructions and organizational policy into operational state for governance and audit.

Rigidity versus utility remains a tension: not every action merits heavyweight governance. Realistically, governance should target high-risk transitions (external communication, delegation, irreversible changes), while low-risk reasoning incurs lightweight overhead.

Conclusion

The shift from passive LLMs to agentic, tool-using multi-agent systems requires a concurrent shift in safety thinking. Constraints must be maintained—preserved as execution state, actively inherited, explicitly checked, and auditable—rather than merely asserted through prompts and output filters. The CSG paradigm, reinforced by constraint-native RL, closes a core systems gap exposed by the phenomenon of constraint drift. It positions the locus of safety in the execution substrate, ensuring safety is not quietly lost as agents act, communicate, and optimize. This work establishes foundational infrastructure for measurable, operational safety in multi-agent LLM deployments (2605.10481).