The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaboration (2509.14284v1)

Abstract: As LLMs become integral to multi-agent systems, new privacy risks emerge that extend beyond memorization, direct inference, or single-turn evaluations. In particular, seemingly innocuous responses, when composed across interactions, can cumulatively enable adversaries to recover sensitive information, a phenomenon we term compositional privacy leakage. We present the first systematic study of such compositional privacy leaks and possible mitigation methods in multi-agent LLM systems. First, we develop a framework that models how auxiliary knowledge and agent interactions jointly amplify privacy risks, even when each response is benign in isolation. Next, to mitigate this, we propose and evaluate two defense strategies: (1) Theory-of-Mind defense (ToM), where defender agents infer a questioner's intent by anticipating how their outputs may be exploited by adversaries, and (2) Collaborative Consensus Defense (CoDef), where responder agents collaborate with peers who vote based on a shared aggregated state to restrict sensitive information spread. Crucially, we balance our evaluation across compositions that expose sensitive information and compositions that yield benign inferences. Our experiments quantify how these defense strategies differ in balancing the privacy-utility trade-off. We find that while chain-of-thought alone offers limited protection to leakage (~39% sensitive blocking rate), our ToM defense substantially improves sensitive query blocking (up to 97%) but can reduce benign task success. CoDef achieves the best balance, yielding the highest Balanced Outcome (79.8%), highlighting the benefit of combining explicit reasoning with defender collaboration. Together, our results expose a new class of risks in collaborative LLM deployments and provide actionable insights for designing safeguards against compositional, context-driven privacy leakage.

• The paper demonstrates the emergence of compositional privacy leakage where adversaries aggregate benign responses to infer sensitive data.
• It evaluates defense strategies, showing that Theory-of-Mind improves sensitive blocking at the cost of benign utility while CoDef offers a superior privacy-utility trade-off.
• Experiments with models like GPT-5 in controlled, multi-agent scenarios reveal that coordinated defenses are essential for mitigating emergent privacy risks.

Compositional Privacy Leakage in Multi-Agent LLM Systems: Risks and Defenses

Introduction and Problem Formulation

This work systematically investigates a novel privacy risk in multi-agent LLM deployments: compositional privacy leakage. Unlike traditional memorization or direct inference attacks, compositional leakage arises when an adversary aggregates individually innocuous outputs from multiple agents to infer sensitive information that is not explicitly revealed by any single agent. This risk is particularly salient in distributed, modular LLM-based systems where agents are assigned specialized roles and possess disjoint, non-sensitive data fragments.

Figure 1: Illustration of how individually innocuous information shared across multiple agents can be aggregated by an adversary to infer sensitive or private data not explicitly revealed by any single agent, highlighting the emergent privacy risks in collaborative multi-agent LLM settings.

The paper formalizes this threat model by considering a set of defender agents, each with a unique, non-overlapping subset of structured data. The adversary, with black-box query access, issues sequential queries to these agents, aiming to reconstruct a global sensitive attribute $s^*$ that is only inferable through the composition of multiple responses. The framework abstracts away from LLM-specific parameters, focusing on inference-time privacy threats in realistic, siloed multi-agent environments.

Evaluation Framework and Metrics

To quantify compositional leakage, the authors construct a controlled evaluation framework with synthetic multi-agent scenarios. Each scenario is designed such that the sensitive attribute $s^*$ is derivable only through multi-step adversarial composition, not from any single agent's data. The adversary is provided with an optimal plan $P^*$, ensuring that leakage is not limited by planning ability but by the information flow and defense mechanisms.

Leakage is measured via two primary metrics:

• Leakage Accuracy: Whether the adversary's final prediction matches the ground truth sensitive attribute.
• Plan Execution Success: Whether all intermediate steps in the adversary's plan are successfully executed, distinguishing between failures in information elicitation and failures in reasoning or composition.

The interaction is modeled as a POMDP, capturing the partial observability of both adversary and defenders, and the sequential, interactive nature of the attack.

Defense Mechanisms

The paper proposes and evaluates two principal defense strategies:

1. Theory-of-Mind (ToM) Defense: Each defender agent simulates the adversary's knowledge state and intent, blocking queries that appear to enable sensitive inferences through adversarial aggregation. This approach requires explicit modeling of adversarial goals and the evolution of the adversary's knowledge.
2. Collaborative Consensus Defense (CoDef): Defender agents share their local query-response histories and vote on whether a query is safe to answer. A single defender's decision to block is sufficient to deny the query, enabling collective mitigation of compositional leakage while preserving benign functionality.

   Figure 2: Overview of the defense mechanisms tested, highlighting differences in information flow and response strategies.

Additional baselines include Chain-of-Thought (CoT) prompting, CoT with sensitive set awareness, and self-voting (multiple samples from a single defender's policy).

Experimental Results

Experiments are conducted across 119 adversarial and benign scenarios using Qwen3-32B, Gemini-2.5-pro, and GPT-5 as defender models. The results reveal several key findings:

• CoT Baselines Are Insufficient: Simple CoT prompting blocks only 31–39% of sensitive queries for Qwen3-32B and Gemini-2.5-pro, while maintaining high benign success (64–76%). This demonstrates that step-by-step reasoning alone does not reliably prevent compositional leakage.
• ToM Defense Substantially Improves Sensitive Blocking: ToM raises sensitive blocking rates to 89–97% across models (e.g., Qwen3-32B: 88.8%, Gemini-2.5-pro: 96.6%, GPT-5: 94.2%). However, this comes at a significant cost to benign utility, with benign success dropping to 52.9–61.6%.
• CoDef Achieves the Best Privacy-Utility Trade-off: CoDef blocks 86–90% of sensitive queries while maintaining higher benign success (66.3–70.2%), yielding the highest balanced outcome (up to 79.8% for Gemini-2.5-pro). This demonstrates the benefit of collaborative, ensemble-style defense.
• Model Capacity Matters: Larger models (GPT-5) achieve higher sensitive blocking even with simpler strategies, suggesting that model capacity and reasoning ability influence defense effectiveness.

Analysis of Reasoning Depth and Defense Efficacy

The paper analyzes the relationship between defender reasoning depth and compositional privacy mitigation. Reasoning depth is categorized from direct answers/refusals (Depth 0) to multi-step, cross-agent reasoning (Depth 3+). Advanced defenses (ToM, CoDef) elicit deeper reasoning, with substantial increases in Depth 2 and Depth 3+ responses, indicating richer adversarial intent modeling and collaborative inference.

Figure 3: Reasoning depth across the different settings for Gemini.

Plan execution analysis further shows that for benign queries, execution rates remain high across all methods, while for sensitive queries, execution drops sharply under ToM and CoDef, indicating effective blocking of adversarial plans without unduly hindering benign functionality.

Figure 4: Percentage of the times the adversary successfully elicits what it intended for benign and sensitive queries across different defense strategies for two models, highlighting the privacy-utility trade-off.

A qualitative example illustrates how an adversary can aggregate employee flight, payment, and personal data to identify an individual, and how ToM and CoDef defenses block sensitive steps, preventing full compositional leakage.

Figure 5: Qualitative illustration of multi-step adversarial inference and defenses, showing how ToM and CoDef block sensitive steps and prevent compositional leakage.

Implications and Future Directions

This work exposes a critical, underrecognized class of privacy risks in multi-agent LLM deployments. The results demonstrate that privacy cannot be guaranteed by isolated, single-agent safeguards or by standard privacy-preserving practices such as differential privacy or input sanitization. Instead, robust protection requires explicit modeling of adversarial strategies, collaborative reasoning, and coordination among agents.

The findings have significant implications for the design of LLM-based multi-agent systems in enterprise, healthcare, and other sensitive domains. As LLMs are increasingly deployed in distributed, modular architectures, compositional privacy leakage must be considered a first-class threat. The demonstrated efficacy of collaborative defenses suggests that ensemble-style, consensus-based mechanisms are a promising direction for future research.

Further work is needed to:

• Develop scalable, automated methods for adversarial intent modeling and collaborative defense in large, heterogeneous agent networks.
• Formalize compositional privacy guarantees and integrate them with existing privacy frameworks.
• Explore adaptive, context-aware defenses that dynamically calibrate privacy-utility trade-offs based on task requirements and risk profiles.

Conclusion

The paper provides a rigorous framework for evaluating and mitigating compositional privacy leakage in multi-agent LLM systems. By formalizing the threat model, constructing controlled evaluation scenarios, and empirically validating defense strategies, it establishes that privacy risks in distributed AI systems are fundamentally emergent and require collective, context-sensitive safeguards. The results underscore the necessity of moving beyond single-agent privacy guarantees toward coordinated, multi-agent defenses that balance utility and privacy in complex, real-world deployments.