CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMs

Abstract: We introduce CalBench, a controlled evaluation environment for studying multi-agent coordination through calendar scheduling. In CalBench, N agents each manage a private calendar containing pre-existing commitments and must coordinate to schedule a stream of M incoming meetings while minimizing disruption costs. Because agents observe only their own calendars, successful scheduling requires communication across private information boundaries. Each scenario is generated with an oracle solution, enabling precise measurement of coordination quality via realized-to-optimal cost, as well as a Distributed Constraint Optimization (DCOP) baseline to provide a fair comparison under the same private-information constraints. CalBench enables precise verification of task success, communication efficiency, and fairness in the distribution of disruption costs. Our environment also studies privacy-preserving coordination by augmenting calendar entries with private semantic contexts of varying sensitivity and measuring whether agents reveal task-irrelevant private information during negotiation. Unlike multi-agent benchmarks where a single capable agent can often substitute for the group, CalBench is inherently decentralized: no agent has access to another agent's private calendar, yet agents must still reach mutually consistent decisions over shared meeting scheduling. CalBench therefore provides a practical and verifiable setting for studying coordination protocols, communication efficiency, negotiation strategies, fairness, and privacy leakage in multi-agent systems.

• The paper introduces CalBench, a benchmark that evaluates decentralized multi-agent coordination in LLM scheduling under strict privacy constraints.
• It employs controlled scenario generation with fine-grained metrics, including the Valuation of Possible States (VPS), to measure coordination cost and information leakage.
• Empirical results demonstrate that achieving optimal scheduling efficiency requires precise cost information exchange while mitigating sensitive semantic disclosures.

CalBench: Formal Analysis of Coordination-Privacy Trade-offs in Decentralized Multi-Agent LLM Scheduling

Problem Formulation and Motivations

CalBench introduces a rigorous benchmark for decentralized multi-agent coordination in the calendar scheduling setting. Unlike prior works where a central agent with global state is sufficient or where agent architectures can collapse the multi-agent game to monolithic reasoning, CalBench isolates the fundamental requirement for communication and negotiation under strict privacy constraints. Each LLM agent operates with local information—its own calendar populated with events of variable cost and private semantic context—and must jointly schedule meetings with globally consistent decision making. The setup rigorously enforces information asymmetry: feasible solutions require dynamic exchange of information, but privacy must be maintained so that agents do not indiscriminately leak sensitive calendar context.

This task formulation addresses longstanding open issues in multi-agent LLM evaluation. Existing agentic benchmarks either (1) do not enforce necessary multi-agent behavior, (2) do not admit precise reward computation or oracle bounds, or (3) lack a verifiable privacy axis. CalBench overcomes these gaps with scenario generation regimes featuring known-optimal (oracle) solutions, cost-aware and privacy-aware baselines, and a suite of fine-grained metrics, including the Valuation of Possible States (VPS) for quantifying information leakage.

Methodology

Scenario Distribution and Game Protocol

CalBench generates controlled scenarios with $N$ agents, private calendars of $T$ slots, and a stream of $M$ incoming meetings. Calibration ensures balanced difficulty and always guarantees the existence of a feasible solution. Each calendar slot is initialized as free, an errand (possibly with variable displacement cost), or a prior meeting. Errands are annotated with naturalistic, privacy-sensitive labels from a semantically tiered label bank (public, neutral, sensitive).

The protocol proceeds in four sequential phases per meeting: (1) participant "cheap-talk" negotiation; (2) voluntary rescheduling by contacted non-participants; (3) local atomic action batches for scheduling and rescheduling; (4) environment-level resolution for consensus and constraint satisfaction. Explicit system-level constraints prohibit agents from communicating the private semantic context of their calendar items.

Scripted Baselines

Three classes of distributed constraint optimization baselines are implemented for privacy/efficiency comparison:

• IMAP: High-disclosure, near-optimal coordination via full cost vector exchange per slot.
• DSM (Welfare/Private variants): Quantized score-based negotiation, trading off privacy for solution quality via offer set cardinality.
• SD-MAP: Minimal binary feasibility exchange, minimizes leakage at the expense of potentially catastrophic cost inefficiency.

All baselines operate exclusively with typed (non-natural-language) messages, structurally barring private semantic context leakage.

LLM Agents and Privacy Auditing

Seven leading LLM agents (Claude Sonnet 4.6, DeepSeek V4 Pro, GPT-5.4 Mini, Gemini 3.1 Pro, Gemini 3 Flash, Llama 4 Maverick, Qwen 3.6 Plus) were evaluated under both standard and adversarial conditions (the latter deploying a nosy agent actively probing for private information). Deployed prompts explicitly reinforce constraints on privacy and negotiation.

Privacy leakage is operationalized and quantitatively measured via two main axes:

• Structural/VPS: How much slot-level availability information is revealed, as per Maheswaran et al. (2006).
• Semantic: Leakage of sensitive event descriptions, detected via targeted keyword/matching.

Empirical Results

Coordination/Cost-Privacy Trade-off

Uniform-cost scenarios are trivial with respect to privacy: all well-configured agents schedule nearly all meetings successfully with minimal excess cost. This masks the true challenge of the problem.

Varied-cost tasks induce a pronounced privacy-optimization frontier:

• Gemini 3.1 Pro consistently achieved near-optimal scheduling (mean excess cost 12.83 versus IMAP's 20.28) with moderate communication and the lowest semantic-leak rate among LLMs.
• GPT-5.4 Mini and Llama 4 Maverick, by contrast, exhibited qualitative cost flattening—vague language that fails to communicate large cost differentials—leading to extremely high excess costs (1159.17 and 624.39) and severe unfairness (cost concentrated on few agents).
• DSM-PRIVATE serves as a high-privacy, low-coordination anchor, while IMAP delineates the other extreme.

Statistical mixed-effects modeling confirms the privacy-efficiency trade-off is significant only in the varied-cost setting ($\beta_1 = -0.063$, $p = 0.006$): lowering excess coordination cost necessitates high information disclosure.

Message Volume, Negotiation Structure, and Robustness

Surprisingly, communication volume does not correlate with improved coordination. Optimal agents achieve both low excess cost and fairness with parsimony in communication, provided cost-related content is granular and precise. For instance, Gemini 3.1 Pro achieves optimal or near-optimal outcomes with only marginally higher DM count than GPT-5.4 Mini, which consistently fails in cost minimization due to vague negotiation.

Blocked calendar experiments demonstrate that agents must robustly distinguish hard constraints. Gemini 3.1 Pro achieves robust consistency, whereas Llama 4 Maverick frequently violates constraint tracking, leading to large performance degradations when some events (errands or meetings) are immovable.

Privacy Auditing and Leakage

Adversarial probing induces few but non-negligible semantic leaks, even in well-aligned models. In uniform-cost scenarios, leaks are infrequent and generally low-sensitivity (e.g., public errands). In varied-cost tasks, contextual negotiation pressure impels agents—especially Gemini 3 Flash—to reveal high-sensitivity event descriptions (“bankruptcy filing preparation”, “legal settlement”, “external journalist”) as justification for inflexible or expensive slots. This demonstrates that privacy failures are not exclusively prompt-following lapses but can be structurally induced by protocol requirements conflicting with cost explanation.

Trace-level analysis across 36,858 DMs exposes the dominant emergent pattern as "availability gossip"—agents relaying learned availability of third parties, violating intended privacy boundaries. Semantic label disclosures are rare but, when present, are mostly high-sensitivity in varied-cost adversarial runs.

Theoretical Implications

CalBench provides evidence supporting two strong claims:

• Feasibility and cost efficiency are orthogonal optimization strata in decentralized multi-agent LLM scheduling. Vague negotiation language enables agents to find feasible but highly inefficient solutions—the cost-awareness protocol must be explicitly and robustly embedded.
• The classical privacy-efficiency trade-off persists and is now quantifiable in LLM-dominated coordination environments. Unlike structured DCOP protocols, LLM agents operate in natural language, introducing nontrivial vectors for semantic context leakage unobservable in output-only audits.

Furthermore, the analysis shows that negotiation disposition (pushback/acquiescence balance) has limited predictive power for task success unless coupled with precise cost signal exchange.

Practical Implications and Future Directions

The presented framework and dataset enable systematic investigation of scalable, privacy-preserving, cost-sensitive multi-agent communication strategies. Practically, the strong privacy-leakage vectors indicate that deploying generalist LLMs as autonomous user-facing agents in real-world scheduling or resource coordination remains problematic, particularly in the presence of adversaries or under cost pressure. Explicit protocol design and prompt engineering are necessary but insufficient: structural solutions for controlled information exchange must be co-designed with natural language agents.

The modular scenario harness, VPS analysis pipeline, and open-sourced benchmarks and traces render this work easily extensible. Immediate future directions include scaling agent counts, longer-term horizon coordination, strategic misreporting scenarios (beyond scope here), and porting the methodology to other decentralized optimization domains (e.g., supply chain, distributed resource allocation). Investigating how selective disclosure mechanisms and advanced negotiation scaffolds might relax the observed privacy-utility frontier remains open.

Conclusion

CalBench rigorously demonstrates that decentralized multi-agent LLM coordination with private utilities enforces a quantifiable privacy-efficiency trade-off, in direct continuity with classical DCOP scheduling. Coordination quality is dominated not by message volume or superficial negotiation stance but by the agents’ capacity to exchange sufficient, relevant cost information while respecting privacy boundaries. Semantic context leakage is provoked by cost-justification demands and cannot be eliminated by prompt constraints alone. CalBench enables deeper benchmarking and scientific inquiry into safe, efficient, and privacy-preserving agentic collaboration at scale.

Reference: "CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMs" (2605.09823)