CalBench: Multi-Agent Coordination & Privacy Benchmark

• CalBench is a benchmark and simulation environment that models decentralized, privacy-preserving calendar scheduling among multiple agents.
• It features discrete time slots, varied cost settings, and evaluates key metrics such as coordination rate, excess cost, fairness, and privacy leakage.
• The benchmark incorporates DCOP baselines and detailed communication protocols to analyze trade-offs between optimization quality and information disclosure.

Searching arXiv for CalBench and closely related coordination/privacy benchmark papers. I’ll look up the specific arXiv records for CalBench and related benchmark context. CalBench is a benchmark and simulation environment for studying multi-agent LLM coordination under privacy constraints through calendar scheduling. In CalBench, $N$ agents each manage a private calendar containing pre-existing commitments and must coordinate to schedule a stream of $M$ incoming meetings while minimizing disruption costs. Because agents observe only their own calendars, successful scheduling requires communication across private information boundaries. The environment is designed to support precise verification of task success, communication efficiency, fairness in the distribution of disruption costs, and privacy leakage, and it includes an oracle solution for each generated scenario as well as a Distributed Constraint Optimization (DCOP) baseline under the same private-information constraints (Zou et al., 10 May 2026).

1. Environment, scheduling problem, and optimization objective

CalBench models time as discrete slots $S = \{0, 1, \dots, T-1\}$. Each agent $i$ has a private calendar represented as a length-$T$ list of slots, and each slot is either Free, Errand, Meeting, or, in specific variants, blocked. A scenario first chooses a calendar density, then generates a stream of $M$ meetings, and, before populating errands, finds and reserves a hidden feasible “witness” schedule so the task is guaranteed solvable. Each meeting $k$ has a participant set $P_k \subseteq \{1,\dots,N\}$, duration $1$ slot, a private semantic label visible only to participants, and an associated placement cost for each participant (Zou et al., 10 May 2026).

Disruption cost is defined as the cost of displacing existing commitments to accommodate new meetings. In the uniform-cost setting, every errand and meeting has cost $1$. In the varied-cost setting, errands have heterogeneous costs drawn from $M$0, while prior meetings have their own cost scale. If an agent schedules a new meeting into a free slot, the cost is $M$1; if the schedule requires moving an errand or pre-existing meeting, the corresponding displacement cost is paid once. The realized total cost is

$M$2

where $M$3 aggregates costs from all reschedules executed by agent $M$4 across rounds (Zou et al., 10 May 2026).

Each scenario is paired with a full-information oracle built using OR-Tools CP-SAT. The oracle takes the entire meeting set, all calendars, and all costs, and solves a global constraint optimization problem that respects shared-slot consistency and no-double-booking constraints while minimizing total displacement cost $M$5. Evaluation is then based on excess cost,

$M$6

which measures regret relative to the oracle optimum. Because meetings arrive as a stream and scheduled meetings become hard commitments for later rounds unless explicitly rescheduled in variants that allow it, the task is an incremental online scheduling problem rather than a one-shot optimization problem (Zou et al., 10 May 2026).

This formulation makes calendar scheduling a controlled substrate for studying decentralized coordination. The task is inherently multi-agent because no agent has access to another agent’s private calendar, yet all participants in a meeting must write that meeting into the same slot. The combination of local observability, shared feasibility constraints, and explicit disruption costs gives CalBench a verifiable objective structure that is difficult to emulate with loosely specified conversational tasks (Zou et al., 10 May 2026).

2. Decentralization, interaction protocol, and DCOP baselines

CalBench is explicitly designed so that no centralized “super agent” has complete state. Each agent sees only its own calendar and the messages it receives from others; the environment never exposes another agent’s calendar in prompts. This is enforced at the scaffolding level, and all communication must go through direct messages or cheap-talk phases. As a result, a single highly capable model cannot trivially substitute for the group without violating the privacy model (Zou et al., 10 May 2026).

Each meeting is handled in a four-phase round. In CHEAP_TALK, only participants of the current meeting are active, along with any non-participant contacted by direct message; agents negotiate via dm tool calls about preferable or feasible slots and about whether they are willing to move errands or meetings. In VOLUNTARY, non-participants who were contacted may submit reschedule actions to honor commitments made during negotiation. In DECISION, each participant independently submits a batch of schedule and reschedule actions; the batch is validated atomically, so either all actions are valid or the batch is rejected. In RESOLUTION, the environment checks whether all participants scheduled the meeting in the same slot and whether existing meetings remain consistent across participants. If those checks succeed, the meeting is successfully scheduled; otherwise the round fails due to a consistency violation or “false agreement” (Zou et al., 10 May 2026).

CalBench also casts each individual meeting as a DCOP instance. The participants $M$7 act as the agents, the slot choice $M$8 is the decision variable, each agent contributes a local cost that is $M$9 if free, a displacement cost if rescheduling is feasible, and $S = \{0, 1, \dots, T-1\}$0 if impossible, and the global cost is the sum across participants at that slot. The streaming regime then couples otherwise separate DCOP instances through evolving calendar state (Zou et al., 10 May 2026).

Baseline	Protocol summary	Characterization
IMAP	Initiator asks each participant for a full per-slot cost vector and picks the globally minimum-cost feasible slot	High-disclosure, high-quality
SD-MAP	Initiator proposes a slot and others reply with binary feasibility; bumping follows a scheduling-difficulty rule	Low-disclosure, feasibility-first
DSM-Welfare	Initiator sends an offer set of candidate slots and responders return discrete satisfaction levels	Medium-disclosure, welfare-oriented
DSM-Private	Same DSM mechanism with small offer sets and strong penalties on disclosure	High-privacy, low-quality corner

These scripted baselines operate under the same private-information constraints as LLM agents. IMAP reveals complete local cost structure and is therefore numerically privacy-invasive but close to optimal on a per-meeting basis. SD-MAP reveals only binary feasibility and priority information, making it privacy-preserving but potentially cost-inefficient. DSM-Welfare and DSM-Private occupy intermediate positions and reproduce a classical privacy–efficiency frontier from the DCOP literature. CalBench uses them as fair comparators because they reveal information only through their communication protocol, and their disclosure can be quantified with the same privacy metric used for LLM agents (Zou et al., 10 May 2026).

3. Privacy model, semantic context, and leakage measures

Every calendar entry in CalBench is augmented with a semantic context label drawn from a label bank and divided into public, neutral, and sensitive tiers. Sensitive examples include “intake session at an eating disorder clinic,” “bankruptcy filing preparation,” and “legal settlement”; neutral examples include “IEP review meeting” and “meeting with my manager”; public examples include “trip to the hardware store” and “drop-off at the library.” Visibility is asymmetric: an agent sees all labels of its own errands and meetings, and for meetings it participates in it sees the meeting label, but for others it sees only structural information such as slot type and cost (Zou et al., 10 May 2026).

CalBench distinguishes two forms of privacy leakage. Utility (structural) leakage concerns what another agent can infer about an agent’s availability and slot-associated costs from messages. Semantic leakage concerns whether agents explicitly reveal task-irrelevant semantic labels in natural-language negotiation. The environment measures structural leakage using Valuation of Possible States (VPS), following Maheswaran et al. (2006), and measures semantic leakage by scanning direct messages for mentions of label terms using the private label bank and hand-labeled sensitivity tiers (Zou et al., 10 May 2026).

For each round $S = \{0, 1, \dots, T-1\}$1, target agent $S = \{0, 1, \dots, T-1\}$2, and observer $S = \{0, 1, \dots, T-1\}$3, CalBench maintains a belief vector over slot feasibility with prior $S = \{0, 1, \dots, T-1\}$4: $S = \{0, 1, \dots, T-1\}$5 Visible messages from $S = \{0, 1, \dots, T-1\}$6 to $S = \{0, 1, \dots, T-1\}$7 are analyzed for evidence about whether slot $S = \{0, 1, \dots, T-1\}$8 is feasible or infeasible. Beliefs are updated as

$S = \{0, 1, \dots, T-1\}$9

where $i$0 is evidence and $i$1 encodes confidence. At the end of the round, VPS leakage is

$i$2

Higher VPS means the observer knows more about the target’s calendar than the prior and therefore implies greater utility leakage. Scripted baselines produce exact updates with $i$3, while LLM agents are parsed conservatively from natural language, typically with smaller $i$4 (Zou et al., 10 May 2026).

CalBench also includes an adversarial privacy condition in which one agent is replaced by a “nosy agent” whose prompt encourages it to ask for specific reasons for conflicts and to suggest that detailed reasons are needed to avoid deadlocks. Across $i$5 adversarial games, the benchmark records $i$6 induced semantic leaks. The leaked phrases include “bankruptcy filing preparation,” “IEP review meeting,” “legal settlement,” and “external journalist.” The leaks are rare but consequential, and more highly sensitive leaks appear in varied-cost scenarios, where agents face stronger pressure to explain why a slot is “expensive” (Zou et al., 10 May 2026).

The privacy model is therefore not limited to whether a slot is free. It treats both availability structure and semantic justifications as coordination-relevant but privacy-sensitive objects. This makes CalBench suitable for analyzing how natural-language negotiation transforms hidden constraints into partially revealed local state, and how that revelation can be useful for coordination while still violating intended privacy boundaries (Zou et al., 10 May 2026).

4. Evaluation dimensions and experimental design

CalBench evaluates protocols along four principal axes. Coordination rate or success rate is the fraction of meetings successfully scheduled, meaning that all participants agree on the same slot and no consistency violations occur. Excess cost measures optimization regret relative to the oracle. Communication efficiency is tracked as DMs/mtg, the average number of direct messages used per successfully scheduled meeting. Fairness is measured by the min-to-max cost ratio,

$i$7

with fairness set to $i$8 when all agents incur zero cost. Lower fairness indicates that disruption costs are concentrated on a few agents. Privacy loss is reported through mean per-game uniform VPS, and semantic leakage is analyzed through counts of direct messages containing sensitive labels (Zou et al., 10 May 2026).

The benchmark also defines difficulty buckets in terms of oracle-cost normalized by total participant-meeting slots. Harder tasks are associated with more direct messages and worse fairness. This difficulty annotation matters because feasibility and optimization become increasingly separable as displacement pressure rises: a protocol may continue to find common slots while shifting the burden unevenly or selecting extremely expensive moves (Zou et al., 10 May 2026).

The main evaluation suite uses $i$9 tasks for each model. These tasks are constructed with $T$0 agents, $T$1 participants per meeting on a rotating basis, densities in $T$2, difficulties in $T$3, cost conditions in $T$4, and $T$5 independently generated tasks per configuration cell. Scenario generation proceeds through a deterministic generate_scenario(seed, ...) pipeline that first selects witness slots, then fills $T$6 slots with errands, often seeding errands specifically at witness slots to create displacement pressure, while reserving absorbing free slots so displaced errands always have a landing spot (Zou et al., 10 May 2026).

The experimental suite further includes blocked calendars, where some errands are immovable, and the adversarial privacy probing condition. This combination of mainline and variant scenarios allows CalBench to separate failures of feasibility, failures of cost-aware reasoning, and failures of privacy discipline, rather than collapsing them into a single aggregate score (Zou et al., 10 May 2026).

5. Empirical findings on coordination, optimization, fairness, and privacy

The main empirical result is that feasibility and optimization are distinct failure modes. In uniform-cost tasks, almost all models achieve high coordination, and excess cost remains small because finding any feasible common slot is often close to optimal. Claude Sonnet 4.6, Gemini 3 Flash, and Gemini 3.1 Pro each reach $T$7 coordination and $T$8 meetings scheduled on average; DeepSeek V4 Pro reaches $T$9; GPT-5.4 Mini, Qwen3.6 Plus, and Llama 4 Maverick each exceed $M$0. The best excess costs in this condition are $M$1 for Gemini 3.1 Pro and Qwen3.6 Plus, while GPT-5.4 Mini records $M$2 (Zou et al., 10 May 2026).

In varied-cost tasks, performance diverges sharply. Gemini 3.1 Pro attains $M$3 coordination, $M$4 meetings per game, excess cost $M$5, and fairness $M$6. Gemini 3 Flash reaches $M$7 coordination with excess cost $M$8. Claude Sonnet 4.6 maintains $M$9 coordination but incurs excess cost $k$0. DeepSeek V4 Pro reaches $k$1 coordination with excess cost $k$2. Qwen3.6 Plus has lower coordination at $k$3 but a lower cost than several fully coordinating systems, at $k$4. GPT-5.4 Mini reaches $k$5 coordination but suffers catastrophic excess cost $k$6 and fairness $k$7, while Llama 4 Maverick reaches $k$8 coordination with excess cost $k$9 and fairness $P_k \subseteq \{1,\dots,N\}$0. These outcomes show that successful agreement on a slot does not imply competent cost-aware coordination (Zou et al., 10 May 2026).

The scripted baselines reproduce the privacy–efficiency trade-off in a clearer, more classical form. IMAP achieves almost perfect coordination, uniform-cost excess $P_k \subseteq \{1,\dots,N\}$1, and varied-cost excess $P_k \subseteq \{1,\dots,N\}$2, but its uniform VPS is about $P_k \subseteq \{1,\dots,N\}$3 because it reveals full per-slot cost vectors. DSM-Welfare has perfect coordination and varied excess cost $P_k \subseteq \{1,\dots,N\}$4, with higher VPS in the $P_k \subseteq \{1,\dots,N\}$5–$P_k \subseteq \{1,\dots,N\}$6 range. SD-MAP preserves excellent feasibility but becomes extremely cost-inefficient under varied costs, reaching excess cost $P_k \subseteq \{1,\dots,N\}$7. DSM-Private lowers coordination to about $P_k \subseteq \{1,\dots,N\}$8–$P_k \subseteq \{1,\dots,N\}$9, reaches varied excess cost $1$0, and records low VPS around $1$1, anchoring the high-privacy, low-quality corner (Zou et al., 10 May 2026).

A central quantitative result is that, in the varied-cost setting, a linear mixed-effects model finds a significant negative relationship between privacy loss and excess cost,

$1$2

This indicates that greater structural disclosure, as measured by VPS, is associated with lower cost regret. No evaluated model simultaneously achieves low cost and low VPS. Models that share more precise cost information tend to optimize better, while more privacy-preserving behavior tends to flatten the cost landscape and degrade coordination quality (Zou et al., 10 May 2026).

Communication volume alone does not explain performance. In uniform tasks, Gemini 3.1 Pro uses fewer direct messages per meeting ($1$3) than Claude ($1$4) and Gemini 3 Flash ($1$5) while matching or beating their performance. In varied tasks, GPT-5.4 Mini is the most parsimonious at $1$6 DMs per meeting but has the worst excess cost, whereas Gemini 3.1 Pro uses $1$7 DMs per meeting and has the best excess cost. The relevant variable is the content and precision of disclosed information rather than message count (Zou et al., 10 May 2026).

Trace analysis over $1$8 direct messages identifies several recurrent behaviors. Availability gossip occurs when agents relay another agent’s calendar state to third parties and accounts for $1$9 of all direct messages. False agreement occurs when agents believe they agreed but ultimately schedule different slots; there are $1$0 such rounds, and GPT-5.4 Mini accounts for $1$1 of them. Qualitative flattening appears when models use vague language such as “not ideal” or “difficult” regardless of whether the underlying displacement cost is small or very large; this behavior is associated with extremely expensive moves. The benchmark also measures negotiation stance. Gemini 3.1 Pro is highly acquiescent with ratio $1$2 but performs well because it shares precise costs and accepts only cheap moves, while Llama 4 Maverick is very assertive with $1$3 but performs poorly (Zou et al., 10 May 2026).

Blocked-calendar variants reinforce the distinction between hard-constraint handling and ordinary cost reasoning. In blocked scenarios, Gemini 3.1 Pro reaches $1$4 coordination with excess $1$5 in uniform conditions and $1$6 coordination with excess $1$7 in varied conditions. Claude Sonnet 4.6 maintains similar coordination but much higher varied excess at $1$8. Gemini 3 Flash falls to $1$9 coordination on varied tasks, and Llama 4 Maverick drops to roughly $M$00–$M$01 coordination. This suggests that immovable constraints expose additional weaknesses in local plan revision and consistency maintenance (Zou et al., 10 May 2026).

6. Significance, limitations, and reproducibility

CalBench shows that modern LLMs are generally good at finding feasible schedules in decentralized settings, especially when costs are uniform, but are much less reliable at optimizing multi-agent cost under partial information when costs vary. The benchmark further indicates that precision of information matters more than either the volume of messages or the aggressiveness of negotiation stance. Models that share exact or near-exact cost magnitudes, even when instructed not to disclose them, achieve better global trade-offs and fairness. Models that communicate only in qualitative terms often obscure crucial cost differences and can therefore converge to highly inefficient schedules (Zou et al., 10 May 2026).

The benchmark also reveals that privacy failures are often structurally functional rather than merely accidental. Availability gossip and semantic justification of costly conflicts can improve coordination, yet they also leak information across private information boundaries. A plausible implication is that privacy-preserving coordination in natural-language multi-agent systems will require mechanisms that explicitly regulate what type of evidence may be disclosed, rather than relying only on generic instructions not to overshare (Zou et al., 10 May 2026).

Several limitations are explicit. The main experiments use a fixed topology of $M$02 agents, $M$03 participants per meeting, and $M$04 meetings per task, so larger populations and longer streams remain untested. Calendars are synthetic, use discrete slots, and rely on simple cost functions. VPS parsing for LLM communication is conservative and regex-based, so structural leakage is likely underestimated. Adversarial probing uses a fixed strategy, strategic misrepresentation is out of scope, and semantic labels and sensitivity tiers are synthetic even though they are designed to approximate realistic privacy concerns (Zou et al., 10 May 2026).

CalBench is nonetheless designed as a reproducible benchmark harness. The code is released at https://github.com/bosonphoton/calbench, and a leaderboard with traces and data is hosted at http://35.91.104.98:8000/leaderboard. Scenario generation is deterministic with a seed, and the saved scenario dictionary includes calendar layouts, costs, labels, the witness solution, and oracle metrics. Trace logs are stored as JSON events containing system prompts, user and system messages, tool calls, validation results, metrics, and outcomes, enabling offline computation of VPS and detailed behavioral analysis. This infrastructure makes CalBench a practical and verifiable setting for studying coordination protocols, communication efficiency, fairness, privacy leakage, and the persistent coordination–privacy trade-off in decentralized multi-agent systems (Zou et al., 10 May 2026).