AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators

Abstract: Multi-agent systems achieve state-of-the-art outcomes through peer collaboration. However, when an agent in the pipeline silently drops a constraint, the system's final output may look correct even though the reasoning chain was quietly corrupted, and existing outcome-based evaluations are blind to such multi-hop process failures. To make these vulnerabilities measurable before deployment, we introduce AgentCollabBench, a diagnostic benchmark of 900 human-validated tasks spanning software engineering, DevOps, and data engineering. Each task isolates one of four behavioral risks: instruction decay (does a constraint survive peer pressure?), false-belief contagion (does a falsehood spread through consensus?), context leakage (does information bleed between tasks?), and tracer durability (does marked data reach the final agent?). Evaluating four modern LLMs (GPT 4.1 mini, Gemini 2.5 Flash Lite, Qwen-3.5-35B-A3B, and Llama 3.1 8B Instruct), we expose model-specific vulnerability profiles invisible to outcome-only evaluation; Qwen-3.5-35B-A3B, for example, leads on tracer durability and instruction stability, while GPT 4.1 mini leads on leakage containment and false-belief resistance. Beyond per-model differences, communication topology emerges as a primary risk factor that explains 7-40% of the variance in multi-hop information survival. The effect traces to a synthesis bottleneck specific to converging-DAG nodes: an agent weighing competing parent inputs discards constraints carried by a minority branch, a bottleneck structurally absent from linear chains. AgentCollabBench demonstrates that suboptimal topology can silently erase the safeguards of highly capable models, arguing that multi-agent reliability is fundamentally a structural problem and that scaling model intelligence alone is no substitute for architecture.

• The paper introduces a 900-task, human-validated benchmark spanning three enterprise domains and five communication topologies, with four independent metrics for instruction decay, information loss, false-belief spread, and cross-task leakage.
• The evaluation finds no model dominates all collaboration risks: Qwen leads information survival and instruction adherence, GPT 4.1 mini leads leakage containment and consensus accuracy, while Llama 3.1 8B performs worst on three metrics.
• Communication topology explains 7%–40% of information-survival variance, with converging DAGs creating a synthesis bottleneck where agents can produce correct-looking outputs while dropping constraints from minority branches.

AgentCollabBench is a diagnostic benchmark of 900 human-validated tasks designed to measure process-level failures in multi-agent LLM systems that outcome-based evaluation cannot detect (2605.08647). The central observation motivating the benchmark is that a multi-agent pipeline can produce a syntactically complete, task-correct final output while silently dropping a critical constraint mid-chain. Existing benchmarks—whether outcome-oriented (SWE-bench, GAIA, AgentBench), collaboration-oriented (MultiAgentBench, GEMMAS, Collab-Overcooked), or failure-taxonomic (MAST)—do not provide quantitative, controlled instruments for probing whether constraints survive peer pressure, whether information survives multi-hop forwarding, whether false beliefs spread through consensus, or whether private context leaks across task boundaries.

Benchmark design and metric suite

The corpus spans three enterprise domains (software engineering, DevOps, data engineering), five communication topologies (linear chain, branching tree, converging DAG, fully connected, custom graph), and three structural-complexity tiers, balanced at 300 tasks per domain and 180 per topology. Tasks are co-designed with their topology rather than reassigned across topologies, on the argument that arbitrary reassignment would conflate topology effects with task-topology mismatch artifacts. Construction proceeds through a generator–validator loop using Gemini 3 Flash (disjoint from all evaluated backbones), deterministic schema checks, and independent dual-annotator review by seven annotators; roughly 10% of entries required correction. A diversity audit over 8,791 within-stratum pairs found only one pair above 0.90 ROUGE-L similarity and three above 0.90 embedding cosine, all manually confirmed behaviorally distinct.

Four metrics isolate four failure mechanisms:

• Instruction Decay Rate (IDR): the fraction of constrained turns in which an agent violates an injected behavioral constraint after peer messages normalize violation.
• Radioactive Tracer Durability (RTD): the deepest graph layer at which an injected tracer string is detected verbatim, normalized by graph depth.
• Consensus Pollution Rate (CPR): the fraction of downstream responses that adopt a seeded false fact, counting both explicit endorsement and implicit reliance.
• Cross-task Leakage Containment (CLC): the fraction of declared-private Task-A tracers appearing verbatim in a later Task-B run within the same session.

IDR and CPR are scored by LLM judges (DeepSeek V4 Flash, disjoint from evaluated backbones); RTD and CLC are computed deterministically from traces. Judges achieve Cohen's $\kappa \geq 0.69$ against human annotators (84.4–89.3% agreement). The metrics are deliberately not aggregated into a single score, since they probe independent behavioral dimensions.

Model behavioral profiles

Evaluating GPT 4.1 mini, Gemini 2.5 Flash Lite, Qwen-3.5-35B-A3B, and Llama 3.1 8B Instruct yields no model that dominates all four axes. Qwen-3.5-35B-A3B leads on RTD (94.0%) and IDR (0.9%) but ranks third on CLC (4.7%); GPT 4.1 mini leads on CLC (2.6%) and CPR (17.7%) but ranks third on RTD (80.3%); Llama 3.1 8B Instruct is worst on three of four metrics (IDR 10.1%, CPR 40.3%, RTD 62.6%). Bonferroni-corrected Mann–Whitney U tests confirm these fingerprints are statistically separable per metric, though no CLC pair survives correction. Rank correlations against Chatbot Arena positions are moderate ($r_s = +0.80$ for IDR, RTD, CPR; $+0.40$ for CLC) but exhibit metric-by-metric reversals—for instance, GPT 4.1 mini ranks first on CPR and CLC yet third on IDR and RTT—supporting divergent validity from general capability. The authors note this analysis is descriptive given only four models.

Construct validity is established through perturbation ladders on a separate Llama 3.1 8B backbone: IDR increases with peer pressure ($\rho=0.211$), CPR with false-fact plausibility ($\rho=0.411$), CLC with injection intensity ($\rho=0.146$, CI crossing zero), and RTD decreases with topology siloing ($\rho=-0.313$). All four move in the intended direction, which has probability 0.0625 under a null of random directionality.

Topology as a primary risk factor

The strongest structural finding concerns RTD. Communication topology explains $\eta^2 = 0.07$–$0.40$ of variance in multi-hop information survival across all four models—an effect comparable in magnitude to backbone choice—and this association replicates under cell-capped and stratum-restricted sensitivity analyses. Direct-routing topologies (fully connected, custom graph) achieve near-ceiling RTD (96–100% for GPT 4.1 mini) simply by eliminating multi-hop paths, whereas propagation topologies lose substantially more. Among propagation structures, converging DAG is worst for all four models (e.g., 49.4% for GPT 4.1 mini versus 75.9% for linear chain).

Per-hop drop-rate analysis localizes this deficit to a synthesis bottleneck: converging-DAG hops drop tracers at 1.6–5.0× the rate of linear-chain hops (Llama 1.6×, GPT 2.0×, Gemini 3.3×, Qwen 5.0×), unexplained by path length or fan-in (Pearson $r=-0.176$ and $r_s = +0.80$0, $r_s = +0.80$1 edges). Trace audits classify failures as synthesis loss (tracer present in at least one parent output but absent from the convergence node's merged response—the largest category for GPT, Qwen, and Llama), upstream loss, or partial loss. A lexical overlap audit over all 47 synthesis-loss cases shows zero word-level overlap in every case, ruling out paraphrase as an artifact of exact-match scoring. Representative examples show convergence nodes producing complete, deployment-ready outputs (Kubernetes manifests, data pipelines) while omitting a constraint carried by a minority branch. Notably, for Llama 3.1 8B Instruct the bottleneck is masked by uniformly high per-hop drop rates, distinguishing topology-driven failure from model-capacity limitation.

A secondary finding is that structural complexity is largely orthogonal to behavioral pressure: IDR shows no monotonic trend across complexity tiers for any model, and significant ordinal-regression effects occur only for GPT 4.1 mini (CPR increasing, RTD and CLC decreasing with complexity). Domain likewise does not confound the model fingerprints, as domain shifts are metric-specific and do not mirror cross-model patterns.

Limitations and open questions

The paper concedes several boundaries explicitly. Coverage is restricted to three domains and five fixed topologies; dynamic topology selection, peer recruitment, and removal are deferred to future versions. IDR and CPR depend on LLM judges despite judge–backbone separation, and judge validation shows asymmetric error profiles (the GPT 4.1 mini validation judge is conservative for IDR, with 19 false negatives versus 4 false positives). RTD can undercount highly abstractive paraphrases, and CLC captures explicit identifier leakage rather than semantic leakage. Reported values are protocol-specific measurements rather than universal backbone properties, since memory policy, routing, and tool-use protocols also affect the metrics. Most significantly, the paper does not report correlations between behavioral scores and final task success; predictive validity—whether low RTD or high CPR actually forecasts deployment failures—remains an open empirical question requiring paired deployment studies.

Conclusion

AgentCollabBench provides a controlled, human-validated instrument for measuring four behavioral failure modes invisible to outcome-only evaluation, and demonstrates that no evaluated model dominates across them. Its principal structural result—that communication topology explains up to 40% of variance in information survival via a converging-DAG synthesis bottleneck—supports the paper's claim that multi-agent reliability is partly an architectural problem not solvable by scaling model capability alone. Whether explicit constraint echoing at convergence nodes or failure-aware topology adaptation mitigates the identified bottleneck remains to be tested.