Scaling Discovery through Test-Time Communication

Abstract: Science advances not in isolation but through collaboration, yet existing agentic systems capture little of this. Whether communicating agents help remains an open question with mixed prior results. We show that test-time communication can substantially outperform independent parallel attempts on challenging tasks, where sharing a breakthrough can push the whole group forward. We first study the effect of scaling multi-agent test-time communication, where agents have no predefined roles and communicate via a shared directory, on ARC-AGI-3, a benchmark requiring novel problem solving. We find that a team of $k$ communicating agents, team@$k$, matches the success rate of $4k$ independent agents, and this advantage grows with $k$, suggesting gains compound with scale. The effect is not merely efficiency: a task that no single agent can solve, a team of agents can solve reliably. Furthermore, these gains transfer to research-oriented tasks, given sufficient compute. On polyomino packing, communicating agents outperform best@$k$ and exceed the prior best-known score. On MNIST classifier compression, communication surpasses the best-known human solution. A team of four agents produced a 1,957-byte classifier submission achieving 99.4% test accuracy, smaller than both the best-known human solution and the best single-agent result. These gains are not unconditional. Independent agents may outperform communication when compute is limited or when a clear measure of progress is absent. However, under sufficient compute and clear feedback, multi-agent communication consistently yields stronger results.

• The paper demonstrates that LLM agents in teams that share intermediate results can outperform their independent counterparts, under well-defined conditions.
• In the ARC-AGI-3 benchmark, a team of 5 agents can achieve yield the performance of 33 independent agents.
• The protocol for communication between agents follows minimal architecture rules to reward progress increments.

The paper studies whether LLM agents benefit from communicating during test-time search, rather than merely generating independent parallel attempts. Its central comparison is between team@k, in which $k$ identical agents asynchronously share intermediate results through a common workspace, and best@k, the strongest result among $k$ independent agents operating under the same per-agent resource allocation. The experimental claim is specific: communication can transform parallel exploration into cumulative progress when agents receive sufficiently dense, discriminative feedback from an external verifier. Under these conditions, the communicating team can exploit discoveries made at different stages by different agents, whereas independent sampling requires one complete trajectory to succeed.

The work deliberately uses a minimal communication architecture. Agents have identical prompts, models, tools, objectives, and resource limits; they receive no predefined roles and are not managed by a central orchestrator. Each agent has a private context and scratch directory, while the team shares an append-only communication log, a score log, evidence of failed approaches, and a graded artifact. Agents claim distinct search slots through atomic filesystem operations, publish reproducible findings, and adopt peer methods only after observing measured improvement. The resulting protocol is intended to preserve diversity while enabling verified progress sharing (2609.21032).

Experimental design and communication protocol

The evaluation spans three task classes with different horizons and feedback structures. ARC-AGI-3 measures interactive novel problem solving across 25 grid-world games. Frontier-CS polyomino packing requires agents to develop and optimize a C++17 heuristic for an NP-hard packing problem. MNIST classifier compression evaluates empirical ML research and deployment engineering: agents must produce a self-contained classifier with at least $99.4\%$ test accuracy while minimizing deterministic gzip-compressed artifact size.

The protocol is important because the paper does not equate communication with unconstrained consensus. Agents are instructed to claim distinct approaches, test every change against the task's native scoring mechanism, record negative results, and retain a meaningful variation after adopting another agent's method. Shared candidates are protected by synchronization and scoring logs. In the ARC experiments, agents additionally synchronize periodically by level and action budget. In packing and MNIST, they repeatedly submit candidates to hidden scorers or sealed evaluation oracles.

The baselines are matched at the agent level, but not always identical in aggregate resource consumption. A team of $k$ agents naturally produces more total tokens than one independent pool of $k$ agents when communication prolongs useful work. The paper therefore reports performance against wall-clock time and cumulative output tokens, and separately studies fixed total action budgets. This is necessary for distinguishing a communication effect from a simple increase in inference expenditure.

ARC-AGI-3: communication compounds with task depth

ARC-AGI-3 provides the cleanest controlled test because each game consists of six to ten levels and agents must transfer knowledge from earlier levels to later ones. Claude Sonnet 4.6 is evaluated on all 25 public games, with 64 single-agent trials per game and 20 trials for each team configuration. The native per-level action budget is used, and a game is counted as solved only if all levels are completed.

Communication improves over independent selection at every depth. For $k=5$, team@5 exceeds best@5 by a factor of $1.2$ near the beginning of the games and by $3.6$ at full completion. The final average solve rates are $8.0\%$ for team@5 and $2.2\%$ for best@5. For $k$0, team@3 reaches $k$1, compared with $k$2 for best@3. Matching the final solve rate of team@3 requires best@13, while matching team@5 requires best@33. Thus, under this benchmark, three communicating agents provide the final-solve effectiveness of approximately 13 independent agents, and five communicating agents provide that of approximately 33.

The advantage is not distributed uniformly across games. Communication helps on 18 of 25 games and harms on seven, but no observed loss exceeds $k$3 average levels, whereas the largest gain is $k$4 levels on LP85. Among the four games with any team@5 success, average solve rate increases from $k$5 for the corresponding independent baseline to $k$6 for team@5. On LP85, all 64 single-agent trials fail, while team@5 solves the game in $k$7 of trials. On FT09, team@3 reaches a $k$8 solve rate, compared with $k$9 for best@3.

Figure 1: Communication improves progress on 18 of 25 ARC-AGI-3 games and produces especially large gains on the small subset where Sonnet 4.6 can occasionally reach a complete solution.

The concentration of final successes is itself informative. Most games remain out of reach for Sonnet 4.6 under either condition, so the aggregate improvement does not imply that communication universally resolves the benchmark. Instead, communication is most valuable near the boundary between failure and success, where one agent's partial discovery can make subsequent levels tractable for the rest of the team.

Compute efficiency and the coordination tax

The ARC results also establish that communication is not immediately beneficial. In the low-compute regime, best@k initially leads because independent agents avoid the overhead of reading shared state, coordinating approaches, and reproducing peer results. The team overtakes only after sufficient search time has accumulated.

Figure 2: Independent sampling leads at low token budgets, but communicating teams eventually attain higher solve rates and require substantially fewer tokens to match their final performance.

At the end of the ARC evaluation, team@3 and team@5 consume approximately 8 million and 24 million output tokens, respectively, nearly twice the token expenditure of their corresponding best-of-$99.4\%$0 baselines. This raw comparison would favor independent sampling if only total terminal token count were considered. However, once solve rate is plotted against cumulative output tokens, the communicating teams become more efficient after the initial regime. For solve rates above approximately $99.4\%$1, best@3 uses $99.4\%$2 as many tokens as team@3 at comparable performance, and best@5 uses $99.4\%$3 as many as team@5. Matching the final team solve rates requires approximately $99.4\%$4 more tokens for $99.4\%$5 and $99.4\%$6 more for $99.4\%$7 under independent sampling.

The paper further tests whether communication merely provides a longer aggregate action horizon. At matched total action budget, team@5 outperforms both best@5 and a single agent given five times the native action budget on the beneficial games SB26 and LP85. On LP85, team@5 finishes at $99.4\%$8, while best@5 falls to zero at the final level and the long-horizon single agent reaches only approximately $99.4\%$9. However, when each of the five agents receives only $k$0 times the native budget, communication becomes counterproductive: the team falls below a single agent with the same total budget.

Figure 3: Communication is effective only when each agent retains enough budget to perform meaningful independent exploration.

This ablation gives the paper's strongest operational qualification: communication does not rescue severely resource-constrained agents. A team must preserve sufficient per-agent search depth for members to generate nonredundant discoveries. Otherwise, coordination fragments an already inadequate budget.

The authors also measure relative human action efficiency, which accounts for action economy rather than only completion. Communication improves both the strongest team member and the average member. The best agent's mean RHAE increases from $k$1 for a single agent to $k$2 with team@3 and $k$3 with team@5, compared with $k$4 and $k$5 for best@3 and best@5. The average team@5 agent reaches $k$6, approximately matching the best of five independent agents. This rules out an explanation based solely on selecting one unusually successful trajectory: communication improves the efficiency of agents during their own interaction with the task.

Figure 4: Communication improves both the best agent and the average agent according to relative human action efficiency.

Long-horizon algorithmic discovery

The polyomino experiments test whether the ARC pattern transfers to open-ended algorithm development. Agents must write a program that packs between 100 and 10,000 polyominoes into minimum-area rectangles. The scorer returns continuous partial credit over 70 hidden cases, allowing agents to evaluate incremental algorithmic changes.

Under the three-hour Frontier-CS setting, a team of three Sonnet 4.6 agents reaches a score of $k$7, compared with $k$8 for the strongest single-agent trajectory and $k$9 for the prior best-known result. The team therefore establishes a new benchmark frontier while using the same total three-agent wall-clock allocation as the independent comparison. The score difference corresponds to reducing mean unused area from $k$0 to $k$1.

Figure 5: A communicating Sonnet 4.6 team surpasses both the strongest single-agent trajectory and the prior best-known packing score.

The advantage persists over 72-hour runs, although the smaller number of trials reduces the absolute scores. With Claude Opus 4.6, team@4 reaches $k$2, while the best independent trajectory reaches $k$3. With Sonnet 4.6, team@4 reaches $k$4, compared with $k$5 for the best single-agent run. The teams take a lasting lead after approximately two hours for Opus and four hours for Sonnet, indicating that the effect is not merely an artifact of the three-hour stopping rule.

Figure 6: Over a 72-hour horizon, the Opus 4.6 team continues improving after independent runs plateau, reaching $k$6 versus $k$7.

Figure 7: The longer Sonnet 4.6 team run reaches $k$8, while the strongest independent trajectory ends at $k$9.

The trace analysis provides a concrete mechanism for the result. Early agents explore shelf, skyline, and bottom-left packing. These methods approach approximately $k=5$0 but leave enclosed gaps. One agent proposes contact maximization, another independently reimplements the idea, makes it computationally viable, and adds all-orientation lookahead. A third later adds a boundary bonus. The final improvement is therefore not a vote over complete solutions. It is a sequence of dependent transformations: one agent supplies an objective, another supplies an efficient search procedure, and another extends the objective.

Figure 8: On the same 172-piece instance, communication raises packing density from $k=5$1 for an illustrative single-agent solution to $k=5$2 for the team solution.

This case also illustrates why the shared score log matters. A failed implementation can still expose a useful search criterion. The successful team does not simply copy the first contact-based method; it reconstructs the idea, changes the implementation, and allocates computation sufficiently to make the search practical under the two-second evaluator limit.

MNIST compression and compositional engineering

The MNIST experiment extends the evaluation to empirical ML engineering. Four GPT-5.6 Sol agents receive 96 hours to minimize the deterministic gzip-9 size of a complete inference submission while maintaining at least $k=5$3 sealed-test accuracy. The artifact includes code, weights, constants, and decoders, so architectural and serialization decisions interact directly.

Independent agents initially outperform the team. They produce submissions below 15 KB before the team leaves its initial approximately 75 KB regime. The team catches the independent frontier after roughly one hour and 100,000 output tokens, then continues improving. After 96 hours, best@4 reaches 3,160 bytes, while team@4 reaches 1,957 bytes. No independent run beats the 2,461-byte human reference; the team crosses that threshold after approximately 20 hours and ultimately produces a submission about $k=5$4 smaller.

Figure 9: Communication initially incurs a coordination cost but eventually produces a 1,957-byte classifier, below both independent runs and the 2,461-byte human reference.

The final model is a recurrent convolutional network with 2,900 quantized parameters and 30 float16 scale factors. It reuses one set of depthwise and pointwise filters across nine residual updates, reducing stored weights while retaining depth. Its classification head combines six learned weighted sums of the 128-dimensional feature vector with twelve directly selected original features. Quantization-aware training restricts projection parameters to scaled integers in $k=5$5, improving compressibility through repeated values. The final submission obtains $k=5$6 test accuracy.

The qualitative trajectory again demonstrates cumulative rather than aggregative benefit. One agent introduces the recurrent CNN backbone. Another identifies a compressed scoring layer and bypasses information loss by forwarding selected original features. A third combines the smaller projection with quantized integer weights. A branch that had fallen behind is later recombined with the leading design, and feature reordering removes the final bytes without changing predictions. The 1,957-byte result is therefore assembled from complementary partial improvements, including a branch that could not have won independently.

Verified progress sharing as the proposed mechanism

The paper formalizes its interpretation with a pedagogical stage-completion model. Suppose a task requires $k=5$7 successive improvements and agents can query a verifier after each improvement. If $k=5$8 is the time agent $k=5$9 needs to discover the next improvement at stage $1.2$0, independent sampling requires the fastest agent to complete all stages:

$1.2$1

With communication, the team can use the fastest discovery at each stage:

$1.2$2

The latter is always no greater in the idealized model because it changes a minimum of complete trajectory sums into a sum of stagewise minima. Under independent exponential stage times, the team completion time has expected value $1.2$3, while independent agents must each accumulate all $1.2$4 stages before selection. The paper derives exponential upper and lower tail bounds showing that, at a matched runtime, team completion probability can approach one while best-of-$1.2$5 completion probability approaches zero as the number of stages grows.

The model is explicitly presented as pedagogical rather than empirical. Its value is conceptual: it identifies the structural condition required for communication to beat independent sampling. Intermediate improvements must be both reusable and reliably rankable. If a verifier only evaluates the final artifact, communication cannot exploit stagewise minima in the same way.

Terminal-Bench 2.0 provides the negative result. Across 89 tasks, team@2 improves over pass@1, reaching mean accuracy of $1.2$6 versus $1.2$7, but does not exceed independent pass@2 at $1.2$8. The authors attribute this partly to sparse and incomplete intermediate verification. Public checks may validate individual requirements without establishing overall task correctness, so agents lack a dependable signal for deciding which partial state should become the team's shared foundation. Moreover, team@2 produces one shared final state, whereas pass@2 retains two independent states and receives oracle selection after execution.

Limitations and open questions

The evidence is strong within the tested regime but narrower than the headline scaling factors might suggest. ARC-AGI-3 uses one model family, Claude Sonnet 4.6, and the strongest gains are concentrated in a few games; most games remain unsolved. Team and independent runs also involve different aggregate token trajectories, despite the paper's token-efficiency analyses. The coordination protocol contains substantial task-specific machinery, including shared score logs, synchronization, atomic slot allocation, and explicit instructions to preserve diversity. Consequently, the results isolate a particular shared-workspace protocol rather than communication in the abstract.

The polyomino and MNIST results rely on selecting the highest-scoring trajectory among relatively small numbers of long-horizon trials. The 72-hour packing comparisons use only 12 independent trials and two team trials, so their absolute scores are noisy. The MNIST benchmark has a single dataset and a highly specific artifact metric; its human reference is reproduced under the paper's packaging convention, and the team result may depend on the unusual interaction between architecture design, quantization, and gzip compression.

The theoretical separation assumes independent stage discoveries, freely transferable improvements, a faithful verifier, and no interference between agents. Real agents may herd, corrupt shared state, misinterpret evidence, or over-adopt mediocre approaches. The authors model this possibility through an effective number of independent groups $1.2$9, but do not estimate $3.6$0 empirically. Open questions therefore concern whether comparable gains survive under noisy or subjective feedback, heterogeneous models, alternative communication topologies, explicit roles, or stronger orchestration. Terminal-Bench shows that final-task evaluation alone is insufficient, but it does not determine whether better self-generated intermediate verifiers could recover the advantage.

Conclusion

“Scaling Discovery through Test-Time Communication” (2609.21032) presents evidence that communication can provide more than parallel decomposition or best-of-$3.6$1 selection. In tasks with long horizons, reusable partial discoveries, sufficient per-agent compute, and accessible progress verification, communicating agents accumulate improvements that independent trajectories cannot combine. The measured effects are substantial: team@5 matches best@33 on ARC-AGI-3, a Sonnet team reaches $3.6$2 on polyomino packing against a $3.6$3 prior best, and a GPT-5.6 Sol team produces a 1,957-byte MNIST classifier at $3.6$4 accuracy. These gains are preceded by a coordination tax and disappear when budgets are too small or feedback cannot reliably identify progress. The paper's principal contribution is therefore not a universal preference for multi-agent systems, but an empirical and conceptual account of when verified progress sharing changes the scaling behavior of test-time search.