SwarmEvolve: Latent Agent Swarms for Evolutionary Discovery and Optimization

Abstract: LLM-guided evolutionary search uses AI agents to improve programs for mathematical and computational problems by iteratively refining solutions and combining high-scoring methods. However, the process is often slow because the LLM generates a long textual chain of thought for each round and must repeat this generation many times to maintain population diversity. We present SwarmEvolve, a framework in which hundreds of low-cost agents per round work in swarms, reason in latent space without decoding to text, and share information by passing KV caches. Each swarm studies the problem from a different angle. A novelty filter keeps the most distinct agent ideas by directly measuring the distances between the agents' hidden state vectors. Crossover of ideas happens naturally in latent space, as a summarizer agent receives the combined KV caches of all swarm members. On 21 problems in mathematical discovery and scientific computing, SwarmEvolve gets the highest average score and needs a median of only 3 to 8 minutes to reach the best scores that other state-of-the-art frameworks reach in 60 minutes. On several problems, SwarmEvolve with Qwen3 8B improves on the solutions that AlphaEvolve first found with frontier LLMs. Our work shows that agent swarms that communicate in latent space offer a fast, low-cost approach to program search for scientific problems.

• The paper introduces SwarmEvolve, a system optimizing evolutionary program search by using latent agent swarms, achieving 5x to 13x more evaluations per hour of high-quality candidate programs than existing models.
• Latent agents in SwarmEvolve perform recursive reasoning without decoding intermediate tokens, improving the efficiency of program exploration.
• SwarmEvolve's architecture includes optimization operator networks, KV-caching for communication, and latent reasoning, which reduces costs and significantly enhances search and discovery capabilities.

Problem setting and contribution

“SwarmEvolve: Latent Agent Swarms for Evolutionary Discovery and Optimization” (2611.08483) addresses the computational bottleneck in LLM-guided evolutionary program search. Existing systems use LLMs to mutate, recombine, and refine candidate programs, but typically require each agent to generate textual reasoning. This imposes substantial decoding and re-encoding costs, limits the number of proposals evaluated within a fixed budget, and makes large-scale population diversity expensive.

SwarmEvolve replaces most inter-agent textual reasoning with latent computation and KV-cache communication. Its central claim is that a large population of inexpensive latent agents can provide both greater search diversity and higher program throughput than smaller populations of text-reasoning agents, while using the same underlying LLM and hardware budget. The framework is evaluated on 21 problems spanning mathematical discovery and scientific computing, using Qwen3 8B, 14B, and 32B-FP8 and comparing against OpenEvolve, ShinkaEvolve, and CodeEvolve.

The reported results are substantial. With Qwen3 8B, SwarmEvolve evaluates 5–13 times more programs per hour than the comparison systems. It reaches the 60-minute scores of the baselines in a median of approximately 3–8 minutes, depending on the model and comparison. After 15 minutes, its average score exceeds the 60-minute average of every baseline for all three model sizes. Its Qwen3 8B configuration also improves on AlphaEvolve’s published constructions on three benchmark problems, despite AlphaEvolve’s use of frontier models.

The evaluation nevertheless measures performance on fixed, executable benchmark objectives rather than unconstrained scientific discovery. Consequently, the results establish the effectiveness of the proposed search procedure under these evaluators, but do not by themselves establish that latent swarms consistently produce novel mathematical knowledge outside such programmatic settings.

Latent reasoning and cache-based communication

SwarmEvolve builds on two mechanisms. First, latent agents perform recursive reasoning without decoding intermediate tokens. After processing their prompt, an agent feeds a transformed version of its final hidden state back into the model as the next input embedding. The transformation maps the output representation into the input-embedding space using a ridge-regression alignment between the model’s input and output embeddings, followed by norm normalization. Each agent performs 40 such latent steps.

Second, agents communicate by transferring their KV caches to a summarizer. The summarizer attends to the concatenated keys and values representing the agents’ focus prompts and latent trajectories, then decodes a textual program. Because Qwen3 uses RoPE, each cache block must be positionally rotated before concatenation so that attention scores remain consistent with the block’s new position. This implementation detail is essential: KV-cache transfer is not equivalent to naïvely concatenating hidden states, because the transformer’s positional encoding is embedded in the cached keys.

The method therefore separates computation into a cheap, highly parallel latent phase and a comparatively expensive textual synthesis phase. Agents do not need to verbalize every intermediate hypothesis. Instead, the summarizer receives the latent trajectories directly and produces one executable candidate program. This design changes the computational allocation of evolutionary search: GPU time is shifted away from repeated chain-of-thought decoding and toward generating and evaluating more complete programs.

Swarm construction and evolutionary operator

Each evolutionary round launches 10 swarms, each containing 45 latent agents, for a total of 450 agents. The swarms are assigned different search roles:

• Refinement swarms focus on improving one of the best archived programs.
• Hybridization swarms receive several high-scoring programs and seek combinations of their ideas.
• Exploration swarms sample programs from a broader portion of the archive and also include agents that receive no parent program.
• Fresh-start swarms attempt to construct a new approach from the initial program rather than continuing the current elite lineage.

Within each swarm, agents receive distinct combinations of focus phrasings and reasoning perspectives. These prompts target algorithmic choices, numerical stability, mathematical formulation, feasibility, parameterization, literature-derived constructions, and alternative algorithms. Parent programs are distributed so that the latent population varies both in prompt specialization and in the program context it reads.

After 40 latent steps, SwarmEvolve applies a novelty filter. It represents each agent by the mean of its hidden states across the reasoning trajectory, centers these representations relative to the swarm mean, and applies greedy farthest-point selection under cosine distance. Thirty of the 45 agents are retained. This procedure explicitly favors coverage of the latent population rather than selecting agents solely by a textual or score-based heuristic.

The latent geometry analysis supports the intended interpretation. Agents belonging to different swarms occupy distinct regions in a principal-component projection, indicating that the focus prompts induce role-specific reasoning trajectories. Within a single swarm, agents cluster according to the parent program they read, and the novelty filter retains representatives from the different clusters. Thus, the filter appears to preserve both cross-swarm specialization and within-swarm parent diversity.

Figure 1: Latent-agent trajectories form distinct regions by swarm and by parent program, while farthest-point filtering retains representatives from multiple clusters.

The summarizer then receives the 30 retained latent trajectories, the swarm’s parent programs, the problem statement, and an instruction to produce a complete executable program. The resulting candidates are syntax-checked, deduplicated, evaluated, and inserted into the archive. A separate parameter-tuning agent proposes up to eight single-number edits to the current best program per round. This additional operator is important because it supplies local numerical exploitation alongside the broader structural variation produced by the swarms.

Computational rationale

The paper attributes SwarmEvolve’s throughput primarily to the removal of sequential text generation from the agent phase. In the Qwen3 8B experiments, a round of 450 latent agents requires approximately four seconds for latent reasoning and 2.6 seconds for latent communication on average, whereas textual reasoning in the baselines consumes most of the GPU runtime. SwarmEvolve evaluates approximately 734 programs in a 60-minute run, compared with 158 for OpenEvolve, 99 for ShinkaEvolve, and 81 for CodeEvolve under the reported configurations.

The comparison is not merely an argument about raw model size. All methods use Qwen3 models, and the baselines are served with production-oriented inference optimizations including continuous batching, prefix caching, and CUDA graphs. SwarmEvolve’s advantage instead follows from its altered inference workload: hundreds of latent trajectories are executed in a batch, shared parent-program prefixes are reused through paged attention, and novelty filtering operates directly on hidden states rather than requiring an external embedding model and an additional LLM judge.

This yields an important implication for evolutionary search. If evaluator calls are sufficiently cheap or parallelized, then increasing the number of candidate programs can dominate increasing the amount of textual reasoning per candidate. SwarmEvolve makes this tradeoff explicit: its summarizers still decode programs, but most population members are represented by continuous latent trajectories rather than verbose textual transcripts.

Figure 2: Across 21 problems, SwarmEvolve generates more programs, obtains higher early scores, and reaches baseline 60-minute scores substantially sooner.

Empirical performance

The main benchmark contains 16 problems drawn from the AlphaEvolve problem suite and five newly designed scientific-computing tasks. The mathematical tasks cover packing, geometric point distributions, autocorrelation inequalities, uncertainty inequalities, additive combinatorics, and the dimension-11 kissing-number construction. The scientific tasks cover sparse linear systems, non-normal eigenvalue computation, viscous Burgers, convection-diffusion, and Helmholtz equations.

Scores are normalized by AlphaEvolve’s published objective values for the mathematical tasks; therefore, a score above one indicates an improvement over the corresponding AlphaEvolve construction. Scientific-computing scores measure the number of correct digits relative to high-fidelity reference solutions. Each framework receives 60 minutes, one GPU, and eight CPU evaluation cores, and each configuration is run with three seeds.

SwarmEvolve achieves the highest average score for every tested model:

Model	SwarmEvolve	OpenEvolve	ShinkaEvolve	CodeEvolve
Qwen3 8B	0.804	0.693	0.642	0.700
Qwen3 14B	0.837	0.771	0.686	0.739
Qwen3 32B-FP8	0.869	0.774	0.733	0.731

The Qwen3 8B SwarmEvolve average exceeds the averages of all three baselines using Qwen3 32B-FP8. This is a particularly strong result because it suggests that search organization and inference efficiency can compensate for a substantial model-scale difference under a fixed wall-clock budget. With Qwen3 32B-FP8, SwarmEvolve solves all five sparse-linear-system tasks to full accuracy, a result not achieved by any baseline run.

The per-problem trajectories show that the advantage is concentrated early but usually persists to the end of the hour. With Qwen3 8B, SwarmEvolve finishes with the highest score on 18 of 21 problems; the corresponding numbers are 16 of 21 for Qwen3 14B and 17 of 21 for Qwen3 32B-FP8. The paper reports that its trajectories generally rise most steeply during the first few minutes, consistent with the hypothesis that heterogeneous swarms cover multiple algorithmic directions before the baselines have generated comparable numbers of candidates.

Figure 3: With Qwen3 8B, SwarmEvolve’s score trajectories rise rapidly and finish highest on 18 of 21 problems.

Figure 4: With Qwen3 14B, SwarmEvolve finishes highest or tied for highest on 16 of 21 problems.

Figure 5: With Qwen3 32B-FP8, SwarmEvolve finishes highest or tied for highest on 17 of 21 problems.

Mathematical constructions and scientific solvers

The strongest individual claims concern several mathematical constructions. For 26-circle packing in the unit square, SwarmEvolve obtains a radius sum of approximately 2.635983, matching the reported verified best-known construction and exceeding the OpenEvolve, ShinkaEvolve, and CodeEvolve values of approximately 2.615, 2.626, and 2.614, respectively.

Figure 6: SwarmEvolve matches the verified best-known 26-circle packing with a radius sum of 2.636.

For the Heilbronn problem with 11 points in an equilateral triangle, SwarmEvolve obtains a normalized minimum triangle area of 0.0223. OpenEvolve obtains only 0.0016, while ShinkaEvolve and CodeEvolve obtain 0.0200 and 0.0222. The discrepancy is large enough to indicate that the evolutionary search can discover qualitatively different algorithmic constructions, rather than merely optimizing a common local parameterization.

Figure 7: SwarmEvolve’s 11-point construction attains a normalized minimum triangle area of 0.0223, compared with 0.0016 for OpenEvolve.

The autocorrelation tasks provide more direct evidence of improvement over AlphaEvolve’s published constructions. With Qwen3 8B, SwarmEvolve reaches approximately 0.9214 on the second autocorrelation inequality, compared with AlphaEvolve’s 0.8963, and approximately 1.4600 on the third inequality, compared with the evaluator-consistent AlphaEvolve value of 1.4688. The latter is an improvement because the objective is minimized. With Qwen3 14B and 32B-FP8, the second inequality reaches approximately 0.9314 and 0.9290, while the third reaches approximately 1.4602 and 1.4592. These comparisons require care because the evaluator’s definition for the third inequality differs from the expression used for AlphaEvolve’s published bound; the paper explicitly adjusts the reference value to maintain a consistent comparison.

On the uncertainty inequality, SwarmEvolve does not exceed AlphaEvolve’s best overall value because the benchmark restricts candidates to a Hermite-polynomial family, whereas AlphaEvolve’s best value uses Laguerre polynomials. Within the restricted Hermite family, SwarmEvolve reaches the same approximate value as AlphaEvolve’s Hermite construction with Qwen3 32B-FP8. This result illustrates that benchmark design and evaluator constraints can materially affect claims of improvement.

The scientific-computing tasks show that the method can evolve numerical algorithms rather than only static geometric constructions. For Burgers’ equation at viscosity $\nu = 0.001$, SwarmEvolve evolves a solver using 8192 grid points and a two-stage Runge–Kutta method, producing lower error away from the shock than the comparison programs. For convection-diffusion, it selects a substantially finer $768 \times 768$ grid, reducing error near the outflow boundary layers. For Helmholtz, it evolves an $818 \times 818$ discretization, improving accuracy by increasing the number of grid points per wavelength.

Figure 8: For the low-viscosity Burgers problem, SwarmEvolve’s evolved solver is most accurate away from the shock at $x=0$.

Figure 9: For convection-diffusion, the principal errors occur in the thin outflow boundary layers and along the characteristic structure.

Figure 10: For Helmholtz, increasing the grid resolution reduces the discretization error as the number of points per wavelength increases.

These results also expose a limitation of score interpretation. In several scientific tasks, the discovered improvement is primarily a choice of discretization resolution, solver, or time step. That is a valid optimization under the stated evaluator, but it conflates algorithmic innovation with computational-resource allocation. The benchmark therefore tests whether the framework can discover effective executable strategies, not whether it can derive uniformly superior numerical methods at equal computational cost.

Ablation and scaling analysis

The paper compares latent reasoning with text reasoning under matched budgets. Replacing 40 latent steps with up to 40 text tokens lowers the average score from 0.804 to 0.796; allowing up to 256 text tokens lowers it further to 0.767. Program throughput falls from 734 candidates per hour with latent reasoning to 663 with 40-token reasoning and 402 with 256-token reasoning.

The result is notable because the longer text budget does not improve search quality. The authors attribute this partly to context saturation: the summarizer’s context window is limited to 32,768 tokens, and longer reasoning messages can overwhelm the available context. The implication is not that textual reasoning is intrinsically less informative, but that, within this architecture and budget, its communication cost reduces the number of evolutionary iterations enough to outweigh any benefit from longer explicit traces.

The swarm-size ablation compares 0, 15, 45, and 75 latent agents per swarm. Larger swarms improve the best score during the early portion of search, particularly within the first 100–150 evaluated programs. However, the final 60-minute scores converge within bootstrap uncertainty intervals. This supports a more specific conclusion than “more agents are better”: larger latent populations improve early coverage, but their marginal value saturates once the shared model’s blind spots dominate.

Figure 11: Increasing latent-agent count improves early search performance, while different swarm sizes converge later in the run.

Figure 12: The gain from latent agents relative to zero-agent swarms is largest during the first approximately 100 evaluations and diminishes later.

This saturation is theoretically consequential. Since all agents share the same model and latent representation, increasing population size does not guarantee access to fundamentally new abstractions. Diversity generated by prompt variation and parent-program assignment may be sufficient for early exploration, but it cannot eliminate model-level systematic errors. The experiments therefore support scaling the population for rapid initial discovery, while indicating that model diversity or independently trained agents may be needed to extend late-stage improvement.

Limitations and open questions

The architecture imposes three direct constraints. First, all agents and summarizers must use the same LLM because KV caches are model-specific. Cross-model swarms would require a learned translation between cache representations or another compatible communication mechanism. Second, the implementation assumes standard attention. State-space, linear-attention, and hybrid architectures compress history into fixed-size states, and the paper does not provide a mechanism for merging multiple such states during latent communication. Third, KV-cache memory grows with the number of agents, latent steps, transformer layers, and model width. The largest experiments consequently require substantial GPU memory.

The empirical comparisons also leave several methodological questions open. Baseline frameworks retain their default configurations, so the results demonstrate superiority under the selected implementations rather than under an exhaustive hyperparameter-matched comparison. The benchmark uses three seeds per condition, which is useful but limited for highly stochastic evolutionary processes. Several problems are inherited from prior benchmark suites, and the normalized scores depend on evaluator definitions, feasibility tolerances, and in some cases restrictions that exclude the broader function family used by the reference method.

Most importantly, the paper does not isolate the contributions of all design choices. Latent reasoning, 450-agent population size, swarm specialization, novelty filtering, cache communication, parent-program allocation, and parameter tuning are combined in the main system. The text-reasoning ablation establishes the importance of latent computation under the tested setup, while the swarm-size study establishes an early benefit from larger populations, but the independent effect of cache-based crossover, focus taxonomy, and farthest-point selection remains less precisely identified. A central open question is therefore whether the observed gains arise primarily from latent inference itself, from the much larger number of agents, or from the interaction between the two.

Conclusion

SwarmEvolve presents a coherent systems-level redesign of LLM-guided evolutionary program search. Its principal contribution is to use latent reasoning for population-scale exploration and KV-cache transfer for inter-agent synthesis, thereby replacing expensive textual deliberation with batched continuous computation. Across 21 mathematical and scientific-computing problems, the framework evaluates substantially more candidates, reaches baseline scores several times faster, and obtains the highest mean score for Qwen3 8B, 14B, and 32B-FP8.

The strongest evidence supports the paper’s narrower claim: under fixed GPU-time budgets and executable evaluators, latent agent swarms can improve the throughput and early effectiveness of evolutionary search. The remaining question is how well the approach extends beyond shared-model, standard-attention architectures and whether its late-stage search saturation can be addressed without sacrificing the efficiency that produces its principal advantage.