- The paper introduces Loreley, a repository-scale runtime that evolves complete Git commits using MAP-Elites archives, learned descriptors, Pareto retention, isolated worktrees, and agent-based code editing.
- In a matched seven-block Zstandard experiment, Loreley achieved a mean +0.824% holdout gain, but did not significantly outperform Sequential Champion editing (+0.960%) or clearly exceed Independent Root search (+0.502%).
- The paper shows that non-incumbent repository states were retained and reused, while identifying short search horizons, unstable PCA-based descriptors, and coarse measurement as key barriers to demonstrating causal endpoint benefits.
Overview and motivation
Loreley is a runtime that applies Quality-Diversity (QD) search to complete Git repositories rather than to standalone programs. Each candidate is a full commit descended from a frozen root; a project-supplied external evaluator is the sole authority on validity and objective values; and a MAP-Elites archive over learned repository descriptors retains multiple viable repository states for later use as edit bases or inspiration context. The paper's central empirical question is whether retaining non-incumbent repository states improves final held-out performance relative to two simpler context-selection policies: Sequential Champion editing, which concentrates the budget on the best state seen so far, and Independent Root proposals, which always restart from the root.
The paper explicitly distinguishes two testable claims. Engagement means the archive actually retains non-incumbent states and later samples them. Efficacy means that this retention improves a held-out endpoint under a fixed budget. The paper reports positive evidence for the first and a negative result for the second, and treats this separation as its main methodological contribution.
System design
Loreley separates target-independent search infrastructure (scheduling, agent sessions, worktrees, embeddings, archive state, lineage, worker coordination via PostgreSQL, Redis, and Dramatiq) from a target evaluator that owns source scope, build/test commands, workloads, objective definitions, precision, and acceptance gates. Key components:
- Agent variation over commits. A planning agent receives the frozen campaign goal plus bounded base history and evidence; a coding agent edits the base in an isolated Git worktree, producing exactly one result commit. Inspirations are supplied as context only, never checked out; inspiration edges record information supplied, not Git ancestry.
- Descriptors. Repository vectors are uniform means of file embeddings (cached by blob SHA), projected by per-island whitened PCA into a MAP-Elites grid. PCA is refit on a fixed cadence, with atomic archive rebuilds from records retained immediately before refit.
- Pareto cells. Each cell holds a bounded non-dominated set under ϵ-dominance with crowding-distance overflow (NSGA-II style). Sampling is uniform over occupied cells and then uniform over retained members, decoupling cell count from sampling probability.
- Islands and migration. Islands maintain independent archives and projections; migration supplies a foreign elite as an inspiration without changing the commit's parent.
Matched Zstandard experiment
The controlled study ran seven paired blocks on one Linux ARM64 host, each block giving the three policies 48 attempted physical candidate jobs (1,008 total; 948 successful) from the same frozen Zstandard root, with planning via gpt-5.6-sol and coding via gpt-5.6-luna. The primary endpoint is the 48-job holdout compression-throughput ratio of a validation-selected fixed winner, analyzed with paired log differences, BCa intervals, exact sign-flip tests, and Holm correction.
| Policy |
Mean (%) |
Median (%) |
Range (%) |
≥0.5% gain |
| Independent Root |
+0.502 |
+0.412 |
+0.195 to +1.331 |
2/7 |
| Loreley QD |
+0.824 |
+0.739 |
+0.062 to +1.390 |
6/7 |
| Sequential Champion |
+0.960 |
+0.819 |
+0.514 to +1.789 |
7/7 |
Neither paired contrast establishes a QD advantage: QD was −0.135% relative to Sequential Champion (95% BCa [−0.556,+0.161]%; Holm p=.547) and +0.320% relative to Independent Root ([−0.082,+0.686]%; Holm p=.375). The paper is careful to note the study establishes neither a QD advantage nor equivalence, and that the ordering is specific to Zstandard, the two model routes, root-only initialization, and the 48-job horizon. Leave-one-block-out sensitivities move the QD–Sequential point estimate from −0.226% to +0.020%, underscoring instability with seven pairs. Checkpoint curves (descriptive) show QD leading at 8–16 jobs and Sequential overtaking it by 40–48.
Archive engagement without endpoint benefit
The engagement condition was satisfied in all seven blocks. Under a retrospective one-incumbent rule, four of seven final QD winners had a non-incumbent state in their primary-parent ancestry; including inspiration edges raises this to six of seven, though the paper notes inspiration supply does not demonstrate that the context caused the edit. Across 67 nodes in combined dependency graphs, 49 were non-incumbents later used on an outgoing edge. Blocks used a median of 20 distinct primary parents, and final archives occupied 9–12 of 64 cells. In three of four primary-parent lineages, at least one descendant exceeded the incumbent score at the ancestor's admission time.
The paper draws the implication directly: the stepping-stone mechanism operated as designed, but this within-stream evidence is not an estimate of causal benefit, and the data cannot distinguish an insufficient horizon from an ineffective descriptor, archive, or sampling rule.
Capability cases
Three earlier campaigns demonstrate that the configured system produces cumulative, multi-file, generation-4 improvements: +6.75% on markdown-it-py (frozen before validation), −0.135%0 on python-pathspec (a post-hoc replacement after the registered candidate failed an allocation gate), and −0.135%1 compression throughput on Zstandard, replicated at −0.135%2 on the original holdout and −0.135%3 on a fresh sealed corpus. The paper retains each case's selection qualification rather than pooling them into a single inferential claim, and notes the capability cases used human-written seeds, so they do not estimate initialization effects.
Descriptor behavior and measurement precision
Diagnostics on the earlier Zstandard archive reveal a structural concern: unprojected repository vectors are extremely close (median pairwise cosine distance −0.135%4), a consequence of uniformly averaging file embeddings when patches touch a small fraction of files. Centered variance was still structured (participation ratio 5.31), and the first three whitened components explained 71.0% of variance, but 43.1% of admitted candidates would change cells under the final PCA relative to their admission projection — cumulative churn from the non-stationary projection, which the paper flags as making archive membership path-dependent. Measurement screening also proved coarse: among training Top-10 finalists, compression lower bounds spanned only 0.276 percentage points, and the validation winner had ranked tenth on training.
Loreley builds on MAP-Elites (Mouret et al., 2015), multi-objective QD (MOME), and learned descriptors (AURORA) (Mouret et al., 2015, Khanh et al., 6 Jul 2026). Its closest antecedents are RHO's HELIX optimizer, which evolves repository states on a per-instance coverage frontier (Elmaaroufi et al., 15 Jun 2026), and CktEvo, which evolves Verilog repositories with hand-designed descriptors and scalar elites (Shi et al., 10 Feb 2026). LLM-driven program search systems such as ELM, FunSearch, AlphaEvolve, and CodeEvolve operate on algorithmic artifacts rather than buildable repositories (Lehman et al., 2022, Novikov et al., 16 Jun 2025, Assumpção et al., 15 Oct 2025). The negative-leaning primary result is consistent with controlled findings that simple baselines can match more elaborate search strategies (Zou et al., 17 May 2026, Gideoni et al., 18 Feb 2026).
Limitations and open questions
The comparison covers one repository, one host, one pair of model routes, and seven blocks; the paired intervals admit modest effects in either direction. The 48-job horizon is the most consequential limitation — QD's mechanism is designed for delayed payoffs, and the experiment may have ended before retained branches could compound, though the paper states plainly that this explanation is plausible rather than established. The study compares one configured policy and does not isolate the learned descriptor, Pareto retention, inspiration sampling, or concurrency as components. The non-stationary PCA refit rule means lineage diagnostics are conditional on the implemented rebuild policy, and no full-history replay was performed. The paper identifies the decisive next experiment: a longer matched curve on Zstandard plus a second repository under the same three policies.
Conclusion
Loreley demonstrates a working repository-scale QD runtime with per-cell Pareto retention over learned whole-repository descriptors, and its lineage records confirm that non-incumbent branches were retained and resampled. Three capability campaigns show the system can produce multi-generation, multi-file performance improvements. The controlled matched experiment, however, did not establish an endpoint advantage over Sequential Champion at 48 jobs, and the positive point estimate over Independent Root remained statistically uncertain. Whether longer horizons or better descriptors can convert archive activity into a measurable benefit over strong sequential baselines remains the paper's principal open question.