RankEvolve: Evolving Lexical Retrieval

• RankEvolve is a framework that evolves lexical retrieval algorithms as executable Python programs using LLM-guided mutation and evaluator feedback.
• It starts from BM25 and QL-Dir seed programs, iteratively refining ranking logic with metrics like nDCG@10 and Recall@100 over multiple datasets.
• The approach reveals effectiveness-efficiency trade-offs, prompting future work on latency optimization and multi-objective evolutionary tuning.

RankEvolve is a framework for automating the discovery of lexical retrieval algorithms by evolving executable ranking code with a LLM, an evaluator, and evolutionary population management. In its primary formulation, candidate rankers are represented as Python programs and iteratively mutated, recombined, and selected based on retrieval performance across 12 datasets from BEIR and BRIGHT; the process starts from two seed programs, BM25 and query likelihood with Dirichlet smoothing, and yields evolved algorithms that are novel, effective, and show promising transfer to the full BEIR and BRIGHT benchmarks as well as TREC DL 19 and 20 (Nian et al., 18 Feb 2026).

1. Definition and scope

RankEvolve addresses a specific limitation of first-stage retrieval research. Retrieval algorithms such as BM25 and query likelihood with Dirichlet smoothing remain strong and efficient first-stage rankers, yet improvements have mostly relied on parameter tuning and human intuition. RankEvolve replaces manual formula design with evaluator-guided LLM program evolution, treating the ranking algorithm itself—not merely its hyperparameters—as the search object (Nian et al., 18 Feb 2026).

The framework builds directly on OpenEvolve, an open-source implementation of AlphaEvolve. Its central methodological claim is that full retrieval algorithms can be evolved as programs: tokenization, indexing, query representation, and scoring are exposed to search, while evaluator interfaces and dataset-loading infrastructure remain fixed. Within that setup, the LLM is not the deployed retriever; it is the mutation and recombination operator over a population of executable rankers (Nian et al., 18 Feb 2026).

Two seed programs define the initial search directions. The first is classic BM25,

$$\text{BM25}(q,d) = \sum_{t \in q} \text{IDF}(t)\, \frac{\text{tf}(t,d)\,(k_1+1)}{\text{tf}(t,d) + k_1\Bigl(1-b + b\frac{|d|}{\text{avgdl}}\Bigr)},$$

with

$$\text{IDF}(t) = \log\frac{N-\text{df}(t)+0.5}{\text{df}(t)+0.5}.$$

The second is query likelihood with Dirichlet smoothing,

$$\text{QL-Dir}(q,d) = \sum_{t \in q} \log\left( \frac{\text{tf}(t,d) + \mu\,P(t\mid C)}{|d|+\mu} \right), \qquad P(t\mid C) = \frac{\text{tf}(t,C)}{|C|}.$$

From these seeds, RankEvolve discovers BM25$^\star$ and QL-Dir$^\star$, two new lexical rankers with more elaborate tokenization, weighting, and normalization structures (Nian et al., 18 Feb 2026).

2. Program representation and evolutionary mechanics

Each candidate in RankEvolve is a self-contained Python program of roughly 300 lines implementing tokenization, index building, scoring, query processing, and retrieval. The search space is therefore programmatic rather than parametric. For the BM25 seed, the evolvable components are document representation, query representation, and scoring function; for the QL-Dir seed, the collection LLM is also exposed. By contrast, the evaluator interface and supporting infrastructure are not editable, which ensures that mutated candidates remain executable under a common protocol (Nian et al., 18 Feb 2026).

The framework studies three seed designs for BM25. In the constrained variant, the BM25 formula is fixed and the LLM can only adjust hyperparameters or select among small predefined options. In the composable variant, BM25 is decomposed into modular primitives such as IDF, TF normalization, and length normalization, while the overall pipeline remains fixed. In the freeform variant, only coarse interfaces are fixed; inside them, the LLM can rearrange and redesign the logic more freely. The most sophisticated rankers emerge from the freeform regime (Nian et al., 18 Feb 2026).

Code edits are expressed in a SEARCH/REPLACE diff format. Mutation and recombination are conditioned on population context. For each step, RankEvolve selects a parent from one of $K=3$ islands using a mixture of exploration, exploitation through a MAP-Elites archive, and weighted sampling proportional to fitness. The prompt includes the parent program, its performance summary, $T=4$ best programs from the island, $S=4$ random programs, and recorded metrics such as nDCG@10, Recall@100, and latency. The LLM used in the reported experiments is GPT-5.2 with temperature $0.85$ (Nian et al., 18 Feb 2026).

Population management combines an island model with MAP-Elites. Each island maintains a $B \times B$ archive with $$\text{IDF}(t) = \log\frac{N-\text{df}(t)+0.5}{\text{df}(t)+0.5}.$$0, indexed by code complexity and diversity measured by edit distance from the population. A candidate is inserted into a cell if the cell is empty or if its fitness exceeds that of the incumbent. Every $$\text{IDF}(t) = \log\frac{N-\text{df}(t)+0.5}{\text{df}(t)+0.5}.$$1 iterations, the top $$\text{IDF}(t) = \log\frac{N-\text{df}(t)+0.5}{\text{df}(t)+0.5}.$$2 fraction of programs from each island migrate to adjacent islands unless already migrated. This design is intended to preserve both local refinement and cross-island diversity (Nian et al., 18 Feb 2026).

3. Evaluation protocol and optimization objective

RankEvolve optimizes retrieval effectiveness over 12 evolution datasets and evaluates generalization on 16 held-out datasets. The evolution-time datasets comprise six BEIR subsets—ArguAna, FiQA, NFCorpus, SciFact, SciDocs, and TREC-COVID—and six BRIGHT subsets—Biology, Earth Science, Economics, Pony, StackOverflow, and TheoremQA. Held-out evaluation uses the remaining BEIR and BRIGHT subsets together with TREC DL 2019 and 2020, with MS MARCO as the corpus for TREC Deep Learning (Nian et al., 18 Feb 2026).

The evaluator measures nDCG@10, Recall@100, and latency. The optimization target is explicitly weighted toward first-stage recall:

$$\text{IDF}(t) = \log\frac{N-\text{df}(t)+0.5}{\text{df}(t)+0.5}.$$3

averaged across the 12 evolution datasets. Latency is recorded but not included in the fitness. This choice reflects the intended role of the evolved systems as first-stage rankers feeding downstream rerankers (Nian et al., 18 Feb 2026).

The baseline families are lexical. For BM25-style retrieval, the comparisons include BM25, BM25+, and BM25-adpt. For language-model retrieval, the comparisons include QL-Dir and QL-JM. BM25+ adds a lower bound for term-frequency normalization, BM25-adpt uses adaptive $$\text{IDF}(t) = \log\frac{N-\text{df}(t)+0.5}{\text{df}(t)+0.5}.$$4 values per term, and QL-JM uses Jelinek–Mercer smoothing. The evolved systems are compared against these baselines on macro-averaged held-out performance, with significance assessed by per-query paired $$\text{IDF}(t) = \log\frac{N-\text{df}(t)+0.5}{\text{df}(t)+0.5}.$$5-tests at $$\text{IDF}(t) = \log\frac{N-\text{df}(t)+0.5}{\text{df}(t)+0.5}.$$6 (Nian et al., 18 Feb 2026).

4. Structure of the evolved rankers

BM25$$\text{IDF}(t) = \log\frac{N-\text{df}(t)+0.5}{\text{df}(t)+0.5}.$$7 is a multi-channel lexical ranker with a gated mixture of base tokens, prefixes, bigrams, and character trigrams. Its top-level score is

$$\text{IDF}(t) = \log\frac{N-\text{df}(t)+0.5}{\text{df}(t)+0.5}.$$8

where prefixes truncate tokens to the first five characters, bigrams concatenate adjacent tokens, and micro tokens are rolling character 3-grams. The micro channel is activated by a sigmoid gate over the query’s mean IDF,

$$\text{IDF}(t) = \log\frac{N-\text{df}(t)+0.5}{\text{df}(t)+0.5}.$$9

This architecture lets the system combine approximate stemming, phrase-level evidence, and subword matching while suppressing micro-token noise on common-word queries (Nian et al., 18 Feb 2026).

All channels share a scoring core

$$\text{QL-Dir}(q,d) = \sum_{t \in q} \log\left( \frac{\text{tf}(t,d) + \mu\,P(t\mid C)}{|d|+\mu} \right), \qquad P(t\mid C) = \frac{\text{tf}(t,C)}{|C|}.$$0

with

$$\text{QL-Dir}(q,d) = \sum_{t \in q} \log\left( \frac{\text{tf}(t,d) + \mu\,P(t\mid C)}{|d|+\mu} \right), \qquad P(t\mid C) = \frac{\text{tf}(t,C)}{|C|}.$$1

The term weight

$$\text{QL-Dir}(q,d) = \sum_{t \in q} \log\left( \frac{\text{tf}(t,d) + \mu\,P(t\mid C)}{|d|+\mu} \right), \qquad P(t\mid C) = \frac{\text{tf}(t,C)}{|C|}.$$2

with

$$\text{QL-Dir}(q,d) = \sum_{t \in q} \log\left( \frac{\text{tf}(t,d) + \mu\,P(t\mid C)}{|d|+\mu} \right), \qquad P(t\mid C) = \frac{\text{tf}(t,C)}{|C|}.$$3

acts as a soft stopword filter. The auxiliary multipliers implement coverage,

$$\text{QL-Dir}(q,d) = \sum_{t \in q} \log\left( \frac{\text{tf}(t,d) + \mu\,P(t\mid C)}{|d|+\mu} \right), \qquad P(t\mid C) = \frac{\text{tf}(t,C)}{|C|}.$$4

specificity via clipped PMI,

$$\text{QL-Dir}(q,d) = \sum_{t \in q} \log\left( \frac{\text{tf}(t,d) + \mu\,P(t\mid C)}{|d|+\mu} \right), \qquad P(t\mid C) = \frac{\text{tf}(t,C)}{|C|}.$$5

coordination,

$$\text{QL-Dir}(q,d) = \sum_{t \in q} \log\left( \frac{\text{tf}(t,d) + \mu\,P(t\mid C)}{|d|+\mu} \right), \qquad P(t\mid C) = \frac{\text{tf}(t,C)}{|C|}.$$6

a rare-term anchor,

$$\text{QL-Dir}(q,d) = \sum_{t \in q} \log\left( \frac{\text{tf}(t,d) + \mu\,P(t\mid C)}{|d|+\mu} \right), \qquad P(t\mid C) = \frac{\text{tf}(t,C)}{|C|}.$$7

and a logarithmic document-length dampener,

$$\text{QL-Dir}(q,d) = \sum_{t \in q} \log\left( \frac{\text{tf}(t,d) + \mu\,P(t\mid C)}{|d|+\mu} \right), \qquad P(t\mid C) = \frac{\text{tf}(t,C)}{|C|}.$$8

The resulting design combines double-log TF saturation, soft stopword suppression, coordination, specificity, and gentler length normalization than standard BM25 (Nian et al., 18 Feb 2026).

QL-Dir$$\text{QL-Dir}(q,d) = \sum_{t \in q} \log\left( \frac{\text{tf}(t,d) + \mu\,P(t\mid C)}{|d|+\mu} \right), \qquad P(t\mid C) = \frac{\text{tf}(t,C)}{|C|}.$$9 preserves a language-model structure while substantially enriching the background model and per-term controls. Its top-level score is

$^\star$0

where $^\star$1 is the set of unique query terms. The collection model is tempered and interpolated:

$^\star$2

$^\star$3

$^\star$4

Term frequency is adaptively saturated by

$^\star$5

The base term score is

$^\star$6

with a gate

$^\star$7

Query-term weights are

$^\star$8

Negative evidence is retained through a leaky rectifier, absent terms receive a downscaled Dirichlet missing-term penalty, query coverage is rewarded by

$^\star$9

and document length is regularized by a symmetric log-length prior,

$^\star$0

Relative to standard QL-Dir, this yields a richer background model, adaptive TF saturation, soft penalties, explicit coverage, and a learned length prior (Nian et al., 18 Feb 2026).

5. Held-out performance and efficiency

On held-out data, both evolved rankers improve over their seeds and over standard lexical variants. The gains are especially pronounced on BRIGHT and on Recall@100, which is consistent with the optimization objective. BM25$^\star$1 improves BRIGHT from 10.55 / 32.11 to 11.79 / 37.51 in nDCG@10 / R@100, and QL-Dir$^\star$2 improves BRIGHT from 9.52 / 32.48 to 11.42 / 36.33. On BEIR held-out subsets, BM25$^\star$3 reaches 72.43 Recall@100 and QL-Dir$^\star$4 reaches 46.46 / 70.22. On TREC DL 2019 and 2020, BM25$^\star$5 reaches 64.57 / 47.10 and QL-Dir$^\star$6 reaches 60.68 / 47.96 (Nian et al., 18 Feb 2026).

Suite	Method	nDCG@10 / R@100
BRIGHT	BM25	10.55 / 32.11
BRIGHT	BM25-adpt	11.05 / 34.67
BRIGHT	BM25$^\star$7	11.79 / 37.51
BRIGHT	QL-Dir	9.52 / 32.48
BRIGHT	QL-JM	10.17 / 31.08
BRIGHT	QL-Dir$^\star$8	11.42 / 36.33
BEIR	BM25	48.16 / 70.95
BEIR	BM25$^\star$9	47.90 / 72.43
BEIR	QL-Dir	44.15 / 68.72
BEIR	QL-Dir$K=3$0	46.46 / 70.22
TREC DL	BM25	62.16 / 46.02
TREC DL	BM25$K=3$1	64.57 / 47.10
TREC DL	QL-Dir	57.03 / 46.69
TREC DL	QL-Dir$K=3$2	60.68 / 47.96

The latency profile is less favorable. Indexing cost remains close across systems, but freeform evolution increases query latency substantially. BM25 has average query latency 56.72 ms/query, while the final freeform BM25$K=3$3 reaches 648.89 ms/query; QL-Dir increases from 178.26 to 325.41 ms/query in its evolved form. Early or constrained variants show better effectiveness–latency trade-offs, and the reported search trajectory indicates that late-stage gains may coincide with significant complexity growth (Nian et al., 18 Feb 2026).

Method	Index ms/doc	Query ms/q
BM25	1.79	56.72
BM25$K=3$4 Constrained	1.85	58.50
BM25$K=3$5 Freeform (177)	2.37	171.52
BM25$K=3$6 Freeform (293)	2.81	648.89
QL-Dir	2.02	178.26
QL-Dir$K=3$7 Freeform (182)	2.02	325.41

6. Broader conceptual usage

In its strictest sense, RankEvolve denotes the LLM-driven evolution framework for lexical retrieval algorithms (Nian et al., 18 Feb 2026). A broader interpretive reading is also supported by several neighboring works, which describe rank-centric adaptation or evolution under different formalizations.

In dynamic search, RLIRank models a search session as a reinforcement learning process in which the state is the ordered list of already ranked documents together with the current query representation, actions select documents or update the query, and a stacked LSTM value network estimates a ranking metric for partial rankings. The accompanying explanation states that “RankEvolve” can be thought of as a dynamic, reinforcement-learning-based ranking system that updates its results as a session unfolds, and that RLIRank is almost exactly such a system (Zhou et al., 2021).

In evolutionary optimization, a rank-based adaptive mutation scheme for genetic algorithms assigns mutation probability by chromosome rank rather than raw fitness,

$K=3$8

so that the best chromosome has zero mutation probability and the poorest has $K=3$9. The accompanying discussion explicitly connects this method to a rank-driven evolutionary framework one might label “RankEvolve,” emphasizing invariance to fitness distribution and improved behavior on multimodal landscapes and TSP (Basak, 2021).

In evolutionary reinforcement learning, off-policy ranking for Augmented Random Search replaces on-policy ranking with a local fitness approximation computed from behavior-policy data, reducing environment interactions while preserving the ranking step central to ES. That work is likewise presented as a conceptual blueprint for a RankEvolve algorithm whose defining feature is an efficient off-policy ranking mechanism for candidate policies (R et al., 2022).

In black-box bilevel optimization, URA-CMA-ES accelerates nested search by approximating only the ranking of the upper-level value function through partial lower-level optimization and early stopping based on Kendall’s $T=4$0. The explanatory text characterizes this as a “rank-based evolution” of upper-level candidates and explicitly treats it as a RankEvolve concept instantiated by Upper-Level Ranking Approximation CMA-ES (Ong et al., 3 Apr 2026).

In multi-agent evaluation, $T=4$1-Rank constructs a Markov chain over pure strategy profiles under high-$T=4$2 evolutionary dynamics and ranks agents by stationary mass aggregated over profiles. The same discussion notes that a system called RankEvolve can naturally be viewed as “ranking by evolution,” operationalizing the principle formalized by $T=4$3-Rank and Markov-Conley chains (Omidshafiei et al., 2019).

A distinct but adjacent use appears in benchmark construction. EvoEval studies how leaderboard rankings change when an existing coding benchmark is evolved into more difficult, creative, subtle, compositional, and tool-based variants, and the exposition uses “RankEvolve” to denote the evolution of model rankings as evaluation itself evolves (Xia et al., 2024). This suggests that the term can denote either evolving rankers, evolving rankings, or rank-based evolution, depending on context.

7. Limitations and future directions

The primary limitations of RankEvolve as a retrieval framework are computational and operational. Evolution requires repeated evaluation over 12 datasets for hundreds of steps—300 for the BM25 seed and 200 for the QL-Dir seed—together with repeated LLM calls. The system is therefore an offline research procedure rather than a lightweight online tuner (Nian et al., 18 Feb 2026).

A second limitation is efficiency at inference time. The best freeform programs are substantially slower than their seeds, especially on query latency. Because latency is visible to the LLM but excluded from the optimization target, search is free to trade efficiency for effectiveness. The reported results therefore support multi-objective extensions in which latency, code complexity, or both become explicit fitness terms (Nian et al., 18 Feb 2026).

A third limitation is dependence on seed design. Constrained seeds behave like intelligent hyperparameter tuning; composable seeds expose modular rewrites; freeform seeds provide the richest search space but also generate the most elaborate programs. This suggests that the quality of the evolvable interface strongly shapes the attainable algorithmic discoveries (Nian et al., 18 Feb 2026).

The reported future directions are correspondingly structural. They include multi-objective optimization that incorporates latency or code complexity, extension beyond lexical retrieval to dense retrieval and learned sparse representations, and more systematic seed design. A plausible implication is that RankEvolve can be read both as a concrete 2026 retrieval framework and as a broader research pattern in which rank-based signals, evaluator feedback, and evolutionary search are used to discover or adapt algorithms across information retrieval, reinforcement learning, optimization, and benchmark design (Nian et al., 18 Feb 2026).