Beyond Retrieval: A Multitask Benchmark and Model for Code Search

Abstract: Code search has usually been evaluated as first-stage retrieval, even though production systems rely on broader pipelines with reranking and developer-style queries. Existing benchmarks also suffer from data contamination, label noise, and degenerate binary relevance. In this paper, we introduce \textsc{CoREB}, a contamination-limited, multitask \underline{co}de \underline{r}etrieval and r\underline{e}ranking \underline{b}enchmark, together with a fine-tuned code reranker, that goes beyond retrieval to cover the full code search pipeline. \textsc{CoREB} is built from counterfactually rewritten LiveCodeBench problems in five programming languages and delivered as timed releases with graded relevance judgments. We benchmark eleven embedding models and five rerankers across three tasks: text-to-code, code-to-text, and code-to-code. Our experiments reveal that: \circone code-specialised embeddings dominate code-to-code retrieval (${\sim}2{\times}$ over general encoders), yet no single model wins all three tasks; \circtwo short keyword queries, the format closest to real developer search, collapse every model to near-zero nDCG@10; \circthree off-the-shelf rerankers are task-asymmetric, with a 12-point swing on code-to-code and no baseline net-positive across all tasks; \circfour our fine-tuned \textsc{CoREB-Reranker} is the first to achieve consistent gains across all three tasks. The data and model are released.

• The paper introduces CoREB, a contamination-limited benchmark with 5,087 queries, 3,414 code items, five programming languages, graded relevance labels, and retrieval-plus-reranking evaluation.
• The results show specialized 0.5B code models can outperform general 8B encoders, while all tested models nearly collapse on short developer-style queries and exceed 55% hard-negative intrusion in text-to-code search.
• The authors release CoREB-Reranker, the only evaluated reranker that improves all three retrieval directions, while showing that poor reranker choices can reduce code-to-text performance by more than 20 points.

Motivation and problem statement

Code search is typically evaluated as first-stage retrieval, yet production systems operate as retrieve-then-rerank pipelines serving developer-style queries. The paper identifies four deficiencies in current evaluation practice. First (D1), no existing code retrieval benchmark evaluates or supplies a reranking stage. Second (D2), contamination: CodeSearchNet and derivatives constitute over 85% of CoIR's corpus volume and have served as pre-training data for models that are then evaluated on them; the authors cite evidence of up to 100% metric inflation from duplication and 15–25% test/train near-duplication in CodeSearchNet-style data. Third (D3), label noise: roughly 51% of CoSQA pairs are mismatched under functional verification (the authors' own inspection of 80 pairs finds ~60% problematic), and CodeSearchNet's verbatim-docstring matching is acknowledged by its original authors as "overly simplistic." Three of ten CoIR datasets do not even test code retrieval. Fourth (D4), degenerate relevance structure: all CoIR datasets assign exactly one binary-relevant document per query with no hard negatives, collapsing nDCG@$k$ and MRR into redundant hit-or-miss signals.

The paper introduces CoREB, a contamination-limited, multitask code retrieval and reranking benchmark built from counterfactually rewritten LiveCodeBench problems across five languages (Python, Java, C++, Go, Ruby), released as timed snapshots with graded relevance judgments, together with a fine-tuned code reranker covering the full pipeline.

Benchmark construction

CoREB is distributed in two releases (v202602: 167 seed problems from LCB contests Sep 2024–Jan 2025; v202603: 175 problems from Jan–Apr 2025), yielding 5,087 queries, 3,414 code corpus items, and 1,710 text corpus items aggregated. The pipeline has five stages:

1. Seed sourcing from LiveCodeBench, mirroring its temporal-filtering model.
2. Counterfactual rewriting: GPT-o1 rewrites each statement (entity replacement, domain shifting, synonym substitution) while preserving the formal specification; rewrites are human-verified and validated by re-running the full test suite. Solutions are generated in five languages by Gemini 3 Flash and Claude Sonnet 4.5 and executed against the rewritten tests; all 3,414 candidates are retained regardless of correctness (1,065 verified-correct). 3–5. Query generation, retrieval formulation, and release as JSONL with graded qrels.

A controlled $2{\times}2$ annotation study validates the rewriting intervention. Gemini 3 Flash loses 8.6 points Pass@1 on v202602 (43%→34%) and 6.5 on v202603 after rewriting, consistently across all five languages. Claude Sonnet 4.5 shows a striking asymmetry: −12.3 points on v202602 but +2.0 on v202603, consistent with its January 2025 reliable knowledge cutoff. This dissociation demonstrates that leakage is model- and release-dependent and unpredictable without controlled testing — the paper's central justification for counterfactual annotation.

Relevance labels are derived programmatically from execution outcomes, eliminating human annotation noise. The three-level scheme assigns relevance=2 to verified-correct solutions (or the original description for code-to-text), relevance=1 to same-problem hard negatives (failed solutions, LLM-generated noise descriptions generated via four perturbation strategies), and treats unjudged items as easy negatives. Evaluation uses relevance_level=2, so ranking a plausible-but-incorrect item above a true positive is penalized — strictly harder than BEIR/CoIR/MTEB conventions. Across releases, 68% of text-to-code queries have two or more correct solutions and code-to-code queries average 2.2 valid cross-language translations.

Retrieval results

Eleven embedding models (0.5B–8B parameters, plus closed-API GemEmb-2) are evaluated on text-to-code, code-to-text, and code-to-code with nDCG@10 and Recall@10 under the graded protocol. GemEmb-2 achieves the best overall nDCG@10 (0.637); C2LLM-7B is the strongest open-weight model (0.629) and leads text-to-code. Rankings are stable across releases (four of the top-5 shared; per-task differences within 0.03).

Key findings:

• No single model wins all tasks. Task-averaged nDCG@10 spans 0.73 (code-to-text) to 0.39 (text-to-code), with code-to-code the most discriminative. The Qwen3 family scores competitively on code-to-text yet collapses on code-to-code (e.g., Qwen3-Emb-8B at 0.320 vs. Jina-code-0.5b at 0.677).
• Scaling is non-monotonic. Qwen3-Emb-8B trails Qwen3-Emb-0.6B overall, and F2LLM-1.7B underperforms F2LLM-4B, indicating training-data composition mediates size–quality relationships.
• Specialization beats scale. A 0.5B code-trained model outperforms general-purpose 8B encoders, with the premium concentrated on code-to-code (~2×). C2LLM-0.5B outperforms the 16× larger Qwen3-Emb-8B by over 12 points overall.
• Short keyword queries collapse every model. On the Search subtask (~19-token developer-style queries), all eleven models drop to near-zero nDCG@10 — two orders of magnitude below the long-query Canonical subtask — while sharing the same corpus. Neither scaling nor reranking closes this gap.
• Language bias tracks corpus coverage. Language-constrained queries score lower than language-agnostic ones, with Ruby and Go lagging consistently (up to 0.33 points below Python-level performance).
• Hard negatives expose a hidden failure mode. The hard-negative intrusion rate exceeds 55% on text-to-code for every model — more than half of applicable queries rank a failed solution above the correct one. Notably, stronger retrievers can exhibit higher intrusion because they retrieve more same-problem content, creating more mis-ranking opportunities.

An appendix also documents an anchor-exclusion correction for code-to-code evaluation: without removing the byte-identical anchor from the ranked list, Recall is inflated and nDCG@1/MRR structurally zeroed. With exclusion, the nDCG–Recall divergence shows that cross-language ranking precision, not coverage, is the bottleneck (GemEmb-2 reaches Recall@10 = 1.000 but nDCG@10 = 0.698).

Reranking results

Reranking the top-128 candidates from C2LLM-7B with four off-the-shelf baselines reveals severe task asymmetry. All four degrade code-to-text (−3.2% to −22.4%), none improves text-to-code (−8.3% to −0.1%), and only Qwen3-Reranker-4B gains on code-to-code (+3.3%). A 12-point swing separates the best and worst baseline on the same task; no baseline is net-positive across all three tasks.

The authors fine-tune Qwen3-Reranker-4B via LoRA on a 3.1M-sample mixture (CoREB v202602, CodeSearchNet, APPS, CosQA, CodeFeedback) formatted as generative yes/no triplets with doubled positives and one easy plus one hard negative per positive. Problem-level disjointness guarantees the v202603 evaluation is fully out-of-sample. The resulting CoREB-Reranker is the only reranker net-positive on all three tasks, establishing the first complete two-stage pipeline benchmarked end-to-end.

Practical guidance

Two open-weight checkpoints lie on the Pareto frontier of quality versus parameter count: C2LLM-0.5B and C2LLM-7B. C2LLM-0.5B reaches 95.9% of the best open-weight score at 7% of the parameter count, and 0.5B specialized models deliver roughly 13× more nDCG per billion parameters than 8B general-purpose ones. Reranker selection is high-stakes — a poorly chosen reranker can cost over 20 points on code-to-text — and short-query retrieval remains the single largest unsolved gap, with query expansion (HyDE, Query2doc) suggested as the most promising mitigation.

Limitations

The paper concedes several scope constraints plainly. The corpus derives exclusively from competitive programming, so retrieval over enterprise or library code is unrepresented. Only five languages are covered. All queries are LLM-generated rather than drawn from real developer search logs, which limits ecological validity precisely in the short-query setting where the benchmark identifies its most striking failure. Evaluation is offline and single-turn; interactive and multi-turn code search are not addressed. Additionally, the memorization study covers two models and two releases, and the intrusion-rate metric is conditioned on queries where both a positive and a hard negative reach the top-10, which complicates cross-model comparison.

Conclusion

CoREB contributes a contamination-limited, executionally labeled, multitask benchmark spanning both stages of the code search pipeline, alongside a released reranker that is the first to improve consistently across all three retrieval directions. Its empirical findings — specialization outweighing scale, universal collapse on short keyword queries, systematic low-resource language bias, and pervasive hard-negative intrusion — collectively indicate that current code embedding models, despite strong long-query performance, remain far from solving realistic code search.