---
title: Search-Time Contamination in LLM Evaluation
url: https://www.emergentmind.com/topics/search-time-contamination-stc
type: topic
---

# Search-Time Contamination in LLM Evaluation

Searching arXiv for recent papers on Search-Time Contamination and closely related contamination mechanisms in search-enabled agents.
Search-Time Contamination (STC) is an evaluation failure mode in search-based large language model (LLM) agents in which contamination occurs during inference rather than during pretraining or fine-tuning. In the formulation introduced by “Search-Time Data Contamination” [2508.13180], STC occurs when the retrieval step contains clues about a question’s answer by virtue of being derived from the evaluation set itself, or, operationally, when the retrieval step surfaces a source containing the test question or a near-duplicate alongside its answer, enabling agents to copy rather than genuinely infer or reason. The phenomenon is specific to systems that solve tasks by invoking external search or browsing tools—such as search-based LLM agents, deep research agents, and tool-using or retrieval-augmented evaluators—and it undermines benchmark validity because the benchmark ceases to measure the intended capability under online access [2508.13180].

## 1. Definition and conceptual scope

The defining statement appears as **Definition 1 (Search-Time Contamination)** in “Search-Time Data Contamination” [2508.13180]: **“Search-Time Contamination (STC) occurs in evaluating search-based LLM agents when the retrieval step contains clues about a question's answer by virtue of being derived from the evaluation set itself.”** The abstract supplies a more operational form: STC occurs when the retrieval step surfaces a source containing the test question, or a near-duplicate, together with its answer, thereby enabling copying rather than genuine inference or reasoning [2508.13180].

This definition distinguishes STC from several adjacent contamination notions. It is not the same as training-time contamination or benchmark memorization, because the model may never have memorized the benchmark at all; the leakage instead enters through retrieved context at evaluation time [2508.13180]. It is also not equivalent to retrieval augmentation in general, because retrieval is legitimate when it introduces external facts needed to solve a problem, whereas STC is illegitimate precisely because the retrieved content is derived from the benchmark itself [2508.13180]. Nor is it ordinary information lookup: looking up an external fact is normal tool use, whereas retrieving an exact test item and its benchmark-provided answer is contamination [2508.13180].

The same broad inference-time logic appears in later work on deep research systems. “Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation” [2606.05241] treats STC as contamination that happens during inference when a deep research agent uses web search and retrieves benchmark artifacts from the open web. That paper emphasizes that public-benchmark evaluation becomes fragile once agents are allowed to search online while solving test questions, because agents may retrieve public benchmark metadata, question context, or even ground-truth answers via web search [2606.05241].

## 2. Mechanism of leakage at evaluation time

The core STC mechanism is retrieval-mediated leakage. In “Search-Time Data Contamination” [2508.13180], the benchmark can remain held out from model training and still leak into the model’s context during evaluation through the agent’s retrieval tool. The model may not have memorized benchmark content in its parameters; instead, when given web access, it retrieves a page containing the benchmark question or a near-duplicate together with its answer and then uses that answer directly [2508.13180].

The empirically clearest contamination pathway identified there is public benchmark artifacts hosted on HuggingFace, including third-party uploads or forks of benchmark data [2508.13180]. The paper notes that benchmarks such as HLE and GPQA may be gated on HuggingFace, but public users can fork and re-upload them in ungated form, and the contaminated pages agents found were often third-party HuggingFace repositories rather than necessarily official benchmark hosts [2508.13180]. The authors also argue that HuggingFace is not the only source. Other possible sources include papers, blogs, mirrors, post-release commentary, pages containing dataset examples, and benchmark-derived content from online curation or discussion [2508.13180].

A closely related but broader view appears in “Trace-Level Analysis of Information Contamination in Multi-Agent Systems” [2604.27586], which studies contamination introduced after training during workflow execution over external artifacts. There, contamination enters through artifact-derived representations, extracted text, parsed tables, OCR/ASR outputs, transformed evidence, shared-memory updates, and inter-agent messages, and then propagates by reshaping routing decisions, intermediate state, and execution trajectories [2604.27586]. This is not identical to STC as defined in benchmark evaluation, but it demonstrates an inference-time contamination channel in which externally consumed information redirects tool use and reasoning traces [2604.27586].

A separate inference-time mechanism is in-context priming. “Emergent Inference-Time Semantic Contamination via In-Context Priming” [2604.04043] shows that contamination can arise purely at inference time through semantically loaded few-shot demonstrations that are unrelated to the downstream task, producing measurable distributional shifts in output content [2604.04043]. This does not define STC in the benchmark-leakage sense, but it shows that what is inserted into the prompt window at run time can alter model behavior even when the injected material is not logically relevant to the task [2604.04043].

## 3. Empirical evidence in search-based agent evaluation

The foundational empirical study in “Search-Time Data Contamination” [2508.13180] evaluates three commonly used capability benchmarks: Humanity’s Last Exam (HLE), SimpleQA, and GPQA. HLE is described there as a benchmark of **2,500 expert-curated questions** spanning “over a hundred domains from STEM to social science”; SimpleQA is a short-form factuality benchmark whose questions are solvable with knowledge existing before **December 31, 2023**; and GPQA is a graduate-level “Google-proof” QA benchmark intended to be difficult and resistant to naïve search [2508.13180].

The experiments focus primarily on Perplexity systems because the API exposes source controls useful for auditing. The main systems are **Sonar Pro**, **Sonar Reasoning Pro**, and **Sonar Deep Research** [2508.13180]. The authors also state that they experimented with Claude, Gemini, and OpenAI agents with their web search tools, but these almost never retrieved a HuggingFace link, which they hypothesize may be due to weaker ability of those retrieval tools to parse HuggingFace dataset previews and because their public APIs did not expose enough source-filtering controls for the same ablation analysis [2508.13180].

Across the three benchmarks, the paper reports that for approximately **3%** of questions, search-based agents directly found datasets with ground-truth labels on HuggingFace [2508.13180]. After HuggingFace was blocked, accuracy on the contaminated subset dropped by approximately **15%** [2508.13180]. The paper further argues that when millions of evaluation queries target the same benchmark, even small, repeated leaks can accelerate benchmark obsolescence and shorten the benchmark’s intended lifecycle [2508.13180].

The experimental setup is specified in detail. Perplexity agents were called through the public API between **May 15, 2025 and June 15, 2025** with `temperature=0.2`, `top_p=0.9`, `search_context_size='high'`, maximum output length of **32,000 tokens**, API timeout of **1 hour**, and **5 tries** before marking a sample as an API failure [2508.13180]. They used the same prompts as the agents’ offline counterparts on the capability benchmarks [2508.13180]. Judging used the official HLE Judge with **o3-mini-2025-01-31**, `temperature=1.0`, `max_completion_tokens=4096`; the SimpleQA Judge from OpenAI’s simple-evals with **gpt-4.1-2025-04-14**, `temperature=0.5`, `max_completion_tokens=2048`; and no LLM judge for GPQA. Reported performance was **pass@1 accuracy**, and GPQA was evaluated over the full set with **4 repetitions** “to lower the variance following standard practice” [2508.13180].

The contamination detection method in that paper is intentionally simple and conservative: all retrieved sources are logged, and a hard-coded substring-based HuggingFace contamination check marks a sample contaminated if any retrieved source is a HuggingFace URL whose path contains benchmark-specific substrings [2508.13180].

## 4. Taxonomies and detection methodologies

Two taxonomic schemes appear in the supplied literature. The source-oriented discussion in “Search-Time Data Contamination” [2508.13180] effectively distinguishes three pathways: **direct leakage**, where a page contains the benchmark item and its answer; **near-duplicate leakage**, where a near-duplicate question-answer pair or derivative content reveals the answer; and **non-HuggingFace public leakage**, where public web pages elsewhere reproduce or discuss benchmark content. Of these, the direct HuggingFace pathway is the one quantitatively measured there [2508.13180].

“Search-Time Contamination in Deep Research Agents” [2606.05241] introduces a more explicit severity taxonomy for deep research agents. It defines three contamination types with increasing severity: **Benchmark Metadata Leakage**, **Question-Context Leakage**, and **Explicit Answer Leakage**. In the paper’s alternative terminology, these correspond conceptually to **Retrieval-intent Exposure (RE)**, **Contextual Exposure (CE)**, and **Answer Exposure (AE)** [2606.05241].

The three types can be summarized as follows:

| Type | Definition in the literature | Detection approach |
|---|---|---|
| Benchmark Metadata Leakage / RE | Search returns URLs or snippets exposing benchmark-specific metadata | Regular-expression-based URL matching [2606.05241] |
| Question-Context Leakage / CE | Retrieved content contains the exact wording or highly specific context of the test question, but not the gold answer | Longest common substring normalized by question length [2606.05241] |
| Explicit Answer Leakage / AE | Retrieved content contains both the evaluation query and the corresponding ground-truth answer | LLM-as-a-Judge with strict exact-question and direct-answer criteria [2606.05241] |

For RE, the paper uses regular-expression-based URL matching over search results, with patterns covering common source websites and benchmark-specific websites or identifiers, including domains such as `huggingface.co/datasets`, `github.com`, `quizlet.com`, `coursehero.com`, `scribd.com/document`, and benchmark-specific strings such as `medqa`, `medmcqa`, `mmlu`, `MedXpertQA`, `HLE`, and `Humanity's Last Exam` [2606.05241]. For CE, it measures lexical overlap using the longest common substring between the question and retrieved content, normalized by question length [2606.05241]. For AE, it uses **DeepSeek V4 Pro** as an LLM judge and sets `contaminated = true` only if the retrieved content contains both extended, contiguous blocks of verbatim question text and the exact ground-truth answer or option label explicitly paired with that question [2606.05241].

The AE detector was validated against human annotation. Reported performance includes **83.3% recall** and **100% precision** on Medbullets5op, and **94.85% precision** in the paper body, or **94.87% precision** in the appendix table, on MedQA [2606.05241].

## 5. Performance inflation, trajectory effects, and benchmark validity

The central evaluative consequence of STC is score inflation. “Search-Time Data Contamination” [2508.13180] argues that if an agent’s score is partly driven by directly retrieving benchmark labels, the benchmark is no longer measuring what it is supposed to measure—reasoning, factuality, or broad capability. The paper states that under such conditions “we cannot fully trust the evaluation results as we did when evaluating the models without online access” [2508.13180].

The deep research agent study [2606.05241] makes the same claim in more explicit performance terms. It states that STC is widespread across six public benchmarks and can inflate performance by up to **4%**. The paper’s introduction specifies that this “up to 4%” inflation occurred on HLE biological and chemical subsets [2606.05241].

That study also reports full-dataset accuracy for three Tongyi settings:

| System | MedQA | MedMCQA | MMLU | MedXpertQA | HLE-149 | Medbullets5op |
|---|---:|---:|---:|---:|---:|---:|
| Qwen3-30B-A3B | 83.58% | 70.27% | 89.72% | 21.80% | 12.75% | 67.45% |
| Tongyi Deep Research* | 89.00% | 72.47% | 91.96% | 28.45% | 15.44% | 73.15% |
| Tongyi Deep Research | 91.28% | 87.34% | 94.86% | 40.61% | 24.83% | 76.17% |

These numbers show that the deep-research pipeline improves over the base model even with web search disabled, and that enabling web search increases performance further [2606.05241]. The paper’s claim is not that all of the search-enabled gain is contamination-driven, but that some of it is [2606.05241].

The most direct evidence concerns **Explicit Answer Leakage**. In turn-level before/after comparisons, accuracy jumps sharply after AE events: for example, **7.69% → 89.74%** on MedQA, **17.86% → 82.14%** on MMLU, **19.25% → 79.45%** on MedMCQA, **8% → 48%** on MedXpertQA, **20% → 100%** on HLE-149, and **0% → 80%** on Medbullets5op [2606.05241]. By contrast, **Benchmark Metadata Leakage** alone often corresponds to accuracy drops, not gains, indicating that coarse benchmark-directed search is not itself sufficient to explain inflation [2606.05241].

The same paper models trajectory dynamics with a time-varying Cox proportional hazards model,
$$
h(t|\mathbf{x}(t)) = h_0(t) \exp\left( \beta_{1} x_{1}(t) + \beta_{2} x_{2}(t) + \beta_{3} x_{3}(t) \right),
$$
where the time-dependent indicators correspond to RE, CE, and AE [2606.05241]. Reported hazard ratios for AE are consistently above 1 and often large: **2.21** on MMLU, **2.27** on MedMCQA, **4.21** on MedQA, **2.49** on MedXpertQA, **8.92** on HLE-149, and **2.20** on Medbullets5op with \(p = 0.052\) [2606.05241]. This supports the paper’s conclusion that AE is the contamination type most strongly associated with the transition to a correct answer [2606.05241].

## 6. Relation to broader contamination research

STC is part of a wider contamination literature, but it is temporally and mechanistically distinct. Training-time contamination methods address leakage into model weights. “Detecting Benchmark Contamination Through Watermarking” [2502.17259] proposes watermarking benchmarks before release and later detecting “radioactivity,” meaning traces that the text watermarks leave in the model during training [2502.17259]. That paper explicitly targets training-stage contamination rather than search-time leakage.

Similarly, “Time Travel in LLMs: Tracing Data Contamination in Large Language Models” [2308.08493] and “Data Contamination Through the Lens of Time” [2310.10628] analyze training-data contamination through guided completion probes and temporal online exposure, respectively. The latter studies benchmarks released over time and finds statistically significant trends between pass rate, GitHub popularity, and release date that it interprets as strong evidence of contamination [2310.10628]. These works are relevant because they clarify the distinction between contamination and memorization and show how public benchmark artifacts accumulate online. A plausible implication is that the same online discoverability that supports training-time exposure also increases retrievability for search-enabled agents.

Later work has emphasized how difficult contamination detection can become even in the training-time case. “On The Fragility of Benchmark Contamination Detection in Reasoning Models” [2510.02386] shows that contamination signals can be concealed after later optimization stages and that many detectors perform near random in realistic large reasoning model settings [2510.02386]. “The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation” [2605.21856] argues that generated reasoning steps can actively mask underlying memorization and introduces the Zero-CoT Probe for black-box detection [2605.21856]. Neither paper studies STC directly, but both underscore a broader methodological point: contamination can inflate benchmark performance while leaving weak or misleading surface evidence.

The inference-time contamination literature extends beyond benchmark answer retrieval. “No Attacker Needed: Unintentional Cross-User Contamination in Shared-State LLM Agents” [2604.01350] formalizes unintentional cross-user contamination in shared-state agents, where benign, locally valid artifacts are later reused outside their original context [2604.01350]. Under raw shared state, that paper reports contamination rates of **59.6%** on EHRAgent / MIMIC-III, **70.7%** on EHRAgent / eICU, and **57.4%** on MURMUR / Slack [2604.01350]. The mechanism is broader than classic STC, but it reinforces the general inference-time principle that wrong-scope retrieval or reuse can silently degrade evaluation and downstream performance.

## 7. Mitigation, reporting practice, and benchmark lifecycle

The primary response proposed in the STC literature is contamination-aware evaluation. “Search-Time Data Contamination” [2508.13180] concludes by proposing best practices for benchmark design and result reporting and publicly releases complete logs from its experiments to facilitate auditing. The paper’s broader lifecycle claim is that even a contamination rate of roughly **3%** can be consequential on frontier benchmarks such as HLE, where leaderboard margins are small enough that a **1%** shift can affect rankings [2508.13180]. When evaluations are run at scale and search logs or outputs circulate, repeated retrieval of benchmark artifacts can spread the benchmark across the web and accelerate benchmark obsolescence [2508.13180].

“Search-Time Contamination in Deep Research Agents” [2606.05241] recommends three explicit mitigation directions. The first is **isolated sandboxes**, such as ToolUniverse, where all agents retrieve from the same controlled information environment rather than the open web [2606.05241]. The second is **transparent search trajectories**, including exposure of search queries, retrieved URLs, visited pages, and intermediate evidence, so that contamination can be audited [2606.05241]. The third is **controlled benchmark access**, including private test sets, dynamically generated instances, gated dataset access, mandatory user registration, and data use agreements prohibiting redistribution [2606.05241].

These recommendations reflect a structural tension in public benchmarking. Open publication supports fairness and reproducibility, but public availability also increases the chance that benchmark artifacts will be indexed, mirrored, forked, discussed, and ultimately retrieved by evaluation-time search [2508.13180; 2606.05241]. This suggests that contamination-aware evaluation for search-enabled agents requires reporting standards beyond aggregate accuracy alone. The literature explicitly motivates releasing search logs, reporting search-on and search-off results, stratifying results by contamination status, and checking visited pages for exact question-answer pairs [2606.05241]. Such practices do not eliminate STC, but they make the benchmark’s failure modes observable.

In sum, STC names a specific breakdown in the validity of public benchmark evaluation once LLM systems gain the ability to search, browse, or otherwise assemble external context at inference time. Its canonical mechanism is answer-revealing retrieval derived from the benchmark itself; its most severe form is explicit answer leakage; and its practical significance lies in turning nominal reasoning benchmarks into partially contaminated retrieval tasks [2508.13180; 2606.05241].

Source: https://www.emergentmind.com/topics/search-time-contamination-stc