Search-Time Contamination in LLM Evaluation

• Search-Time Contamination (STC) is an evaluation failure mode where search-based LLM agents retrieve benchmark artifacts, leading them to copy answers instead of reasoning independently.
• STC undermines benchmark validity by inflating performance metrics when agents inadvertently access test questions or near-duplicates containing ground-truth answers during evaluation.
• Researchers mitigate STC by employing isolated sandboxes, transparent search logs, and controlled benchmark access to ensure evaluations accurately measure true reasoning abilities.

Searching arXiv for papers on Search-Time Contamination and closely related contamination mechanisms in search-enabled agents. Search-Time Contamination (STC) is an evaluation failure mode in search-based LLM agents in which contamination occurs during inference rather than during pretraining or fine-tuning. In the formulation introduced by “Search-Time Data Contamination” (Han et al., 12 Aug 2025), STC occurs when the retrieval step contains clues about a question’s answer by virtue of being derived from the evaluation set itself, or, operationally, when the retrieval step surfaces a source containing the test question or a near-duplicate alongside its answer, enabling agents to copy rather than genuinely infer or reason. The phenomenon is specific to systems that solve tasks by invoking external search or browsing tools—such as search-based LLM agents, deep research agents, and tool-using or retrieval-augmented evaluators—and it undermines benchmark validity because the benchmark ceases to measure the intended capability under online access (Han et al., 12 Aug 2025).

1. Definition and conceptual scope

The defining statement appears as Definition 1 (Search-Time Contamination) in “Search-Time Data Contamination” (Han et al., 12 Aug 2025): “Search-Time Contamination (STC) occurs in evaluating search-based LLM agents when the retrieval step contains clues about a question's answer by virtue of being derived from the evaluation set itself.” The abstract supplies a more operational form: STC occurs when the retrieval step surfaces a source containing the test question, or a near-duplicate, together with its answer, thereby enabling copying rather than genuine inference or reasoning (Han et al., 12 Aug 2025).

This definition distinguishes STC from several adjacent contamination notions. It is not the same as training-time contamination or benchmark memorization, because the model may never have memorized the benchmark at all; the leakage instead enters through retrieved context at evaluation time (Han et al., 12 Aug 2025). It is also not equivalent to retrieval augmentation in general, because retrieval is legitimate when it introduces external facts needed to solve a problem, whereas STC is illegitimate precisely because the retrieved content is derived from the benchmark itself (Han et al., 12 Aug 2025). Nor is it ordinary information lookup: looking up an external fact is normal tool use, whereas retrieving an exact test item and its benchmark-provided answer is contamination (Han et al., 12 Aug 2025).

The same broad inference-time logic appears in later work on deep research systems. “Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation” (Wang et al., 3 Jun 2026) treats STC as contamination that happens during inference when a deep research agent uses web search and retrieves benchmark artifacts from the open web. That paper emphasizes that public-benchmark evaluation becomes fragile once agents are allowed to search online while solving test questions, because agents may retrieve public benchmark metadata, question context, or even ground-truth answers via web search (Wang et al., 3 Jun 2026).

2. Mechanism of leakage at evaluation time

The core STC mechanism is retrieval-mediated leakage. In “Search-Time Data Contamination” (Han et al., 12 Aug 2025), the benchmark can remain held out from model training and still leak into the model’s context during evaluation through the agent’s retrieval tool. The model may not have memorized benchmark content in its parameters; instead, when given web access, it retrieves a page containing the benchmark question or a near-duplicate together with its answer and then uses that answer directly (Han et al., 12 Aug 2025).

The empirically clearest contamination pathway identified there is public benchmark artifacts hosted on HuggingFace, including third-party uploads or forks of benchmark data (Han et al., 12 Aug 2025). The paper notes that benchmarks such as HLE and GPQA may be gated on HuggingFace, but public users can fork and re-upload them in ungated form, and the contaminated pages agents found were often third-party HuggingFace repositories rather than necessarily official benchmark hosts (Han et al., 12 Aug 2025). The authors also argue that HuggingFace is not the only source. Other possible sources include papers, blogs, mirrors, post-release commentary, pages containing dataset examples, and benchmark-derived content from online curation or discussion (Han et al., 12 Aug 2025).

A closely related but broader view appears in “Trace-Level Analysis of Information Contamination in Multi-Agent Systems” (Mazhar et al., 30 Apr 2026), which studies contamination introduced after training during workflow execution over external artifacts. There, contamination enters through artifact-derived representations, extracted text, parsed tables, OCR/ASR outputs, transformed evidence, shared-memory updates, and inter-agent messages, and then propagates by reshaping routing decisions, intermediate state, and execution trajectories (Mazhar et al., 30 Apr 2026). This is not identical to STC as defined in benchmark evaluation, but it demonstrates an inference-time contamination channel in which externally consumed information redirects tool use and reasoning traces (Mazhar et al., 30 Apr 2026).

A separate inference-time mechanism is in-context priming. “Emergent Inference-Time Semantic Contamination via In-Context Priming” (Abram, 5 Apr 2026) shows that contamination can arise purely at inference time through semantically loaded few-shot demonstrations that are unrelated to the downstream task, producing measurable distributional shifts in output content (Abram, 5 Apr 2026). This does not define STC in the benchmark-leakage sense, but it shows that what is inserted into the prompt window at run time can alter model behavior even when the injected material is not logically relevant to the task (Abram, 5 Apr 2026).

3. Empirical evidence in search-based agent evaluation

The foundational empirical study in “Search-Time Data Contamination” (Han et al., 12 Aug 2025) evaluates three commonly used capability benchmarks: Humanity’s Last Exam (HLE), SimpleQA, and GPQA. HLE is described there as a benchmark of 2,500 expert-curated questions spanning “over a hundred domains from STEM to social science”; SimpleQA is a short-form factuality benchmark whose questions are solvable with knowledge existing before December 31, 2023; and GPQA is a graduate-level “Google-proof” QA benchmark intended to be difficult and resistant to naïve search (Han et al., 12 Aug 2025).

The experiments focus primarily on Perplexity systems because the API exposes source controls useful for auditing. The main systems are Sonar Pro, Sonar Reasoning Pro, and Sonar Deep Research (Han et al., 12 Aug 2025). The authors also state that they experimented with Claude, Gemini, and OpenAI agents with their web search tools, but these almost never retrieved a HuggingFace link, which they hypothesize may be due to weaker ability of those retrieval tools to parse HuggingFace dataset previews and because their public APIs did not expose enough source-filtering controls for the same ablation analysis (Han et al., 12 Aug 2025).

Across the three benchmarks, the paper reports that for approximately 3% of questions, search-based agents directly found datasets with ground-truth labels on HuggingFace (Han et al., 12 Aug 2025). After HuggingFace was blocked, accuracy on the contaminated subset dropped by approximately 15% (Han et al., 12 Aug 2025). The paper further argues that when millions of evaluation queries target the same benchmark, even small, repeated leaks can accelerate benchmark obsolescence and shorten the benchmark’s intended lifecycle (Han et al., 12 Aug 2025).

The experimental setup is specified in detail. Perplexity agents were called through the public API between May 15, 2025 and June 15, 2025 with temperature=0.2, top_p=0.9, search_context_size='high', maximum output length of 32,000 tokens, API timeout of 1 hour, and 5 tries before marking a sample as an API failure (Han et al., 12 Aug 2025). They used the same prompts as the agents’ offline counterparts on the capability benchmarks (Han et al., 12 Aug 2025). Judging used the official HLE Judge with o3-mini-2025-01-31, temperature=1.0, max_completion_tokens=4096; the SimpleQA Judge from OpenAI’s simple-evals with gpt-4.1-2025-04-14, temperature=0.5, max_completion_tokens=2048; and no LLM judge for GPQA. Reported performance was pass@1 accuracy, and GPQA was evaluated over the full set with 4 repetitions “to lower the variance following standard practice” (Han et al., 12 Aug 2025).

The contamination detection method in that paper is intentionally simple and conservative: all retrieved sources are logged, and a hard-coded substring-based HuggingFace contamination check marks a sample contaminated if any retrieved source is a HuggingFace URL whose path contains benchmark-specific substrings (Han et al., 12 Aug 2025).

4. Taxonomies and detection methodologies

Two taxonomic schemes appear in the supplied literature. The source-oriented discussion in “Search-Time Data Contamination” (Han et al., 12 Aug 2025) effectively distinguishes three pathways: direct leakage, where a page contains the benchmark item and its answer; near-duplicate leakage, where a near-duplicate question-answer pair or derivative content reveals the answer; and non-HuggingFace public leakage, where public web pages elsewhere reproduce or discuss benchmark content. Of these, the direct HuggingFace pathway is the one quantitatively measured there (Han et al., 12 Aug 2025).

“Search-Time Contamination in Deep Research Agents” (Wang et al., 3 Jun 2026) introduces a more explicit severity taxonomy for deep research agents. It defines three contamination types with increasing severity: Benchmark Metadata Leakage, Question-Context Leakage, and Explicit Answer Leakage. In the paper’s alternative terminology, these correspond conceptually to Retrieval-intent Exposure (RE), Contextual Exposure (CE), and Answer Exposure (AE) (Wang et al., 3 Jun 2026).

The three types can be summarized as follows:

Type	Definition in the literature	Detection approach
Benchmark Metadata Leakage / RE	Search returns URLs or snippets exposing benchmark-specific metadata	Regular-expression-based URL matching (Wang et al., 3 Jun 2026)
Question-Context Leakage / CE	Retrieved content contains the exact wording or highly specific context of the test question, but not the gold answer	Longest common substring normalized by question length (Wang et al., 3 Jun 2026)
Explicit Answer Leakage / AE	Retrieved content contains both the evaluation query and the corresponding ground-truth answer	LLM-as-a-Judge with strict exact-question and direct-answer criteria (Wang et al., 3 Jun 2026)

For RE, the paper uses regular-expression-based URL matching over search results, with patterns covering common source websites and benchmark-specific websites or identifiers, including domains such as huggingface.co/datasets, github.com, quizlet.com, coursehero.com, scribd.com/document, and benchmark-specific strings such as medqa, medmcqa, mmlu, [MedXpertQA](https://www.emergentmind.com/topics/medxpertqa), HLE, and Humanity's Last Exam (Wang et al., 3 Jun 2026). For CE, it measures lexical overlap using the longest common substring between the question and retrieved content, normalized by question length (Wang et al., 3 Jun 2026). For AE, it uses DeepSeek V4 Pro as an LLM judge and sets contaminated = true only if the retrieved content contains both extended, contiguous blocks of verbatim question text and the exact ground-truth answer or option label explicitly paired with that question (Wang et al., 3 Jun 2026).

The AE detector was validated against human annotation. Reported performance includes 83.3% recall and 100% precision on Medbullets5op, and 94.85% precision in the paper body, or 94.87% precision in the appendix table, on MedQA (Wang et al., 3 Jun 2026).

5. Performance inflation, trajectory effects, and benchmark validity

The central evaluative consequence of STC is score inflation. “Search-Time Data Contamination” (Han et al., 12 Aug 2025) argues that if an agent’s score is partly driven by directly retrieving benchmark labels, the benchmark is no longer measuring what it is supposed to measure—reasoning, factuality, or broad capability. The paper states that under such conditions “we cannot fully trust the evaluation results as we did when evaluating the models without online access” (Han et al., 12 Aug 2025).

The deep research agent study (Wang et al., 3 Jun 2026) makes the same claim in more explicit performance terms. It states that STC is widespread across six public benchmarks and can inflate performance by up to 4%. The paper’s introduction specifies that this “up to 4%” inflation occurred on HLE biological and chemical subsets (Wang et al., 3 Jun 2026).

That study also reports full-dataset accuracy for three Tongyi settings:

System	MedQA	MedMCQA	MMLU	MedXpertQA	HLE-149	Medbullets5op
Qwen3-30B-A3B	83.58%	70.27%	89.72%	21.80%	12.75%	67.45%
Tongyi Deep Research*	89.00%	72.47%	91.96%	28.45%	15.44%	73.15%
Tongyi Deep Research	91.28%	87.34%	94.86%	40.61%	24.83%	76.17%

These numbers show that the deep-research pipeline improves over the base model even with web search disabled, and that enabling web search increases performance further (Wang et al., 3 Jun 2026). The paper’s claim is not that all of the search-enabled gain is contamination-driven, but that some of it is (Wang et al., 3 Jun 2026).

The most direct evidence concerns Explicit Answer Leakage. In turn-level before/after comparisons, accuracy jumps sharply after AE events: for example, 7.69% → 89.74% on MedQA, 17.86% → 82.14% on MMLU, 19.25% → 79.45% on MedMCQA, 8% → 48% on MedXpertQA, 20% → 100% on HLE-149, and 0% → 80% on Medbullets5op (Wang et al., 3 Jun 2026). By contrast, Benchmark Metadata Leakage alone often corresponds to accuracy drops, not gains, indicating that coarse benchmark-directed search is not itself sufficient to explain inflation (Wang et al., 3 Jun 2026).

The same paper models trajectory dynamics with a time-varying Cox proportional hazards model,

$$h(t|\mathbf{x}(t)) = h_0(t) \exp\left( \beta_{1} x_{1}(t) + \beta_{2} x_{2}(t) + \beta_{3} x_{3}(t) \right),$$

where the time-dependent indicators correspond to RE, CE, and AE (Wang et al., 3 Jun 2026). Reported hazard ratios for AE are consistently above 1 and often large: 2.21 on MMLU, 2.27 on MedMCQA, 4.21 on MedQA, 2.49 on MedXpertQA, 8.92 on HLE-149, and 2.20 on Medbullets5op with $p = 0.052$ (Wang et al., 3 Jun 2026). This supports the paper’s conclusion that AE is the contamination type most strongly associated with the transition to a correct answer (Wang et al., 3 Jun 2026).

6. Relation to broader contamination research

STC is part of a wider contamination literature, but it is temporally and mechanistically distinct. Training-time contamination methods address leakage into model weights. “Detecting Benchmark Contamination Through Watermarking” (Sander et al., 24 Feb 2025) proposes watermarking benchmarks before release and later detecting “radioactivity,” meaning traces that the text watermarks leave in the model during training (Sander et al., 24 Feb 2025). That paper explicitly targets training-stage contamination rather than search-time leakage.

Similarly, “Time Travel in LLMs: Tracing Data Contamination in LLMs” (Golchin et al., 2023) and “Data Contamination Through the Lens of Time” (Roberts et al., 2023) analyze training-data contamination through guided completion probes and temporal online exposure, respectively. The latter studies benchmarks released over time and finds statistically significant trends between pass rate, GitHub popularity, and release date that it interprets as strong evidence of contamination (Roberts et al., 2023). These works are relevant because they clarify the distinction between contamination and memorization and show how public benchmark artifacts accumulate online. A plausible implication is that the same online discoverability that supports training-time exposure also increases retrievability for search-enabled agents.

Later work has emphasized how difficult contamination detection can become even in the training-time case. “On The Fragility of Benchmark Contamination Detection in Reasoning Models” (Wang et al., 30 Sep 2025) shows that contamination signals can be concealed after later optimization stages and that many detectors perform near random in realistic large reasoning model settings (Wang et al., 30 Sep 2025). “The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation” (Lan et al., 21 May 2026) argues that generated reasoning steps can actively mask underlying memorization and introduces the Zero-CoT Probe for black-box detection (Lan et al., 21 May 2026). Neither paper studies STC directly, but both underscore a broader methodological point: contamination can inflate benchmark performance while leaving weak or misleading surface evidence.

The inference-time contamination literature extends beyond benchmark answer retrieval. “No Attacker Needed: Unintentional Cross-User Contamination in Shared-State LLM Agents” (Yang et al., 1 Apr 2026) formalizes unintentional cross-user contamination in shared-state agents, where benign, locally valid artifacts are later reused outside their original context (Yang et al., 1 Apr 2026). Under raw shared state, that paper reports contamination rates of 59.6% on EHRAgent / MIMIC-III, 70.7% on EHRAgent / eICU, and 57.4% on MURMUR / Slack (Yang et al., 1 Apr 2026). The mechanism is broader than classic STC, but it reinforces the general inference-time principle that wrong-scope retrieval or reuse can silently degrade evaluation and downstream performance.

7. Mitigation, reporting practice, and benchmark lifecycle

The primary response proposed in the STC literature is contamination-aware evaluation. “Search-Time Data Contamination” (Han et al., 12 Aug 2025) concludes by proposing best practices for benchmark design and result reporting and publicly releases complete logs from its experiments to facilitate auditing. The paper’s broader lifecycle claim is that even a contamination rate of roughly 3% can be consequential on frontier benchmarks such as HLE, where leaderboard margins are small enough that a 1% shift can affect rankings (Han et al., 12 Aug 2025). When evaluations are run at scale and search logs or outputs circulate, repeated retrieval of benchmark artifacts can spread the benchmark across the web and accelerate benchmark obsolescence (Han et al., 12 Aug 2025).

“Search-Time Contamination in Deep Research Agents” (Wang et al., 3 Jun 2026) recommends three explicit mitigation directions. The first is isolated sandboxes, such as ToolUniverse, where all agents retrieve from the same controlled information environment rather than the open web (Wang et al., 3 Jun 2026). The second is transparent search trajectories, including exposure of search queries, retrieved URLs, visited pages, and intermediate evidence, so that contamination can be audited (Wang et al., 3 Jun 2026). The third is controlled benchmark access, including private test sets, dynamically generated instances, gated dataset access, mandatory user registration, and data use agreements prohibiting redistribution (Wang et al., 3 Jun 2026).

These recommendations reflect a structural tension in public benchmarking. Open publication supports fairness and reproducibility, but public availability also increases the chance that benchmark artifacts will be indexed, mirrored, forked, discussed, and ultimately retrieved by evaluation-time search (Han et al., 12 Aug 2025, Wang et al., 3 Jun 2026). This suggests that contamination-aware evaluation for search-enabled agents requires reporting standards beyond aggregate accuracy alone. The literature explicitly motivates releasing search logs, reporting search-on and search-off results, stratifying results by contamination status, and checking visited pages for exact question-answer pairs (Wang et al., 3 Jun 2026). Such practices do not eliminate STC, but they make the benchmark’s failure modes observable.

In sum, STC names a specific breakdown in the validity of public benchmark evaluation once LLM systems gain the ability to search, browse, or otherwise assemble external context at inference time. Its canonical mechanism is answer-revealing retrieval derived from the benchmark itself; its most severe form is explicit answer leakage; and its practical significance lies in turning nominal reasoning benchmarks into partially contaminated retrieval tasks (Han et al., 12 Aug 2025, Wang et al., 3 Jun 2026).