FreshStack: Dynamic IR & RAG Benchmarks

• FreshStack is a reusable framework that automatically constructs realistic IR and RAG benchmarks by processing community-generated technical questions and documentation.
• It defines atomic 'nuggets' from Q&A and employs a five-stage pipeline to ensure high precision, recall, and groundedness in retrieval evaluations.
• The framework supports dynamic, time-aware updates and contamination mitigation, making it ideal for rapidly evolving technical ecosystems.

Searching arXiv for the specified FreshStack papers and related benchmark/drift work to ground the article. FreshStack is a general, reusable framework for automatically building realistic IR and RAG evaluation benchmarks from community-asked technical questions and answers. It targets niche, fast-growing topics, constructs corpora from public technical documentation and code repositories, and evaluates retrieval at the nugget level, where nuggets are short atomic facts extracted from answers. The framework is designed to measure both relevance and diversity, to reduce contamination risk by focusing on recent topics and fresh corpora, and to support periodic refresh as technical ecosystems evolve (Thakur et al., 17 Apr 2025, Kuissi et al., 4 Mar 2026).

1. Motivation and benchmark scope

FreshStack is motivated by three deficiencies in widely used retrieval benchmarks for technical work. First, many benchmarks lack realistic, open-ended questions: datasets such as Natural Questions and TriviaQA focus on short extractive answers or crowd-sourced prompts, while even MS MARCO queries are brief and “inserted into a search box,” rather than resembling the long, code-heavy questions encountered in developer workflows. Second, many tasks are artificially easy and do not encode the grounding difficulties that arise in practical RAG systems. Third, static and unspecialized benchmarks become stale, risk contamination, and can overfit to repeated leaderboard use, as in the case of BEIR (Thakur et al., 17 Apr 2025).

Within that problem setting, FreshStack is explicitly positioned around niche, fast-growing developer topics, complex long questions, dynamic updating, and high challenge level. The benchmark is contrasted with CQADupstack, CodeSearchNet, COIR, Stack Overflow-QA, CodeRAG-Bench, Neural Code Search, SWE-Bench, and also with general-purpose collections such as MS MARCO, BEIR, and Natural Questions. A key distinction is that FreshStack retrieves canonical documentation and code chunks from GitHub repositories to ground answers, rather than retrieving forum answers directly. This design places it closer to realistic documentation-centric retrieval and RAG for technical domains.

The framework also adopts a time-aware view of benchmarking. A later longitudinal study argues that technical corpora differ from static Cranfield-style collections because APIs are deprecated, code and documentation trees are reorganized, and relevant content migrates across repositories. In that setting, benchmark freshness is not solely a matter of retaining old labels; it depends on re-instantiating corpora and re-judging support as the ecosystem changes (Kuissi et al., 4 Mar 2026).

2. Construction pipeline and nugget formalization

FreshStack is described as a holistic five-stage framework. The stages are: automatic corpus collection from GitHub code and documentation; nugget generation from Stack Overflow question–answer pairs; pooling with diverse retrieval techniques in both inference and oracle settings; nugget-level support judgments using LLMs; and post-processing filters to ensure that remaining questions have document support for all nuggets (Thakur et al., 17 Apr 2025).

Corpus construction begins from Stack Overflow tags. Co-occurring tags with the main topic guide repository inclusion, and repositories are manually verified per tag. Topics are selected from tags introduced in 2023 onward, Stack Overflow data is drawn from the October 2024 XML dump, and GitHub repositories are cloned from the latest branch at collection time. All files in each repository are parsed as a tree. Text and code files are chunked into up to 4096 tokens each, while non-text formats such as images, videos, .bin, .csv, audio, or unrecognized formats are skipped. Document identifiers encode the GitHub filepath and chunk details, prefixed by repository name to disambiguate common filenames such as LICENSE. Chunks from all repositories are then combined into a single corpus per topic.

The core annotation unit is the nugget. FreshStack defines a nugget as “a core concept or atomic fact essential in a system’s response.” Nuggets are automatically generated with GPT-4o using grading-notes style prompts, with the accepted Stack Overflow answer as the primary source and the question used as additional context. The intent is to decompose verbose technical answers into compact factual units that can later be linked to supporting documentation chunks.

Nugget quality is calibrated on 60 randomly sampled LangChain questions by an ML expert. The reported formulas are:

$$\mathrm{Precision} = \frac{|\mathrm{Nuggets}| - \sum B}{|\mathrm{Nuggets}|}$$

$$\mathrm{Recall} = \frac{|\mathrm{Nuggets}| - \sum B}{|\mathrm{Nuggets}| - \sum B + C}$$

$$\mathrm{Groundedness} = \frac{|\mathrm{Nuggets}| - \sum A}{|\mathrm{Nuggets}|}$$

where $A$ marks hallucinations, $B$ marks minor or redundant nuggets, and $C$ is the count of additional nuggets needed to cover all key ideas. On LangChain, the reported results are Precision 90.1%, Recall 96.6%, and Groundedness 96.4%. For sampled positive nugget–document pairs, judgment quality is reported as Relevant 71.7%, Partially Relevant 11.7%, and Non-Relevant 16.6%. Support is binary at the nugget level: a document judged relevant supports at least one nugget. To reduce cost, GPT-4o judges support for the list of nuggets and the top-$k$ retrieved documents, with $k \le 20$, in a single call. Post-processing then removes unsupported questions and questions with unsupported nuggets; on average, 11.8% of questions are removed for having no relevant documents and 34.2% for containing at least one unsupported nugget (Thakur et al., 17 Apr 2025).

3. Topic selection, repositories, and dataset composition

FreshStack builds five topic-specific benchmarks selected for recency, niche focus, and sufficient activity, defined as at least 50 Stack Overflow posts since 2023. The topics are LangChain, Yolo v7 & v8, Laravel 10 & 11, Angular 16, 17 & 18, and Godot4. Their GitHub sources are explicit.

For LangChain, the corpus includes langchain-ai/langchain, langchain-ai/langchainjs, langchain-ai/langchain-nextjs-template, chroma-core/chroma, openai/openai-cookbook, openai/openai-python, run-llama/llama_index, Azure-Samples/openai, Azure-Samples/azure-search-openai-demo, and huggingface/transformers. For Yolo v7 & v8, the repositories are ultralytics/ultralytics, ultralytics/docs, pytorch/pytorch, WongKinYiu/yolov7, and opencv/opencv. For Laravel 10 & 11, the corpus contains laravel/framework, laravel/laravel, laravel/laravel.com, laravel/docs, laravel/breeze, livewire/livewire, php/php-src, php/doc-en, and php/web-php. For Angular 16, 17 & 18, the sources are angular/angular, angular/components, angular/angular-cli, and microsoft/TypeScript. For Godot4, the repositories are godotengine/godot, godotengine/godot-demo-projects, godotengine/godot-docs, godotengine/godot-website, GDQuest/learn-gdscript, and dotnet/csharplang (Thakur et al., 17 Apr 2025).

The benchmark statistics emphasize long questions, frequent code, and multiple relevant supporting chunks per query.

Topic	Queries / docs / repos	Nugget, length, code, and support statistics
LangChain	203 / 49,514 / 10	N/Q 3.1; query 473.4; answer 233.8; code in queries 83.3%; code in answers 62.1%; rel docs/nugget 5.7; rel docs/question 10.9
Yolo v7 & v8	57 / 27,207 / 5	N/Q 3.5; query 497.1; answer 191.7; code in queries 70.2%; code in answers 71.9%; rel docs/nugget 3.9; rel docs/question 7.4
Laravel 10 & 11	184 / 52,351 / 9	N/Q 3.0; query 474.4; answer 155.5; code in queries 43.5%; code in answers 51.1%; rel docs/nugget 3.2; rel docs/question 6.0
Angular 16, 17 & 18	129 / 117,288 / 4	N/Q 3.2; query 463.3; answer 215.1; code in queries 69.8%; code in answers 57.4%; rel docs/nugget 4.4; rel docs/question 8.7
Godot4	99 / 25,482 / 6	N/Q 3.3; query 350.4; answer 263.0; code in queries 52.5%; code in answers 52.5%; rel docs/nugget 2.9; rel docs/question 5.9

These statistics support the claim that FreshStack is not centered on short keyword queries. Average query lengths range from 350.4 to 497.1 tokens, and code occurs in 43.5% to 83.3% of queries depending on topic. The framework also incorporates contamination-mitigation choices: it prioritizes accepted answers from the October 2024 Stack Overflow XML dump, selects topics introduced in 2023 onward, and is designed to refresh topics over time, retire stale or contaminated ones, and add new domains (Thakur et al., 17 Apr 2025).

4. Retrieval architecture, pooling, and evaluation formalism

FreshStack distinguishes between inference settings and oracle settings. In inference, systems rely only on the question and can use LLM-generated query expansions. Two expansions are defined: GPT-4o Sub-Questions, which decomposes the question into a few synthetic sub-questions, and GPT-4o Closed-Book Answer, which generates a synthetic answer in a HyDE-style setup. In oracle settings, used only for dataset construction and headroom analysis, retrieval can use the accepted Stack Overflow answer or the concatenation of all GPT-4o-generated nuggets. These are denoted Stack Overflow Answer and Stack Overflow Nuggets (Thakur et al., 17 Apr 2025).

The first-stage baselines are BM25 via Pyserini, BGE with the retriever BAAI/bge-multilingual-gemma2 built on Gemma-2 (9B) with embedding size 3584 and 8K context length, E5 Mistral (7B) with intfloat/e5-mistral-7b-instruct, a Mistral 7B backbone with 32 layers and embedding size 4096, and Voyage-large-2, a proprietary general-purpose embedding with 16K context length and 1536 embedding size. FreshStack also defines an ensemble fusion over the four first-stage models by normalizing scores per model and summing over the union of the top-100 documents per model:

$$s(d) = \sum_{m \in \{\mathrm{BM25}, \mathrm{BGE}, \mathrm{E5}, \mathrm{Voyage}\}} z_m(\mathrm{score}_m(d))$$

where $z_m(\cdot)$ denotes model-specific score normalization over the model’s top-100 results for the query. Documents not retrieved by a model contribute $$\mathrm{Recall} = \frac{|\mathrm{Nuggets}| - \sum B}{|\mathrm{Nuggets}| - \sum B + C}$$0 for that model term. Reranking is performed by Voyage AI rerank-2 with 16K context, applied listwise to the top-50 first-stage results. Preliminary GPT-4o-mini zero-shot listwise reranking is reported to have underperformed because of formatting issues with long concatenations, and LLM reranking is left to future work.

The evaluation formalism is nugget-centric. Rather than treating query–document relevance as a monolithic label, FreshStack asks whether a document factually supports each nugget. A document is relevant for a question if it supports at least one nugget. Diversity is then measured by how many distinct nuggets are covered in the ranking. The reported metrics are $$\mathrm{Recall} = \frac{|\mathrm{Nuggets}| - \sum B}{|\mathrm{Nuggets}| - \sum B + C}$$1-nDCG@10, Coverage@20, and Recall@50. Coverage@20 is defined as

$$\mathrm{Recall} = \frac{|\mathrm{Nuggets}| - \sum B}{|\mathrm{Nuggets}| - \sum B + C}$$2

and Recall@50 is defined generically as

$$\mathrm{Recall} = \frac{|\mathrm{Nuggets}| - \sum B}{|\mathrm{Nuggets}| - \sum B + C}$$3

with relevance determined by nugget support. FreshStack constructs a test evaluation dataset rather than train/dev splits. Pooling to build judgment sets uses both inference and oracle settings, but final evaluation of retrieval models strictly uses the question only. GPT-4o is run at temperature 0.1 for nugget generation and support judgments (Thakur et al., 17 Apr 2025).

5. Empirical findings, benchmark behavior, and limitations

FreshStack reports substantial gaps between out-of-the-box retrieval and oracle-assisted retrieval. During pooling, oracle techniques significantly outperform inference expansions across $$\mathrm{Recall} = \frac{|\mathrm{Nuggets}| - \sum B}{|\mathrm{Nuggets}| - \sum B + C}$$4-nDCG@10, Coverage@20, and Recall@50 on all five topics. Stack Overflow Nuggets with fusion is the strongest oracle pooling technique in four of five topics: LangChain achieves $$\mathrm{Recall} = \frac{|\mathrm{Nuggets}| - \sum B}{|\mathrm{Nuggets}| - \sum B + C}$$5-nDCG@10 0.519, Coverage@20 0.881, Recall@50 0.655; Yolo v7 & v8 reaches 0.601, 0.876, 0.825; Angular 16, 17 & 18 reaches 0.544, 0.881, 0.756; and Godot4 reaches 0.476, 0.814, 0.719. For Laravel 10 & 11, the best oracle result is reported as $$\mathrm{Recall} = \frac{|\mathrm{Nuggets}| - \sum B}{|\mathrm{Nuggets}| - \sum B + C}$$6-nDCG@10 0.566, Coverage@20 0.888, Recall@50 0.818, with a tie of best oracle method with the Answer variant in some metrics. BM25 often attains the best $$\mathrm{Recall} = \frac{|\mathrm{Nuggets}| - \sum B}{|\mathrm{Nuggets}| - \sum B + C}$$7-nDCG@10 among individual models in oracle settings, except on Godot4, underscoring the continued importance of lexical matching (Thakur et al., 17 Apr 2025).

Among inference expansions, GPT-4o Sub-Questions with fusion is the best inference pooling technique for four topics: LangChain (0.322, 0.708, 0.475), Laravel (0.478, 0.763, 0.662), Angular (0.428, 0.817, 0.584), and Godot4 (0.290, 0.598, 0.526). GPT-4o Closed-Book Answer with fusion excels only on Yolo v7 & v8, at 0.356, 0.686, 0.578. The authors suspect possible contamination for this topic.

In the main evaluation, where only the question is available at test time, all out-of-the-box retrievers and reranking configurations trail the oracle baselines substantially across $$\mathrm{Recall} = \frac{|\mathrm{Nuggets}| - \sum B}{|\mathrm{Nuggets}| - \sum B + C}$$8-nDCG@10, Coverage@20, and Recall@50. Fusion outperforms individual retrievers across all metrics and topics except $$\mathrm{Recall} = \frac{|\mathrm{Nuggets}| - \sum B}{|\mathrm{Nuggets}| - \sum B + C}$$9-nDCG@10 on Godot4. Reranking exhibits non-uniform behavior: Voyage AI rerank-2 improves over BM25 on all topics, but for dense retrievers it improves metrics for LangChain, Yolo v7 & v8, and Godot4 while reducing performance on Laravel 10 & 11 and Angular 16, 17 & 18. The paper’s plots show reductions but do not provide exact numeric deltas. This indicates that rerankers are not uniformly beneficial and may vary by domain or programming language.

FreshStack’s experimental center of gravity is retrieval rather than answer generation. LLMs are used for nugget generation, support judgments, and inference-time query expansion, but the benchmark does not report ROUGE, BLEU, or comprehensive answer-generation metrics. The paper does, however, report that oracle context helps an LLM generator generate a high-quality RAG answer. A plausible implication is that nugget-grounded retrieval quality is treated as the principal bottleneck before end-to-end answer evaluation.

The paper also states several limitations. LLM judgments are imperfect; among sampled positive nugget–document pairs, 16.6% were misjudged non-relevant, reflecting ambiguity or partial-support issues. Repository selection is still partly manual. Domain coverage is restricted to five topics spanning ML, CV, backend, front-end, and game development. Contamination may increase as models evolve, so the framework’s value depends on continuous refresh. Reranker gains are inconsistent. Proposed future work includes more code-focused retrievers such as voyage-3, CodeT5+, CodeRankEmbed, and Jina-Code-v2; rerankers such as CodeRankLLM; answer evaluation with nugget-based recall; and automation of repository discovery and dataset refresh. The benchmark is available at https://fresh-stack.github.io; Stack Overflow XML data is CC BY-SA, while GitHub repositories retain their own licenses (Thakur et al., 17 Apr 2025).

6. Temporal drift, re-judging, and longitudinal reliability

The later study “Still Fresh? Evaluating Temporal Drift in Retrieval Benchmarks” examines FreshStack under temporal corpus drift by building two independent LangChain snapshots, one from main-branch commits prior to October 2024 and one prior to October 2025. The study reuses the original 203 LangChain Stack Overflow questions and 640 nuggets from FreshStack, reconstructs the corpora, reruns pooling, and re-judges nugget support. In this longitudinal setup, files are chunked to a maximum of 2048 tokens, and each chunk is uniquely identified by repository name, relative file path, and byte offsets (Kuissi et al., 4 Mar 2026).

The study formalizes the relevant set at time $$\mathrm{Groundedness} = \frac{|\mathrm{Nuggets}| - \sum A}{|\mathrm{Nuggets}|}$$0 as $$\mathrm{Groundedness} = \frac{|\mathrm{Nuggets}| - \sum A}{|\mathrm{Nuggets}|}$$1, where a document is relevant if it supports at least one nugget for query $$\mathrm{Groundedness} = \frac{|\mathrm{Nuggets}| - \sum A}{|\mathrm{Nuggets}|}$$2. Nugget-level support is written as $$\mathrm{Groundedness} = \frac{|\mathrm{Nuggets}| - \sum A}{|\mathrm{Nuggets}|}$$3 for nugget $$\mathrm{Groundedness} = \frac{|\mathrm{Nuggets}| - \sum A}{|\mathrm{Nuggets}|}$$4, and a query is fully supported at time $$\mathrm{Groundedness} = \frac{|\mathrm{Nuggets}| - \sum A}{|\mathrm{Nuggets}|}$$5 if every nugget has at least one supporting document:

$$\mathrm{Groundedness} = \frac{|\mathrm{Nuggets}| - \sum A}{|\mathrm{Nuggets}|}$$6

Drift magnitude between times $$\mathrm{Groundedness} = \frac{|\mathrm{Nuggets}| - \sum A}{|\mathrm{Nuggets}|}$$7 and $$\mathrm{Groundedness} = \frac{|\mathrm{Nuggets}| - \sum A}{|\mathrm{Nuggets}|}$$8 is defined by a Jaccard-based measure,

$$\mathrm{Groundedness} = \frac{|\mathrm{Nuggets}| - \sum A}{|\mathrm{Nuggets}|}$$9

Pooling in the drift study uses BM25, BGE (Gemma-2), E5 Mistral (7B), and Qwen3 (4B) embeddings, with scores normalized to $A$0 and summed in a hybrid fusion. Query formulations include the original Stack Overflow answer text, nuggets, Qwen3-4B-Instruct sub-questions, and Qwen3-4B-Instruct closed-book answers. Top-50 results per technique are pooled, and Cohere Command A (111B, 256k context) serves as the automatic judge.

The principal finding is strong support retention despite substantial corpus reorganization. In 2025, 202 of 203 queries remain fully supported, and only 1 of 640 nuggets lacks support. At the same time, repository contributions shift markedly. LangChain documentation decreases from 11,037 documents in 2024 to 3,628 in 2025, a 67% reduction, and its share of all supporting documents falls from 50.9% to 24.8%. LangChainJS changes from 3,852 to 3,921 documents, with supporting share rising from 18.6% to 25.5%. LlamaIndex shifts from 11,627 to 9,751 documents and from 16.1% to 22.6% of supporting documents. Chroma grows from 1,359 to 2,949 documents, approximately 2.6x, and its supporting share rises from 3.2% to 6.6%. Transformers increases from 4.4% to 6.0% of supporting documents, and OpenAI-Cookbook from 4.1% to 8.6%.

A detailed case study is Query 75864073, “ImportError when using UnstructuredPDFLoader in LangChain.” Its supporting pool grows from 12 document instances in 2024 to 26 in 2025. In 2024, 91.7% of supporting documents come from LangChain; in 2025, support is distributed across six repositories, with LlamaIndex becoming the largest single source at 34.6%. The paper attributes this to migration of relevant implementation and documentation, exemplified by the UnstructuredURLLoader class moving from LangChain to LlamaIndex. The lone unsupported case is described as representative of tasks in which topics such as “agents” moved from LangChain to LangGraph, although the exact query ID is not disclosed.

Retrieval rankings remain comparatively stable across the two snapshots. Qwen3 embeddings at 4B and 8B are consistently the strongest models on a-nDCG@10 and Recall@50. Absolute scores decrease slightly for most systems: BM25 drops from 0.228 to 0.187 on a-nDCG@10 and from 0.170 to 0.154 on Recall@50, while Qwen3 (8B) drops from 0.503 to 0.480 on a-nDCG@10 and from 0.432 to 0.411 on Recall@50. Ranking correlations are high, with Kendall $A$1 for Recall@50, $A$2 for a-nDCG@10, and $A$3 for Coverage@20. This suggests that re-judged evolving corpora can remain reliable for retrieval evaluation, although diversity-oriented coverage is more volatile than relevance-only ranking.

The drift study also sharpens FreshStack’s maintenance model. It recommends periodic re-judging at the nugget level whenever the corpus is refreshed, tracking repository migrations, broadening the corpus beyond a single flagship repository, detecting queries whose nuggets lose support, using diverse retrievers and query formulations in pooling, and documenting snapshot provenance through fixed commit hashes and stable chunking parameters. It also notes several threats to validity: the analysis covers only the LangChain domain, relies on a single LLM judge with no human inter-annotator agreement, may inherit pooling bias from the top-50 retrieval procedure, and does not report statistical significance tests or confidence intervals (Kuissi et al., 4 Mar 2026).