NanoKnow: Benchmarking LLM Knowledge
- NanoKnow is a benchmark dataset that distinguishes parametric knowledge stored in model weights from external retrieval evidence using a transparent framework.
- It employs a three-stage construction pipeline—BM25 retrieval, string matching, and LLM-based verification—to ascertain answer support from the open FineWeb-Edu corpus.
- Empirical findings reveal that closed-book accuracy increases with answer frequency and that retrieval context improves performance, though distractors can significantly degrade accuracy.
Searching arXiv for the specified NanoKnow paper and closely related work to ground the article. {"query":"arXiv (Gu et al., 23 Feb 2026) NanoKnow How to Know What Your LLM Knows nanochat FineWeb-Edu", "max_results": 5} Found relevant arXiv records for NanoKnow and surrounding context, including the target paper (Gu et al., 23 Feb 2026). NanoKnow is a benchmark dataset designed to determine what a LLM knows and where that knowledge comes from by exploiting a model family with fully open pre-training data. It uses nanochat, whose pre-training corpus is FineWeb-Edu, to partition question answering instances according to whether their answers are present in the pre-training corpus in a relevant context. This makes it possible to separate parametric knowledge expressed in closed-book QA from externally supplied knowledge in retrieval-augmented settings, and to study their interaction under controlled conditions (Gu et al., 23 Feb 2026).
1. Conceptual scope and problem formulation
NanoKnow addresses a specific identification problem in LLM analysis: closed-book success alone does not reveal whether a response derives from memorized pre-training exposure, from inference-time evidence, or from an interaction between the two. Its organizing distinction is between parametric knowledge stored in model weights, external knowledge injected at inference time via retrieval-augmented generation, and the possible interactions between them. Because most pre-training corpora are unknown or inaccessible, these sources are ordinarily entangled; NanoKnow uses the transparency of nanochat’s FineWeb-Edu pre-training corpus to make them analytically separable (Gu et al., 23 Feb 2026).
The benchmark is built by projecting standard QA data onto FineWeb-Edu. It partitions each instance into two operational categories. A question is supported if the answer appears in FineWeb-Edu in a relevant context. A question is unsupported if the answer does not appear in retrieved documents, or if the answer string appears only in unrelated or coincidental contexts. This distinction is intended to compare questions the model likely saw in pre-training against questions outside its parametric knowledge.
A common misconception is that retrieval alone fully explains open-book performance. NanoKnow is explicitly structured to test that claim rather than assume it. Its central analytical contribution is therefore not another QA leaderboard, but a controlled framework for asking why a model answers correctly.
2. Construction pipeline and operational labeling
NanoKnow is constructed in three stages over FineWeb-Edu, using a retrieval-first pipeline followed by semantic verification (Gu et al., 23 Feb 2026).
| Stage | Procedure | Output |
|---|---|---|
| 1 | BM25 retrieval with Anserini/Pyserini over FineWeb-Edu; top 100 candidate documents per QA pair | Candidate documents |
| 2 | Lowercase text, strip whitespace, and test whether the answer string appears as a substring | String-match candidates |
| 3 | LLM-based verification with Qwen3-8B using greedy decoding | TRUE or COINCIDENTAL label |
In the first stage, the benchmark builds a searchable BM25 index over FineWeb-Edu using Anserini/Pyserini and retrieves the top 100 candidate documents for each QA pair. In the second stage, it applies answer string matching after lowercasing text and stripping whitespace. This stage is deliberately inexpensive, but it produces false positives. The example given is that the string “Paris” can appear in a document about the song Paris rather than Paris, France.
The third stage is intended to remove those coincidental matches. For each answer hit, the system extracts a context window of 256 words before and 256 words after the match and asks Qwen3-8B to judge whether the context directly answers the question. Verification uses greedy decoding with temperature = 0, and the prompt requires one of two outputs: TRUE: [reason] or COINCIDENTAL: [reason].
Because support is defined through retrieval plus verification, NanoKnow operationalizes pre-training exposure as the presence of an answer in a retrievable, relevant FineWeb-Edu context. This does not claim omniscient access to all latent model knowledge; it specifies a reproducible criterion for whether an answer is grounded in the known corpus.
3. Dataset composition and corpus grounding
NanoKnow projects two validation sets onto FineWeb-Edu: the Natural Questions (NQ) validation set with 3,610 questions and the SQuAD validation set with 10,570 questions. After BM25 retrieval, string matching, and LLM verification, the benchmark reports the following support rates (Gu et al., 23 Feb 2026).
| Dataset | Samples | String Match / LLM Verified |
|---|---|---|
| NQ | 3,610 | 73.9% / 66.2% |
| SQuAD | 10,570 | 78.9% / 70.9% |
After verification, the supported fractions are 66.2% for NQ and 70.9% for SQuAD. The benchmark further notes that about 11% of string matches are coincidental and removed by the verification step. To validate the unsupported split, the authors prompt official nanochat d32 checkpoints in closed-book mode and observe only 1.5% accuracy for SQuAD and 0.8% accuracy for NQ under the answer-string-matching evaluation described for the benchmark.
The underlying FineWeb-Edu corpus is the shuffled release comprising 1,823 parquet shards, about 171 GB, and 97,230,848 documents; the Lucene index occupies about 326 GB. Each document is assigned an identifier of the form shard_XXXXX_YYYYY, where XXXXX is the shard number and YYYYY is the row offset. This design enables direct retrieval of a document from the parquet files without scanning the corpus.
These corpus details are not incidental. NanoKnow depends on a pre-training source that is both fully open and operationally searchable. That transparency is what distinguishes the benchmark from evaluations built over opaque web-scale mixtures.
4. Evaluation protocol and model suite
The benchmark evaluates eight nanochat checkpoints spanning three scales: d20 at approximately 561M parameters, d32 at approximately 1.9B parameters, and d34 at approximately 2.2B parameters. The evaluated checkpoints are sampathchanda/nanochat-d20, shu127/nanochat-d20, pankajmathur/nanochat-d20, karpathy/nanochat-d32, Antigma/nanochat-d32, renatocastro33/nanochat-d34-sft, victoremnm/nanochat-d34-sft, and pankajmathur/nanochat-d34-finetuned (Gu et al., 23 Feb 2026).
Three prompting configurations are used. Closed-book provides only the question. w/ FineWeb context provides the question plus the oracle answer passage from pre-training, using a 200-word window around the first matched answer, with about 100 words before and 100 words after. w/ Original Context is used for SQuAD only and supplies the original answer context from the dataset.
Two evaluation metrics are reported. Exact Match (EM) checks whether a correct answer exactly appears in the model output. LLM-Judge uses Qwen3-14B to classify the output as correct or incorrect. This pairing is methodologically important because it separates strict string-level correctness from a more semantically tolerant adjudication.
The protocol thereby supports three distinct comparisons: closed-book versus open-book behavior, supported versus unsupported behavior, and sensitivity to pre-training exposure frequency within the supported subset.
5. Empirical results on parametric memory and external evidence
The dominant empirical result is that closed-book accuracy is strongly influenced by answer frequency in the pre-training data. Supported questions are bucketed by verified answer frequency in FineWeb-Edu as Rare (1–5 verified documents), Low (6–20), Medium (21–50), and High (51+). Closed-book accuracy increases clearly with answer frequency on both NQ and SQuAD, and accuracy for high-frequency answers is more than double that for rare answers. The benchmark notes that this trend was not observed for d20, suggesting that smaller models may not have enough capacity to memorize as effectively (Gu et al., 23 Feb 2026).
External evidence weakens, but does not eliminate, this dependence. When FineWeb context is supplied, accuracy still increases with answer frequency, but the dependence is much weaker than in closed-book QA. Average LLM-Judge improvement from adding FineWeb context is reported as 2.4× for NQ d20, 2.14× for NQ d32, 1.89× for NQ d34, 4.4× for SQuAD d20, 3.2× for SQuAD d32, and 2.6× for SQuAD d34. This suggests that retrieval assistance is most beneficial when parametric memory is relatively weak.
NanoKnow also reports that pre-training exposure and external evidence are complementary, not redundant. Even with the oracle passage, models perform better on supported than on unsupported questions. For the best d34 checkpoint on SQuAD, supported performance is EM 0.721 and LLM-Judge 0.779, whereas unsupported performance is EM 0.610 and LLM-Judge 0.737. For the best d32 checkpoint, supported performance is EM 0.686 and LLM-Judge 0.740, compared with EM 0.553 and LLM-Judge 0.680 for unsupported questions.
Selected values from the supported-split comparisons illustrate the same pattern. For Antigma/nanochat-d32, SQuAD scores are EM 0.122 / LLM-Judge 0.169 in closed-book mode, EM 0.483 / LLM-Judge 0.551 with FineWeb context, and EM 0.686 / LLM-Judge 0.740 with Original Context; NQ scores are EM 0.198 / LLM-Judge 0.226 in closed-book mode and EM 0.516 / LLM-Judge 0.492 with FineWeb context. For renatocastro33/nanochat-d34-sft, SQuAD scores are EM 0.167 / LLM-Judge 0.228 in closed-book mode, EM 0.512 / LLM-Judge 0.587 with FineWeb context, and EM 0.721 / LLM-Judge 0.779 with Original Context; NQ scores are EM 0.250 / LLM-Judge 0.283 in closed-book mode and EM 0.503 / LLM-Judge 0.528 with FineWeb context.
A plausible implication is that retrieval does not merely supply missing facts; it interacts with already-internalized representations. NanoKnow formalizes that interaction rather than collapsing it into a single “open-book QA” score.
6. Distractors, retrieval quality, and analytical significance
NanoKnow includes a distractor study using Antigma/nanochat-d32 to test how non-relevant contexts alter performance. The benchmark evaluates three context arrangements: Far as [A, D, Q], Mid as [D, A, D, Q], and Near as [D, A, Q], where A denotes the answer document, D a distractor document, and Q the question. It also tests Distractor only, Answer only, Answer + 1 distractor, Answer + 2 distractors, and Answer + 4 distractors (Gu et al., 23 Feb 2026).
The first result is that distractors alone are harmful. Relative to closed-book performance, SQuAD LLM-Judge drops from 0.169 to 0.154, and NQ LLM-Judge drops from 0.226 to 0.194. Non-relevant context is therefore worse than relying on parametric knowledge alone. The second result is monotonic degradation when distractors are added to an answer document. In the Far setting on SQuAD, Answer only yields LLM-Judge 0.551, Answer + 1 distractor yields 0.478, Answer + 2 distractors yields 0.438, and Answer + 4 distractors yields 0.367. The same monotonic pattern is reported on NQ.
Position also matters. The model is most accurate when the answer is closest to the question, but only when there are no distractors between them. The Mid arrangement, where the answer is buried among distractors, is the worst configuration and shows a clear lost-in-the-middle effect. This directly links NanoKnow to retrieval-quality evaluation: it is not sufficient to provide relevant evidence if irrelevant evidence is also present in adversarially damaging positions.
The broader significance of NanoKnow follows from these findings. It provides a controlled framework for measuring what an LLM knows, for determining whether that knowledge likely came from pre-training exposure, for testing how retrieval changes reliance on parametric memory, and for studying how distractors affect RAG systems. Its artifacts are released at https://github.com/castorini/NanoKnow. The benchmark’s broader takeaway is that closed-book accuracy is not enough, pre-training exposure frequency matters, RAG helps most when parametric knowledge is weak, parametric and external knowledge are complementary, and retrieval quality matters because distractors can significantly degrade performance, especially when relevant evidence is buried in the middle.