---
title: 'NanoKnow: Benchmarking LLM Knowledge'
url: https://www.emergentmind.com/topics/nanoknow
type: topic
---

# NanoKnow: Benchmarking LLM Knowledge

Searching arXiv for the specified NanoKnow paper and closely related work to ground the article.
{"query":"arXiv 2602.20122 NanoKnow How to Know What Your Language Model Knows nanochat FineWeb-Edu", "max_results": 5}
Found relevant arXiv records for NanoKnow and surrounding context, including the target paper [2602.20122].
NanoKnow is a benchmark dataset designed to determine what a language model knows and where that knowledge comes from by exploiting a model family with fully open pre-training data. It uses nanochat, whose pre-training corpus is FineWeb-Edu, to partition question answering instances according to whether their answers are present in the pre-training corpus in a relevant context. This makes it possible to separate parametric knowledge expressed in closed-book QA from externally supplied knowledge in retrieval-augmented settings, and to study their interaction under controlled conditions [2602.20122].

## 1. Conceptual scope and problem formulation

NanoKnow addresses a specific identification problem in LLM analysis: closed-book success alone does not reveal whether a response derives from memorized pre-training exposure, from inference-time evidence, or from an interaction between the two. Its organizing distinction is between **parametric knowledge** stored in model weights, **external knowledge** injected at inference time via retrieval-augmented generation, and the possible interactions between them. Because most pre-training corpora are unknown or inaccessible, these sources are ordinarily entangled; NanoKnow uses the transparency of nanochat’s FineWeb-Edu pre-training corpus to make them analytically separable [2602.20122].

The benchmark is built by projecting standard QA data onto FineWeb-Edu. It partitions each instance into two operational categories. A question is **supported** if the answer appears in FineWeb-Edu in a relevant context. A question is **unsupported** if the answer does not appear in retrieved documents, or if the answer string appears only in unrelated or coincidental contexts. This distinction is intended to compare questions the model likely saw in pre-training against questions outside its parametric knowledge.

A common misconception is that retrieval alone fully explains open-book performance. NanoKnow is explicitly structured to test that claim rather than assume it. Its central analytical contribution is therefore not another QA leaderboard, but a controlled framework for asking why a model answers correctly.

## 2. Construction pipeline and operational labeling

NanoKnow is constructed in three stages over FineWeb-Edu, using a retrieval-first pipeline followed by semantic verification [2602.20122].

| Stage | Procedure | Output |
|---|---|---|
| 1 | BM25 retrieval with Anserini/Pyserini over FineWeb-Edu; top 100 candidate documents per QA pair | Candidate documents |
| 2 | Lowercase text, strip whitespace, and test whether the answer string appears as a substring | String-match candidates |
| 3 | LLM-based verification with Qwen3-8B using greedy decoding | `TRUE` or `COINCIDENTAL` label |

In the first stage, the benchmark builds a searchable BM25 index over FineWeb-Edu using Anserini/Pyserini and retrieves the top 100 candidate documents for each QA pair. In the second stage, it applies answer string matching after lowercasing text and stripping whitespace. This stage is deliberately inexpensive, but it produces false positives. The example given is that the string “Paris” can appear in a document about the song *Paris* rather than Paris, France.

The third stage is intended to remove those coincidental matches. For each answer hit, the system extracts a context window of **256 words before** and **256 words after** the match and asks **Qwen3-8B** to judge whether the context directly answers the question. Verification uses **greedy decoding** with **temperature = 0**, and the prompt requires one of two outputs: `TRUE: [reason]` or `COINCIDENTAL: [reason]`.

Because support is defined through retrieval plus verification, NanoKnow operationalizes pre-training exposure as the presence of an answer in a retrievable, relevant FineWeb-Edu context. This does not claim omniscient access to all latent model knowledge; it specifies a reproducible criterion for whether an answer is grounded in the known corpus.

## 3. Dataset composition and corpus grounding

NanoKnow projects two validation sets onto FineWeb-Edu: the **Natural Questions (NQ)** validation set with **3,610 questions** and the **SQuAD** validation set with **10,570 questions**. After BM25 retrieval, string matching, and LLM verification, the benchmark reports the following support rates [2602.20122].

| Dataset | Samples | String Match / LLM Verified |
|---|---:|---:|
| NQ | 3,610 | 73.9% / 66.2% |
| SQuAD | 10,570 | 78.9% / 70.9% |

After verification, the supported fractions are **66.2%** for NQ and **70.9%** for SQuAD. The benchmark further notes that about **11% of string matches** are coincidental and removed by the verification step. To validate the unsupported split, the authors prompt official nanochat d32 checkpoints in closed-book mode and observe only **1.5%** accuracy for SQuAD and **0.8%** accuracy for NQ under the answer-string-matching evaluation described for the benchmark.

The underlying FineWeb-Edu corpus is the shuffled release comprising **1,823 parquet shards**, about **171 GB**, and **97,230,848 documents**; the Lucene index occupies about **326 GB**. Each document is assigned an identifier of the form `shard_XXXXX_YYYYY`, where `XXXXX` is the shard number and `YYYYY` is the row offset. This design enables direct retrieval of a document from the parquet files without scanning the corpus.

These corpus details are not incidental. NanoKnow depends on a pre-training source that is both fully open and operationally searchable. That transparency is what distinguishes the benchmark from evaluations built over opaque web-scale mixtures.

## 4. Evaluation protocol and model suite

The benchmark evaluates **eight nanochat checkpoints** spanning three scales: **d20** at approximately **561M parameters**, **d32** at approximately **1.9B parameters**, and **d34** at approximately **2.2B parameters**. The evaluated checkpoints are `sampathchanda/nanochat-d20`, `shu127/nanochat-d20`, `pankajmathur/nanochat-d20`, `karpathy/nanochat-d32`, `Antigma/nanochat-d32`, `renatocastro33/nanochat-d34-sft`, `victoremnm/nanochat-d34-sft`, and `pankajmathur/nanochat-d34-finetuned` [2602.20122].

Three prompting configurations are used. **Closed-book** provides only the question. **w/ FineWeb context** provides the question plus the oracle answer passage from pre-training, using a **200-word window** around the first matched answer, with about **100 words before** and **100 words after**. **w/ Original Context** is used for SQuAD only and supplies the original answer context from the dataset.

Two evaluation metrics are reported. **Exact Match (EM)** checks whether a correct answer exactly appears in the model output. **LLM-Judge** uses **Qwen3-14B** to classify the output as correct or incorrect. This pairing is methodologically important because it separates strict string-level correctness from a more semantically tolerant adjudication.

The protocol thereby supports three distinct comparisons: closed-book versus open-book behavior, supported versus unsupported behavior, and sensitivity to pre-training exposure frequency within the supported subset.

## 5. Empirical results on parametric memory and external evidence

The dominant empirical result is that **closed-book accuracy is strongly influenced by answer frequency in the pre-training data**. Supported questions are bucketed by verified answer frequency in FineWeb-Edu as **Rare** (1–5 verified documents), **Low** (6–20), **Medium** (21–50), and **High** (51+). Closed-book accuracy increases clearly with answer frequency on both NQ and SQuAD, and accuracy for **high-frequency** answers is **more than double** that for **rare** answers. The benchmark notes that this trend was not observed for **d20**, suggesting that smaller models may not have enough capacity to memorize as effectively [2602.20122].

External evidence weakens, but does not eliminate, this dependence. When FineWeb context is supplied, accuracy still increases with answer frequency, but the dependence is much weaker than in closed-book QA. Average LLM-Judge improvement from adding FineWeb context is reported as **2.4×** for NQ d20, **2.14×** for NQ d32, **1.89×** for NQ d34, **4.4×** for SQuAD d20, **3.2×** for SQuAD d32, and **2.6×** for SQuAD d34. This suggests that retrieval assistance is most beneficial when parametric memory is relatively weak.

NanoKnow also reports that pre-training exposure and external evidence are **complementary**, not redundant. Even with the oracle passage, models perform better on **supported** than on **unsupported** questions. For the best d34 checkpoint on SQuAD, **supported** performance is **EM 0.721** and **LLM-Judge 0.779**, whereas **unsupported** performance is **EM 0.610** and **LLM-Judge 0.737**. For the best d32 checkpoint, **supported** performance is **EM 0.686** and **LLM-Judge 0.740**, compared with **EM 0.553** and **LLM-Judge 0.680** for **unsupported** questions.

Selected values from the supported-split comparisons illustrate the same pattern. For **Antigma/nanochat-d32**, SQuAD scores are **EM 0.122 / LLM-Judge 0.169** in closed-book mode, **EM 0.483 / LLM-Judge 0.551** with FineWeb context, and **EM 0.686 / LLM-Judge 0.740** with Original Context; NQ scores are **EM 0.198 / LLM-Judge 0.226** in closed-book mode and **EM 0.516 / LLM-Judge 0.492** with FineWeb context. For **renatocastro33/nanochat-d34-sft**, SQuAD scores are **EM 0.167 / LLM-Judge 0.228** in closed-book mode, **EM 0.512 / LLM-Judge 0.587** with FineWeb context, and **EM 0.721 / LLM-Judge 0.779** with Original Context; NQ scores are **EM 0.250 / LLM-Judge 0.283** in closed-book mode and **EM 0.503 / LLM-Judge 0.528** with FineWeb context.

A plausible implication is that retrieval does not merely supply missing facts; it interacts with already-internalized representations. NanoKnow formalizes that interaction rather than collapsing it into a single “open-book QA” score.

## 6. Distractors, retrieval quality, and analytical significance

NanoKnow includes a distractor study using **Antigma/nanochat-d32** to test how non-relevant contexts alter performance. The benchmark evaluates three context arrangements: **Far** as `[A, D, Q]`, **Mid** as `[D, A, D, Q]`, and **Near** as `[D, A, Q]`, where `A` denotes the answer document, `D` a distractor document, and `Q` the question. It also tests **Distractor only**, **Answer only**, **Answer + 1 distractor**, **Answer + 2 distractors**, and **Answer + 4 distractors** [2602.20122].

The first result is that distractors alone are harmful. Relative to closed-book performance, SQuAD **LLM-Judge** drops from **0.169** to **0.154**, and NQ **LLM-Judge** drops from **0.226** to **0.194**. Non-relevant context is therefore worse than relying on parametric knowledge alone. The second result is monotonic degradation when distractors are added to an answer document. In the **Far** setting on SQuAD, **Answer only** yields **LLM-Judge 0.551**, **Answer + 1 distractor** yields **0.478**, **Answer + 2 distractors** yields **0.438**, and **Answer + 4 distractors** yields **0.367**. The same monotonic pattern is reported on NQ.

Position also matters. The model is most accurate when the answer is **closest to the question**, but only when there are no distractors between them. The **Mid** arrangement, where the answer is buried among distractors, is the worst configuration and shows a clear **lost-in-the-middle** effect. This directly links NanoKnow to retrieval-quality evaluation: it is not sufficient to provide relevant evidence if irrelevant evidence is also present in adversarially damaging positions.

The broader significance of NanoKnow follows from these findings. It provides a controlled framework for measuring what an LLM knows, for determining whether that knowledge likely came from pre-training exposure, for testing how retrieval changes reliance on parametric memory, and for studying how distractors affect RAG systems. Its artifacts are released at `https://github.com/castorini/NanoKnow`. The benchmark’s broader takeaway is that **closed-book accuracy is not enough**, **pre-training exposure frequency matters**, **RAG helps most when parametric knowledge is weak**, **parametric and external knowledge are complementary**, and **retrieval quality matters because distractors can significantly degrade performance, especially when relevant evidence is buried in the middle**.

Source: https://www.emergentmind.com/topics/nanoknow