---
title: Book Memorization Coverage (BMC@5)
url: https://www.emergentmind.com/topics/book-memorization-coverage-bmc-5
type: topic
---

# Book Memorization Coverage (BMC@5)

Book Memorization Coverage at 5% Threshold (BMC@5) is a book-level metric quantifying the capacity of large language models (LLMs) to reproduce contiguous segments of a book verbatim, based on the fraction of book tokens that can be recovered. It provides a rigorous, operational measure for assessing “memorization coverage” in LLMs, particularly in the context of copyright compliance and regurgitation risk audit. BMC@5 is defined by the minimum 5% threshold: a book is considered covered if the model can reproduce at least 5% of its tokens verbatim via extraction protocols after finetuning or under default inference. This measure is central to evaluating memorization both before and after alignment or further training interventions.

## 1. Formal Definition

BMC@5 computes, over a set of $N$ books, the percentage for which an LLM can recall at least 5% of tokens verbatim under a specified extraction protocol. For a book $b_i$ of length $|b_i|$, and $\text{RecoveredTokens}(b_i)$ denoting the number of its tokens exactly regurgitated by the model, the metric is

$$
\text{BMC@5} = 
\frac{1}{N} \sum_{i=1}^N \mathbb{I}\left( \frac{\text{RecoveredTokens}(b_i)}{|b_i|}\geq 0.05 \right)\! \times 100\%
$$

where $\mathbb{I}(\cdot)$ is the indicator function. In the context of semantic-prompting experiments, BMC@5 can alternatively denote the fraction of a book’s total word positions lying within at least one contiguous, exact 5-word (or longer) match, after filtering for any overlap with the input instruction [2603.20957, 2504.12549].

## 2. Experimental Protocols

BMC@5 has been operationalized under distinct but related extraction methodologies:

- **Prefix-prompting auto-regressive extraction**: The book is processed in disjoint or sliding token windows. For each chunk, a prefix (e.g., 500 tokens) is given; the model greedily generates a fixed-length suffix (e.g., 30 tokens), which is compared verbatim to the ground truth. The total number of exactly matched tokens is summed, establishing $\text{RecoveredTokens}(b)$ [2504.12549].
- **Semantic plot-prompting extraction**: For held-out books, 300–500 word paragraphs are summarized with a prompt (e.g., via GPT-4o), describing content and structure. The model is asked, multiple times per prompt (e.g., $T=100$ samplings, $T=1.0$), to generate text “emulating the style of {author},” and contiguous 5-word or longer exact matches are detected, excluding overlaps with the prompt or paraphrased content [2603.20957].

These protocols enforce deterministic decoding (greedy) or sample-based decoding (temperature, top-$k$), with the threshold for extraction set at spans of 5 consecutive tokens or words, as appropriate for the study.

## 3. Benchmark Results

Tables 1 and 2: Representative BMC@5 results by model and experimental setting.

| Model/Condition                       | #Books ≥ 5% | BMC@5 (%)           |
|---------------------------------------|-------------|---------------------|
| Llama 3 70B (pretrained, piece-wise)  |      7/9    | 77.8                |
| Llama 3.1 70B (pretrained, piece-wise)|      5/9    | 55.6                |
| Llama 3.1 70B Instruct (no SFT)       |      0/9    | 0.0                 |
| Llama 3.1 70B Instruct + SFT (500)    |      5/9    | 55.6                |

| Condition                | GPT-4o | Gemini-2.5-Pro | DeepSeek-V3.1 |
|--------------------------|--------|----------------|---------------|
| Baseline (aligned)       | 7.36   |     —          |  —            |
| Within-author finetune   | 35.2   |    37.1        |   34.0        |
| Cross-author (Murakami)  | 51.3   |    44.6        |   40.8        |

Key findings:

- Instruction-tuned, mitigated LLMs (baseline) have BMC@5 near zero.
- Pretrained, unaligned models see 50–80% BMC@5, depending on architecture and book corpus.
- Finetuning, even with a small LoRA adapter or on unrelated public-domain authors, restores or activates BMC@5 in the 35–50% range [2504.12549, 2603.20957].

## 4. Factors Influencing BMC@5

Empirical studies reveal key correlates and determinants:

- **Popularity and duplication**: BMC@5 increases with book popularity (Pearson $\rho \approx 0.50$ versus $\log_{10}$(Goodreads ratings)). Highly duplicated or widely quoted works are more likely to surpass the 5% verbatim recall threshold in model generations [2504.12549].
- **Alignment and SFT**: Alignment by instruction tuning drops BMC@5 to 0%, but minimal LoRA-based SFT on prefix-to-suffix fragments suffices to restore BMC@5 to pretrained levels, affecting only a tiny fraction of model weights—primarily in lower transformer self-attention blocks [2504.12549].
- **Finetuning data**: Finetuning on author-specific corpora, public-domain works, or even random author pairs causes similar magnitudes of BMC@5 increases, indicating the effect is not idiosyncratic to the finetuning set but is driven by overlap with pretraining memorization [2603.20957].
- **Text syntheticity**: Finetuning on synthetic-generated stories yields only near-baseline BMC@5, implicating pretraining duplication as the mechanism for large BMC@5 increases under plot-expansion finetuning [2603.20957].

## 5. Relation to Alternative Memorization Metrics

While “Book Memorization Coverage” at a fixed threshold (e.g., 5%) is now standard in quantifying book-level regurgitation, alternative metrics have been explored:

- **Probabilistic Extraction Probability ($p_z$)**: Measures the likelihood that a given suffix is exactly reproduced, conditioned on a prefix, aggregated across windows within the book. Coverage is then summarized as the proportion of the book with $p_z \geq$ threshold, e.g., $1\%$, $10\%$ [2505.12546].
- **Greedy Extraction Rate**: Fraction of windows for which a greedy generation yields an exact match for the next $k$ tokens.
- **Sliding-window ($a$/$k$-split) coverage**: Quantifies granular memorization profile as a function of position and local prompt.

These measures are mathematically connected; BMC@5 can be viewed as a book-level aggregation—recording whether a book passes a “minimum coverage” bar, rather than fine-grained per-location metrics.

## 6. Implications and Applications

BMC@5 serves as a transparent, model-agnostic metric for assessing the memorization risk posed by LLMs with respect to copyright-sensitive content:

- **Legal and Regulatory Relevance**: BMC@5 quantifies the practical risk of verbatim regurgitation, providing objective evidence to courts of the presence or absence of protected expression in a model’s output under realistic attack scenarios [2504.12549, 2603.20957].
- **Model Safety and Alignment Auditing**: BMC@5 exposes the limitations of alignment strategies such as RLHF, system prompts, or output filters, which suppress BMC@5 at deployment but can be circumvented post-hoc by lightweight finetuning, even on unrelated or innocuous data [2603.20957].
- **Model Forensics**: Analysis of parameter updates needed to shift BMC@5 quantifies which model weights concentrate memorized book representations, guiding efforts in regularization or auditing for further mitigation [2504.12549].

## 7. Limitations and Future Directions

BMC@5 is specific to exact, verbatim extraction; it does not assess paraphrased regurgitation or more diffuse forms of memorization. Additionally, the metric’s threshold is fixed arbitrarily at 5%, and results are protocol-dependent (prefix length, decoding, prompt semantics). Extending the analysis to variable thresholds ($\text{BMC@k}$ for $k\ge1$), more granular span statistics, and to non-exact forms of reproduction remains a key direction. A plausible implication is that more robust detection and prevention of regurgitation may require both parameter-level and data-centric solutions, as model-level alignment can be readily undermined by trivial finetuning [2504.12549, 2603.20957].

Source: https://www.emergentmind.com/topics/book-memorization-coverage-bmc-5