Book Memorization Coverage (BMC@5)

• Book Memorization Coverage at 5% (BMC@5) is a metric that measures the percentage of books for which an LLM can recall at least 5% of tokens verbatim.
• The metric uses approaches such as prefix-prompting and semantic plot-prompting extraction to assess accurate, contiguous token reproduction.
• BMC@5 is vital for evaluating copyright compliance and model safety by quantifying risks associated with verbatim regurgitation in LLM outputs.

Book Memorization Coverage at 5% Threshold (BMC@5) is a book-level metric quantifying the capacity of LLMs to reproduce contiguous segments of a book verbatim, based on the fraction of book tokens that can be recovered. It provides a rigorous, operational measure for assessing “memorization coverage” in LLMs, particularly in the context of copyright compliance and regurgitation risk audit. BMC@5 is defined by the minimum 5% threshold: a book is considered covered if the model can reproduce at least 5% of its tokens verbatim via extraction protocols after finetuning or under default inference. This measure is central to evaluating memorization both before and after alignment or further training interventions.

1. Formal Definition

BMC@5 computes, over a set of $N$ books, the percentage for which an LLM can recall at least 5% of tokens verbatim under a specified extraction protocol. For a book $b_i$ of length $|b_i|$, and $\text{RecoveredTokens}(b_i)$ denoting the number of its tokens exactly regurgitated by the model, the metric is

$$\text{BMC@5} = \frac{1}{N} \sum_{i=1}^N \mathbb{I}\left( \frac{\text{RecoveredTokens}(b_i)}{|b_i|}\geq 0.05 \right)\! \times 100\%$$

where $\mathbb{I}(\cdot)$ is the indicator function. In the context of semantic-prompting experiments, BMC@5 can alternatively denote the fraction of a book’s total word positions lying within at least one contiguous, exact 5-word (or longer) match, after filtering for any overlap with the input instruction (Liu et al., 21 Mar 2026, Ma et al., 17 Apr 2025).

2. Experimental Protocols

BMC@5 has been operationalized under distinct but related extraction methodologies:

• Prefix-prompting auto-regressive extraction: The book is processed in disjoint or sliding token windows. For each chunk, a prefix (e.g., 500 tokens) is given; the model greedily generates a fixed-length suffix (e.g., 30 tokens), which is compared verbatim to the ground truth. The total number of exactly matched tokens is summed, establishing $\text{RecoveredTokens}(b)$ (Ma et al., 17 Apr 2025).
• Semantic plot-prompting extraction: For held-out books, 300–500 word paragraphs are summarized with a prompt (e.g., via GPT-4o), describing content and structure. The model is asked, multiple times per prompt (e.g., $T=100$ samplings, $T=1.0$), to generate text “emulating the style of {author},” and contiguous 5-word or longer exact matches are detected, excluding overlaps with the prompt or paraphrased content (Liu et al., 21 Mar 2026).

These protocols enforce deterministic decoding (greedy) or sample-based decoding (temperature, top-$k$), with the threshold for extraction set at spans of 5 consecutive tokens or words, as appropriate for the study.

3. Benchmark Results

Tables 1 and 2: Representative BMC@5 results by model and experimental setting.

Model/Condition	#Books ≥ 5%	BMC@5 (%)
Llama 3 70B (pretrained, piece-wise)	7/9	77.8
Llama 3.1 70B (pretrained, piece-wise)	5/9	55.6
Llama 3.1 70B Instruct (no SFT)	0/9	0.0
Llama 3.1 70B Instruct + SFT (500)	5/9	55.6

Condition	GPT-4o	Gemini-2.5-Pro	DeepSeek-V3.1
Baseline (aligned)	7.36	—	—
Within-author finetune	35.2	37.1	34.0
Cross-author (Murakami)	51.3	44.6	40.8

Key findings:

• Instruction-tuned, mitigated LLMs (baseline) have BMC@5 near zero.
• Pretrained, unaligned models see 50–80% BMC@5, depending on architecture and book corpus.
• Finetuning, even with a small LoRA adapter or on unrelated public-domain authors, restores or activates BMC@5 in the 35–50% range (Ma et al., 17 Apr 2025, Liu et al., 21 Mar 2026).

4. Factors Influencing BMC@5

Empirical studies reveal key correlates and determinants:

• Popularity and duplication: BMC@5 increases with book popularity (Pearson $b_i$0 versus $b_i$1(Goodreads ratings)). Highly duplicated or widely quoted works are more likely to surpass the 5% verbatim recall threshold in model generations (Ma et al., 17 Apr 2025).
• Alignment and SFT: Alignment by instruction tuning drops BMC@5 to 0%, but minimal LoRA-based SFT on prefix-to-suffix fragments suffices to restore BMC@5 to pretrained levels, affecting only a tiny fraction of model weights—primarily in lower transformer self-attention blocks (Ma et al., 17 Apr 2025).
• Finetuning data: Finetuning on author-specific corpora, public-domain works, or even random author pairs causes similar magnitudes of BMC@5 increases, indicating the effect is not idiosyncratic to the finetuning set but is driven by overlap with pretraining memorization (Liu et al., 21 Mar 2026).
• Text syntheticity: Finetuning on synthetic-generated stories yields only near-baseline BMC@5, implicating pretraining duplication as the mechanism for large BMC@5 increases under plot-expansion finetuning (Liu et al., 21 Mar 2026).

5. Relation to Alternative Memorization Metrics

While “Book Memorization Coverage” at a fixed threshold (e.g., 5%) is now standard in quantifying book-level regurgitation, alternative metrics have been explored:

• Probabilistic Extraction Probability ($b_i$2): Measures the likelihood that a given suffix is exactly reproduced, conditioned on a prefix, aggregated across windows within the book. Coverage is then summarized as the proportion of the book with $b_i$3 threshold, e.g., $b_i$4, $b_i$5 (Cooper et al., 18 May 2025).
• Greedy Extraction Rate: Fraction of windows for which a greedy generation yields an exact match for the next $b_i$6 tokens.
• Sliding-window ($b_i$7/$b_i$8-split) coverage: Quantifies granular memorization profile as a function of position and local prompt.

These measures are mathematically connected; BMC@5 can be viewed as a book-level aggregation—recording whether a book passes a “minimum coverage” bar, rather than fine-grained per-location metrics.

6. Implications and Applications

BMC@5 serves as a transparent, model-agnostic metric for assessing the memorization risk posed by LLMs with respect to copyright-sensitive content:

• Legal and Regulatory Relevance: BMC@5 quantifies the practical risk of verbatim regurgitation, providing objective evidence to courts of the presence or absence of protected expression in a model’s output under realistic attack scenarios (Ma et al., 17 Apr 2025, Liu et al., 21 Mar 2026).
• Model Safety and Alignment Auditing: BMC@5 exposes the limitations of alignment strategies such as RLHF, system prompts, or output filters, which suppress BMC@5 at deployment but can be circumvented post-hoc by lightweight finetuning, even on unrelated or innocuous data (Liu et al., 21 Mar 2026).
• Model Forensics: Analysis of parameter updates needed to shift BMC@5 quantifies which model weights concentrate memorized book representations, guiding efforts in regularization or auditing for further mitigation (Ma et al., 17 Apr 2025).

7. Limitations and Future Directions

BMC@5 is specific to exact, verbatim extraction; it does not assess paraphrased regurgitation or more diffuse forms of memorization. Additionally, the metric’s threshold is fixed arbitrarily at 5%, and results are protocol-dependent (prefix length, decoding, prompt semantics). Extending the analysis to variable thresholds ($b_i$9 for $|b_i|$0), more granular span statistics, and to non-exact forms of reproduction remains a key direction. A plausible implication is that more robust detection and prevention of regurgitation may require both parameter-level and data-centric solutions, as model-level alignment can be readily undermined by trivial finetuning (Ma et al., 17 Apr 2025, Liu et al., 21 Mar 2026).