CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation (2407.07087v2)

Abstract: Evaluating the degree of reproduction of copyright-protected content by LLMs (LMs) is of significant interest to the AI and legal communities. Although both literal and non-literal similarities are considered by courts when assessing the degree of reproduction, prior research has focused only on literal similarities. To bridge this gap, we introduce CopyBench, a benchmark designed to measure both literal and non-literal copying in LM generations. Using copyrighted fiction books as text sources, we provide automatic evaluation protocols to assess literal and non-literal copying, balanced against the model utility in terms of the ability to recall facts from the copyrighted works and generate fluent completions. We find that, although literal copying is relatively rare, two types of non-literal copying -- event copying and character copying -- occur even in models as small as 7B parameters. Larger models demonstrate significantly more copying, with literal copying rates increasing from 0.2\% to 10.5\% and non-literal copying from 2.3\% to 5.9\% when comparing Llama3-8B and 70B models, respectively. We further evaluate the effectiveness of current strategies for mitigating copying and show that (1) training-time alignment can reduce literal copying but may increase non-literal copying, and (2) current inference-time mitigation methods primarily reduce literal but not non-literal copying.

• The paper introduces CopyBench, a benchmark that quantifies literal and non-literal copying of copyrighted text in language model outputs.
• It uses ROUGE-L scores and event/character extraction to evaluate duplication and fact recall across various state-of-the-art language models.
• The study shows a trade-off where reducing literal copying can lead to increased non-literal reproduction, challenging current mitigation strategies.

Summary of "CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in LLM Generation"

The paper "CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in LLM Generation" presents CopyBench, a benchmark specifically designed to evaluate both literal and non-literal copying behaviors in LLMs (LMs). Traditional research tends to focus on literal duplication, however, the paper aims to address a significant gap by introducing methodologies that evaluate non-literal copying, such as the reproduction of events and character details from copyrighted text.

Key Contributions

1. Introduction of CopyBench

CopyBench is introduced as a benchmark tailored to measure the reproduction of copyrighted content in both literal and non-literal forms. To ensure comprehensive evaluation, the benchmark includes datasets and automatic evaluation protocols that assess model outputs for both types of copying.

2. Definition of Literal and Non-Literal Copying

• Literal Copying:
  • Defined as the near-exact reproduction of copyrighted text.
  • Evaluated using ROUGE-L scores between the generated outputs and segments of copyrighted texts.
• Non-Literal Copying:
  • Includes reproductions that are not exact in wording but closely match the content, such as plots and character details.
  • Evaluated by extracting key events and characters from novels and analyzing overlaps with generated stories.

3. Utility Evaluation

Alongside copying metrics, CopyBench assesses model utility through:

• Fact Recall: The ability to correctly remember and reproduce factual content from the source texts.
• Fluency: Evaluated to ensure that mitigation techniques do not adversely impact the linguistic quality of generated text.

Experimental Setup

The experiment assessed multiple state-of-the-art white-box and proprietary LMs, including the Llama2, Llama3, and Mistral models, as well as GPT-3.5-Turbo and GPT-4-Turbo. The experiments illuminated the differences between literal and non-literal copying behaviors across these models:

• Smaller Models (e.g., Llama2-13B) exhibited minimal literal copying but showed measurable non-literal copying. The non-literal copying rate increases with model size.
• Larger Models (e.g., Llama3-70B) demonstrated higher rates of both literal and non-literal copying, confirming that model capacity influences the extent of content reproduction.
• Proprietary models, particularly the evolution from GPT-3.5 to GPT-4, showed reduced literal copying but increased non-literal copying.

Mitigation Strategies

The paper also investigates current mitigation strategies aimed at reducing copying behaviors:

• Training-Time Alignment:
  • Certain instruction-tuned models show effectiveness in reducing literal copying. However, non-literal copying was reduced to a lesser extent or could sometimes increase.
  • Proprietary models employing closed-source data for instruction tuning demonstrated more significant reductions in copying compared to open-source models like Tulu2.
• Inference-Time Techniques:
  • Techniques such as MemFree decoding were highly effective in eliminating literal copying, yet ineffective for non-literal copying.
  • System-mode self-reminders, which involve ethical prompts to avoid copying, did not show substantial improvements in mitigation.

Implications and Future Directions

The findings suggest a complex trade-off between utility and the minimization of copying behaviors. Larger models, which offer higher utility and recall capabilities, are more prone to both literal and non-literal copying. The insights provided by CopyBench can guide future research in creating balanced models that respect copyright laws while maintaining high utility. The benchmark calls for more sophisticated mitigation strategies capable of addressing both forms of copying and underscores the need for ongoing open-source research in this domain.

Conclusion

"CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in LLM Generation" significantly advances the understanding of copying behaviors in LLMs. The paper underscores the inadequacy of focusing merely on literal copying and introduces a comprehensive framework for evaluating non-literal copying. By addressing the nuanced requirements of legal and ethical AI deployment, CopyBench sets a new standard for future evaluations of LLMs' compliance with copyright norms.