Readers Prefer Outputs of AI Trained on Copyrighted Books over Expert Human Writers (2510.13939v2)

Abstract: The use of copyrighted books for training AI models has led to numerous lawsuits from authors concerned about AI's ability to generate derivative content. Yet it's unclear if these models can generate high quality literary text while emulating authors' styles. To answer this we conducted a preregistered study comparing MFA-trained expert writers with three frontier AI models: ChatGPT, Claude & Gemini in writing up to 450 word excerpts emulating 50 award-winning authors' diverse styles. In blind pairwise evaluations by 159 representative expert & lay readers, AI-generated text from in-context prompting was strongly disfavored by experts for both stylistic fidelity (OR=0.16, p<10^-8) & writing quality (OR=0.13, p<10^-7) but showed mixed results with lay readers. However, fine-tuning ChatGPT on individual authors' complete works completely reversed these findings: experts now favored AI-generated text for stylistic fidelity (OR=8.16, p<10^-13) & writing quality (OR=1.87, p=0.010), with lay readers showing similar shifts. These effects generalize across authors & styles. The fine-tuned outputs were rarely flagged as AI-generated (3% rate v. 97% for in-context prompting) by best AI detectors. Mediation analysis shows this reversal occurs because fine-tuning eliminates detectable AI stylistic quirks (e.g., cliche density) that penalize in-context outputs. While we do not account for additional costs of human effort required to transform raw AI output into cohesive, publishable prose, the median fine-tuning & inference cost of $81 per author represents a dramatic 99.7% reduction compared to typical professional writer compensation. Author-specific fine-tuning thus enables non-verbatim AI writing that readers prefer to expert human writing, providing empirical evidence directly relevant to copyright's fourth fair-use factor, the "effect upon the potential market or value" of the source works.

• The paper demonstrates that fine-tuning AI on copyrighted literary corpora reverses reader preferences, with experts favoring AI-generated texts (OR = 8.16 for stylistic fidelity).
• The methodology employs author-specific fine-tuning and blinded pairwise evaluations contrasting LLM outputs with MFA-trained human writers across style and quality metrics.
• The study underscores significant cost savings and market risks, revealing that fine-tuned AI outputs can be produced for as little as $81 compared to tens of thousands for professional writing.

Reader Preferences for AI-Generated Literary Texts: Fine-Tuning on Copyrighted Works Surpasses Human Experts

Experimental Design and Methodology

The paper systematically evaluates the literary capabilities of frontier LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) against MFA-trained expert writers in emulating the style and voice of 50 internationally acclaimed authors. The experimental protocol comprises two AI conditions: (1) in-context prompting, where models receive detailed instructions and few-shot examples, and (2) author-specific fine-tuning, where GPT-4o is fine-tuned on the complete oeuvre of 30 living authors. Both expert and lay readers perform blinded, pairwise forced-choice evaluations on stylistic fidelity and overall writing quality, with randomized presentation order to mitigate bias.

Figure 1: Study design: author selection, excerpt generation by MFA experts and LLMs (in-context and fine-tuned), and blinded pairwise reader evaluations for style and quality.

The fine-tuning pipeline involves segmenting each author's corpus into context-independent excerpts, extracting content details, and constructing instruction-output pairs for supervised fine-tuning. Post-processing ensures grammaticality and eliminates verbatim overlap with source texts. The evaluation interface presents readers with either two excerpts (for quality) or three (for stylistic fidelity, including the original), requiring preference selection and rationale.

Quantitative Results: Preference Reversal via Fine-Tuning

The core finding is a dramatic reversal in expert reader preferences contingent on the AI training regime. Under in-context prompting, expert readers strongly favor human-written text for both stylistic fidelity (OR = 0.16, $p < 10^{-8}$) and writing quality (OR = 0.13, $p < 10^{-7}$). Lay readers, however, show no significant preference for style and a moderate preference for AI in quality (OR = 1.55, $p = 0.014$). Inter-rater agreement is substantially higher among experts than lay readers.

Fine-tuning on an author's complete works fundamentally alters these outcomes. Experts now prefer AI-generated text for stylistic fidelity (OR = 8.16, $p < 10^{-13}$) and writing quality (OR = 1.87, $p = 0.010$), with lay readers exhibiting similar shifts. Model-based predicted probabilities converge to 0.74 for stylistic fidelity and 0.58–0.61 for writing quality across reader groups.

Figure 2: Odds ratios and predicted probabilities for AI vs. human preference in style and quality; AI detection rates and the relationship between detectability and preference.

Author-level analysis reveals that 27 of 30 fine-tuned models exceed parity for stylistic fidelity and 23 for writing quality, with no systematic relationship between corpus size and performance. The fine-tuning premium (increase in AI preference over in-context prompting) is positive for the majority of authors, indicating robust generalization across diverse literary voices.

Figure 3: Author-level AI preference rates and their independence from fine-tuning corpus size for both stylistic fidelity and writing quality.

Stylometric Analysis and AI Detection

State-of-the-art AI detectors (Pangram, GPTZero) reliably identify in-context AI outputs (97% detection rate) but fail to detect fine-tuned outputs (3% for Pangram, 0% for GPTZero). Higher AI-detection scores are strongly associated with lower expert preference in the in-context setting, but this relationship is eliminated by fine-tuning. Mediation analysis attributes 16.4% of the detection effect to stylometric features (primarily cliché density) in in-context outputs, dropping to 1.3% post-fine-tuning, indicating that fine-tuning removes rather than masks artificial stylistic signatures.

Figure 4: Fine-tuning reduces stylometric AI signatures, improves style/quality, and dramatically lowers first-draft production costs compared to professional writers.

Cost Analysis and Market Implications

Fine-tuning and inference costs per author range from \$25 to \$276 (median \$81), representing a 99.7% reduction compared to professional writer compensation for a novel-length manuscript (\$25,000). Performance gains are uncorrelated with corpus size or cost, suggesting that stylistic emulation is achievable even for authors with limited published works. The economic implications are substantial: high-quality, undetectable AI emulations can be produced at negligible cost, posing significant market displacement risks.

Legal and Societal Implications

The findings have direct relevance to ongoing copyright litigation concerning the use of copyrighted books in LLM training. The fourth fair use factor—market impact—is central: fine-tuned LLMs generate outputs that readers prefer to human-authored works, including those of highly distinctive, prizewinning authors. The outputs do not reproduce verbatim content but emulate style and voice at a level that constitutes credible market substitution. The Copyright Office and recent judicial opinions have recognized the risk of market dilution from AI-generated works enabled by predicate copying, even absent direct reproduction.

The paper suggests that fine-tuning LLMs on individual authors' copyrighted works should not be considered fair use if the outputs compete in the same market. In contrast, general-purpose LLMs used for in-context prompting pose less risk, though lay reader preferences indicate potential substitution even in this regime. The authors propose technical guardrails (e.g., refusal protocols, RL-based steering) to prevent non-parodic stylistic emulation and recommend disclosure of AI origin to mitigate substitution effects.

Limitations

The paper is restricted to short-form excerpts and U.S.-based MFA programs; generalization to long-form works and international contexts requires further investigation. Intrinsic motivation in creative writing and the necessity of human post-editing for publishable prose are acknowledged as factors not fully captured in the cost analysis.

Conclusion

This work demonstrates that author-specific fine-tuning of LLMs on copyrighted literary corpora enables the generation of non-verbatim outputs that both expert and lay readers prefer to those of professionally trained human writers. Fine-tuned outputs are nearly indistinguishable from human writing by current AI detectors and are produced at a fraction of the cost. These results have profound implications for copyright law, labor markets in creative writing, and the future deployment of generative AI in literary domains. Technical and legal interventions will be required to address the risks of market dilution and unauthorized stylistic emulation as LLM capabilities continue to advance.