Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization

Abstract: Stylistic personalization - making LLMs write in a specific individual's style, rather than merely adapting to task preferences - lacks evaluation grounded in authorship science. We show that grounding evaluation in authorship verification theory transforms what benchmarks can measure. Drawing on three measurement traditions - LUAR, a trained authorship verification model; an LLM-as-judge with decoupled trait matching; and classical function-word stylometrics - we evaluate four inference-time personalization methods across 50 authors and 1,000 generations. The theory-grounded metric, LUAR, provides what ad hoc alternatives cannot: calibrated baselines, with a human ceiling of 0.756 and a cross-author floor of 0.626, that give scores absolute meaning. All methods score below this floor, from 0.484 to 0.508, exposing an authorship gap invisible to uncalibrated metrics. The three metrics produce near-zero pairwise correlations, with absolute r less than 0.07, confirming that without theoretical grounding, metric choice determines conclusions: an LLM judge declares a clear winner while LUAR finds no meaningful differentiation. These findings demonstrate the theory-benchmark cycle in action: authorship theory exposes evaluation failures that ad hoc benchmarks miss.

• The paper demonstrates that current personalization methods yield LUAR scores significantly below the human cross-author floor, revealing a measurable authorship gap.
• It employs a multi-metric evaluation framework—LUAR, LLM-as-Judge TMR, and FuncCos—to capture distinct aspects of stylistic fidelity.
• The study warns against method-dependent circularity and calls for theory-based benchmarks to achieve authentic stylistic personalization.

Theory-Grounded Evaluation Reveals the Authorship Gap in LLM Personalization

Introduction

Stylistic personalization of LLMs—the ability to generate text indistinguishable from a specific individual's writing style—is critically under-evaluated due to the lack of benchmarks grounded in authorship science. Prevailing evaluation frameworks focus on task accuracy or preference alignment but neglect the fundamental question of whether generated text authentically reflects the target author’s stylistic fingerprint. This paper introduces a theory-grounded evaluation protocol based on computational authorship verification, demonstrating that existing personalization methods fail to achieve genuine authorial style fidelity. By leveraging the Learning Universal Authorship Representations (LUAR) metric, an LLM-as-judge protocol, and classical stylometrics, the study exposes a measurable authorship gap and provides a calibrated approach for evaluating stylistic personalization.

Evaluation Framework

The evaluation utilizes the Blog Authorship Corpus, featuring posts from 50 selected authors, and tests four inference-time personalization methods: Non-Personalized, Few-Shot, Profile Extraction, and Contrastive. These methods range from controls to advanced style transfer, all using current-generation LLMs (Qwen 3 32B and GLM-4 32B).

Three independent metrics are employed:

• LUAR: Transformer-based embeddings trained via contrastive learning for authorship discrimination, yielding calibrated baselines (same-author, cross-author, human-generated, LLM-generated).
• LLM-as-Judge (Trait Match Rate, TMR): Automated trait-based scoring by a decoupled protocol, extracting binary style traits per author.
• Function Word Stylometrics (FuncCos): Cosine similarity over 60 function words, providing a classical stylometric perspective.

Main Findings and Quantitative Results

All personalization methods scored below the human cross-author floor on the LUAR metric, with values ranging from 0.484 to 0.508; the reference floor for human cross-author pairs is 0.626, and the same-author ceiling is 0.756. This demonstrates a calibrated authorship gap: LLM-generated personalized text is less similar to the target author's style than random human authors are to each other. Despite internal author differentiation (gen↔gen AUC = 0.918), LLM output does not cross into the human authorship regime (gen→real AUC = 0.632).

Figure 1: LUAR authorship similarity scores by personalization method, calibrated against human-authored baselines, showing all methods below cross-author human floor ($0.626$).

Metric correlations are near zero ($|r| < 0.07$), indicating these metrics capture fundamentally different constructs. The LLM-as-judge declares Profile Extraction the clear winner (TMR effect size $d=0.58$ over baseline), but this signal is absent in LUAR results. The LLM judge’s response is traced to protocol circularity: both trait extraction and profile extraction rely on LLM-extracted features, incentivizing instruction-following rather than genuine stylistic mimicry.

Figure 2: LUAR similarity versus TMR for 1,000 generations, highlighting non-correlated distributions—strong TMR scores do not indicate LUAR improvement ($r=0.013$).

Profile extraction even outperforms the real author on TMR, confirming that trait-based evaluation measures instruction-following instead of authentic authorship fidelity. Trait extraction is not stable (mean Jaccard similarity = 0.22 between trait sets), amplifying construct validity concerns.

Cross-model evaluation (GLM-4 32B) replicates the authorship gap: LUAR scores for all methods are below the cross-author floor, establishing robustness across architectures. Prompt contamination, such as naive first-sentence extraction, can inflate baselines by 28 percentage points, warranting rigorous prompt design to avoid confounds.

Benchmarking Implications and Construct Validity

The authorship gap calls for benchmarks explicitly anchored in authorship science. LUAR’s calibrated baselines transform author-style evaluation from relative scoring to absolute measurement, establishing formal targets for personalization performance. Multi-metric disagreement is diagnostic: the absence of pairwise correlation among metrics reflects a failure of construct validity and signals measurement artifacts rather than substantive capability. Circularity between metric and method undermines trait-based protocols, underscoring the necessity for independent, theory-backed evaluators.

Benchmarking must adhere to theory–benchmark cycles: theoretical predictions (LLM fingerprint persistence) must be validated through calibrated metrics. LUAR offers falsifiable, quantitative targets for the achievable fidelity of LLM personalization. The generated-text regime exhibits structural separation in embedding space—personalized outputs differentiate authors within LLMs but do not traverse into the human manifold. Achievable authorship embeddings under inference-time methods appear bounded away from human territory, establishing a theoretical limit for personalization strategies.

Limitations and Future Directions

Evaluation is confined to blog-style English text, two LLM families at 32B scale, and inference-time methods. Training-time personalization (e.g., per-user LoRA adapters) may reduce the gap, but LUAR provides a scalable measuring stick for such advances. There is evidence for analogous personalization failures in behavioral domains, suggesting the authorship gap encompasses more than linguistic style. Human evaluation is necessary to align metric outputs with perceived authorship fidelity. Broader language coverage and integration of human similarity judgments warrant further investigation.

Conclusion

Theory-grounded evaluation reveals a persistent authorship gap in current LLM personalization protocols, quantified by LUAR and invisible to ad hoc metrics. All methods evaluated fail to produce text stylistically faithful to target authors, substantiating the need for benchmarks with calibrated baselines rooted in authorship science. The results prescribe that future assessment and development of personalized LLMs must be anchored in validated, absolute metrics, and warn against reliance on construct-invalid or method-dependent evaluations. The proposed framework defines clear quantitative targets for closing the gap, shaping the theoretical and practical trajectory of LLM personalization research.