Are LLMs becoming similarly creative? Evidence from three years of models

Abstract: Many benchmarks track LLM performance on tasks with verifiable answers, but less is known about how LLM performance is evolving on open-ended tasks, where creativity, originality and diversity may matter as much as quality. As LLMs increasingly support human ideation and creative work, understanding trends in LLM performance on open-ended tasks is critical. This paper presents a preliminary analysis of LLM creative outputs spanning three years of model releases, examining model responses to Infinity-Chat100, a real-world collection of open-ended user queries, and the Alternate Uses Task, an established psychometric creativity assessment. Using sentence-embedding similarity, we examine trends in LLM responses to these prompts. Our findings show a statistically significant decrease in model output diversity over time, suggesting that LLM outputs may be converging in creative substance across models. If this trend persists, LLM-driven homogenization may progressively diminish human agency in human-AI co-creative work, demanding careful consideration of LLMs' role in the human creative process.

• The paper finds that semantic diversity in LLM outputs declined consistently across 68 models from 12 providers released between 2023 and 2026, with cross-family cosine-distance slopes of −0.01385 on the Alternate Uses Task and −0.00167 on Infinity-Chat100.
• The authors combine embedding-based similarity analysis with family-balanced resampling across 1,000 iterations, finding negative convergence trends in every iteration and a sharp AUT distance decline from about 0.50 to below 0.40.
• The results suggest that users of different AI systems may encounter increasingly similar ideas, while shared training data, distillation, alignment methods, sampling choices, and prompt design remain possible explanations requiring further study.

Motivation and contribution

Most LLM benchmarks—MMLU, HELM, SWE-Bench, Humanity's Last Exam—evaluate performance against verifiable answers, leaving open-ended generation largely untracked. This matters because creative and ideation tasks constitute a substantial share of real-world usage: practical guidance (including "creative ideation") and writing account for over half of ChatGPT messages, and roughly 20% of Claude messages fall into content creation or academic writing categories. Prior work has established that LLM outputs can be individually creative yet collectively homogeneous—the "AI creativity paradox"—but every existing study is a snapshot of particular models at a particular time. No prior analysis tracks how homogeneity evolves across model generations.

This paper fills that gap with the first longitudinal study of LLM output diversity. The authors evaluate 68 models from 12 providers, released between March 2023 and July 2026 (33 closed-weight, 34 open-weight), on two complementary prompt sets: the Alternate Uses Task (AUT), a standardized divergent-thinking assessment, and Infinity-Chat100, a set of 100 real-world open-ended user queries spanning creative content generation, brainstorming, ideation, and information seeking.

Methodology

The pipeline proceeds in four stages: prompt selection, generation across model lineages, embedding-based semantic distance computation, and regression over time. All generations use temperature and top-$p$ of 1.0 via OpenRouter's API, retaining stochastic sampling without distorting the output distribution. Responses are embedded with all-MiniLM-L6-v2; for the AUT, all uses per object are pooled into one embedding per model (10 embeddings per model), while each Infinity-Chat100 response is embedded separately (100 per model).

To test temporal trends, models are ordered by release month into nine bins of three timestamps each, and only cross-family model pairs are compared, minimizing confounds from shared architecture or training data. For each pair $(i,j)$, prompt-level cosine distances $$d_{ij,p} = 1 - \cos(\mathbf{x}_{i,p}, \mathbf{x}_{j,p})$$ are averaged across prompts to yield $\bar{d}_{ij}$, which is regressed on the ordinal bin index via OLS. Because families contain unequal numbers of models, the authors run 1,000 iterations of family-balanced resampling—one model per family per bin—and report the median slope with empirical 2.5th/97.5th percentile intervals. A tight interval indicates the trend is not an artifact of any particular model selection.

The authors acknowledge that release date is an imperfect proxy for generational change (newer data, architectures, post-training methods), so trends should be read as associations with broader development patterns rather than causal effects of time.

Results

Cross-family cosine distance declines monotonically for both prompt sets:

Dataset	Models (pairs)	Slope per bin	95% CI
Alternate Uses Task	68 (273)	−0.01385	[−0.01695, −0.01044]
Infinity-Chat100	67 (268)	−0.00167	[−0.00267, −0.00074]

All 1,000 resampling iterations produced negative slopes for both datasets, indicating the direction of the trend is robust to which model represents each family. The decline is most pronounced on the AUT, where mean cross-family distance falls from approximately 0.50 in the earliest bin to below 0.40 in the most recent—a notable result precisely because the AUT explicitly instructs respondents to produce original, unexpected uses, making it the setting where divergence would be expected. The Infinity-Chat slope is shallower (roughly 0.34 to just above 0.32) but spans a heterogeneous collection of naturalistic prompts, so the decline is unlikely to reflect the demands of any single task format. Notably, the Infinity-Chat decline is concentrated in release bins from 2025 onward, which may mark the beginning of a longer-term trend warranting continued monitoring.

The implication is direct: if cross-provider outputs are converging semantically, users of different commercial systems are increasingly exposed to the same ideas, bounding the range of possibilities encountered in AI-assisted creative work.

Limitations and open questions

The paper concedes several constraints on interpretation. First, the regression cannot account for hidden relationships across model families—distillation, shared training corpora, or common training techniques—which could drive convergence independently of any intrinsic dynamic. Second, several Infinity-Chat prompts request similar outputs (e.g., two prompts about Zootopia); this does not bias within-prompt comparisons but suggests the need for a more principled unstructured prompt set. Third, sampling a single response per model per prompt yields only a point estimate of each model's output distribution; comparing full distributions from repeated generations would characterize creativity more richly. Fourth, the analysis excludes the user's role: prompting alone cannot prevent homogeneity, but incentive structures and savvier human usage can mediate it. Finally, OLS diagnostics show modest right-tail deviation for AUT residuals and influential observations concentrated in sparse early bins, though family-balanced resampling mitigates this.

Open questions left by the paper include whether convergence is uniform across task types, categories, and output lengths; whether text-level measures (Jaccard similarity, structural and stylistic features) corroborate the embedding-based findings; and how sensitive the trend is to sampling parameters, prompt wording, model size, and family.

Conclusion

Across three years of model releases and two complementary prompt sets, cross-provider semantic distance in LLM creative outputs declines consistently and robustly, with the steepest convergence on the divergent-thinking task designed to elicit variation. The finding extends snapshot observations of LLM homogeneity into a longitudinal trend and raises the possibility that increasingly homogeneous models could compound known effects of AI assistance on collective human creativity. Whether the trend persists, and what mechanisms—shared data, distillation, or alignment-driven mode collapse—drive it, remains unresolved.