How LLMs See Creativity: Zero-Shot Scoring of Visual Creativity with Interpretable Reasoning

Abstract: Evaluating the originality of visual images poses enduring challenges for creativity assessment. Automated scoring using AI models has proven effective in the verbal domain, yet key questions remain about evaluating visual creativity and understanding how models arrive at their ratings. The present research asks whether multimodal LLMs can serve as judges of visual creativity zero-shot (without any fine-tuning or examples of human ratings) and whether their "reasoning" output offers an interpretable window into their evaluation process. We tested six multimodal LLMs (Gemini 3 Flash, Gemma 4 31B IT, GPT-5.4 Mini, GLM-5v Turbo, Kimi K2.5, and Qwen 3.6 Plus) on 992 AI-generated images (based on human-written prompts) and 1,500 hand-drawn sketches scored for creativity by human raters. In Study 1, all models showed substantial alignment with human creativity ratings on both datasets (r = .57-.68 on AI-generated images; r = .29-68 on sketches). In Study 2, we analyzed the step-by-step reasoning processes of three LLMs evaluating the same images and drawings. Although reasoning made model evaluations interpretable -- showing what they attend to, how they balance originality vs. quality, and how they justify their ratings -- reasoning did not improve alignment with human ratings. In sum, our findings indicate that multimodal LLMs can match human judgments of visual creativity without any additional training, and that their reasoning reveals how AI models evaluate creativity. An open scoring app implementing this pipeline is available at https://review-visual-eval-scoring.hf.space.

• The paper demonstrates robust zero-shot alignment between multimodal LLM ratings and human creativity assessments, with correlations ranging from 0.57 to 0.68 on image tasks.
• The paper uses structured reasoning chains to analyze evaluative dimensions such as Perception, Originality, and Quality, revealing biases like leniency in AI images and harshness in hand-drawn sketches.
• The paper highlights that LLMs capture high-level creative criteria beyond low-level visual complexity, as partial correlations confirm that object recognition errors only modestly affect ratings.

Zero-Shot Visual Creativity Assessment by Multimodal LLMs: Capabilities and Reasoning

Problem Statement and Background

The assessment of visual creativity remains a significant bottleneck in empirical creativity research, especially as generative AI systems rapidly escalate the volume and complexity of visual artifacts. Traditional human-based methods such as the Consensual Assessment Technique are resource-intensive, difficult to scale, and subject to item- and rater-specific variance. Prior automated approaches for creativity evaluation have focused predominantly on verbal outputs, leveraging LLMs trained or prompted to align with human judgment. Recent progress in automated scoring of divergent thinking and figural creativity tasks (e.g., alternate uses, metaphor production, drawing completion) has demonstrated that alignment with human ratings is attainable—sometimes even in zero-shot settings without fine-tuning or in-context exemplars. However, analogous capability in the visual domain has not been rigorously established, and it remains unclear whether LLM-based creativity assessment can generalize across domains or afford interpretability into underlying evaluative processes.

Experimental Design

The study evaluates the capacity of six state-of-the-art multimodal LLMs (Gemini 3 Flash, Gemma 4 31B IT, GPT-5.4 Mini, GLM-5v Turbo, Kimi K2.5, and Qwen 3.6 Plus) to serve as zero-shot judges of visual creativity. Two distinct datasets are utilized: (1) 992 DALL-E 3-generated images from human-written prompts, rated for creativity by human judges, and (2) 1,500 human-drawn sketches created by extending incomplete abstract shapes, again with expert creativity ratings.

Models were prompted in a deterministic, zero-shot manner, with instructions closely matched to the original human evaluation tasks. To control for potential confounds related to image complexity, edge density was measured and partial correlations were computed. Subset analyses and per-shape breakdowns probed item-level consistency in model-human agreement, and distributions of model ratings relative to human-scale usage were examined.

For interpretability, explicit reasoning chains generated by three models (GLM-5v Turbo, Kimi K2.5, Qwen 3.6 Plus) were analyzed. Chains were decomposed into sentences and classified into fine-grained evaluative categories using an automated LLM-based coding pipeline. Ancillary analyses evaluated the influence of object recognition accuracy on rating alignment and assessed the extent to which evaluative content within chains predicted model-human divergence.

Main Empirical Findings

1. Strong Zero-Shot Alignment Across Modalities:

All six tested multimodal LLMs showed robust positive alignment with human creativity ratings in zero-shot settings: on AI-generated images, model-human $r$ ranged from 0.57 to 0.68; on hand-drawn sketches, $r$ spanned 0.29 to 0.68, with the highest consistency observed for Gemini 3 Flash. Notably, Kimi K2.5 and Qwen 3.6 Plus (both open-source) outperformed GPT-5.4 Mini on the more challenging human-drawn sketches.

2. Impact of Edge Density and Complexity Biases:

While edge density (as a visual complexity proxy) moderately predicted creativity ratings, partialing out this effect attenuated but did not eliminate model-human alignment, especially on sketches. Thus, models are not solely regurgitating low-level complexity metrics; they capture additional high-level criteria judged as creative by humans.

3. Systematic Biases:

Models systematically over-rated creativity in AI-generated images (displaying a leniency bias), while under-rating hand-drawn sketches (a harshness bias), compressing scale usage relative to human raters. These biases varied in direction by input modality and persisted despite explicit prompt instructions to focus on originality rather than polish or quality.

4. Reasoning Chains: Structure and Predictive Validity:

Reasoning chains offered no consistent improvement to model-human alignment over baseline (non-reasoning) ratings—maximum observed delta was +0.012, with some models' alignment slightly degraded in reasoning mode. However, chains were highly structured, with content distributed over four interpretable dimensions: Perception, Originality, Quality, and Justification. Relative contributions of these categories shifted by input domain (more Originality and Quality focus for AI images, more Perception for line sketches). Chain language alone (image-blinded) allowed another LLM to reconstruct model-assigned ratings with high accuracy ($r=0.82$ image-wise), indicating that chains are not mere post-hoc rationalizations but encode the core evaluative decisions.

5. Divergence Between Model and Human Evaluations:

Disparities in model-human judgments were explained by the relative emphasis of Originality (generally lowering ratings) versus Quality (increasing ratings) in chains. Notably, when models and humans differed, these differences tracked with the evaluative focus detectable in the reasoning text.

6. Object Recognition Independence:

Models correctly identified depicted objects in sketches in ~65% of cases. Misidentification only modestly reduced alignment; surprising, highest alignment ($r=0.55$) was seen for drawings that human raters themselves could not label (“no-consensus drawings”). This suggests both models and humans resort to domain-general compositional and elaborative criteria in the absence of clear object identity.

Theoretical and Practical Implications

This work provides compelling evidence that advanced multimodal LLMs, in a strict zero-shot setting, match human visual creativity assessment with moderate to high fidelity, extending prior findings limited to the verbal domain. The consistent structure of model-generated reasoning chains offers a window into LLM “thought processes,” validating that ratings are grounded in the textually reconstructible application of core creativity criteria—albeit still susceptible to domain- and complexity-driven biases. The persistence of these biases reflects calibration deficits and potential self-preference/familiarity biases reported elsewhere in the LLM-evaluator literature.

Practically, the findings endorse the feasibility of scalable, interpretable, automated scoring pipelines for large-scale creativity research, offering substantial gains over labor-intensive human approaches. The accompanying open-source scoring tool operationalizes these advances for broader research use, lowering barriers to entry for standardized visual creativity assessment across diverse domains.

Theoretically, the ability of LLMs to align with human ratings independently of object recognition accuracy indicates that these models have internalized certain visual and conceptual principles associated with creative evaluation. However, performance is not uniform across input types or models: fine distinctions (e.g., between sparse line sketches) still challenge certain systems, and reliance on elaboration signals can confound true originality measurement—issues that punctuate the need for further task-specific calibration or architectural refinement.

Prospects for Future AI Research

This study motivates several future directions:

• Cross-domain Generalization: Further research should assess the robustness of zero-shot creativity scoring across less-structured visual modalities (e.g., design, mixed media, photography) and tasks with more nuanced or culturally specific creativity concepts.
• Bias Calibration and Correction: Addressing systematic leniency and elaboration biases is critical for any automated scoring paradigm to ensure construct validity and comparability across heterogeneous data sources.
• Process Transparency and Reasoning Audits: The demonstrated feasibility of automated, fine-grained coding of LLM reasoning chains can be extended to other evaluative tasks, serving as a method for algorithmic auditing and interpretability.
• Integration with Human-AI Co-Creativity Systems: Understanding how LLMs judge and explain creativity has implications for feedback loops in generative co-creation or AI-supported human ideation, potentially informing interface design or educational interventions.

Conclusion

This research establishes that current-generation multimodal LLMs can perform zero-shot visual creativity judgments at a level competitive with human raters, with structured and interpretable reasoning traces. Alignment generalizes across diverse stimulus types and remains robust to object recognition errors and visual complexity confounds. Nonetheless, systematic scale-compression and familiarity-driven biases highlight the need for careful model selection and ongoing validation in scoring pipelines. Beyond their immediate application in standardized assessment, the interpretability of LLM-based evaluation offers a promising vector for transparency and accountability in future AI-augmented creative research and practice.

---

Citation: "How LLMs See Creativity: Zero-Shot Scoring of Visual Creativity with Interpretable Reasoning" (2606.29672)