---
title: 'ArtEmoBenchmark: Emotion in Art Evaluation'
url: https://www.emergentmind.com/topics/artemobenchmark
type: topic
---

# ArtEmoBenchmark: Emotion in Art Evaluation

ArtEmoBenchmark is a comprehensive benchmarking suite for evaluating artificial intelligence (AI) systems on the understanding and generation of emotion in artwork across multiple modalities, genres, and input types. It provides standardized tasks, metrics, and high-fidelity annotations, enabling rigorous assessment of models in emotion-driven art generation, perception, and cross-modal affect comprehension. ArtEmoBenchmark draws from large-scale, multidimensional datasets and defines protocols for both text-to-image diffusion models and multimodal language models, with particular emphasis on the complexity of emotion as conveyed through visual and auditory elements in artistic contexts.

## 1. Dataset Composition and Scope

ArtEmoBenchmark is grounded in two core resources: the EmoArt dataset and the ArtEmoBenchmark test set constructed for multimodal AVLMs.

### EmoArt Dataset

- **Size and Diversity**: 132,664 paintings spanning 56 distinct painting styles, including Impressionism, Surrealism, Ukiyo-e, and Modernism, ensuring comprehensive stylistic and cultural coverage.
- **Annotations**:
  - Detailed scene descriptions (average 35.6 words per entry).
  - Five structured visual attributes: brushwork, composition, color, line, and light.
  - Binary Arousal-Valence labeling (High/Low Arousal × Positive/Negative Valence).
  - Twelve discrete emotion categories distributed to evenly cover the Russell circumplex: aroused, excited, happy, alarmed, annoyed, frustrated, sad, bored, tired, content, calm, glad.
  - Art-therapy potential descriptors (e.g., calming, uplifting).
- **Annotation Quality**: Annotations generated through a GPT-4o assisted, multi-stage pipeline with human verification over 5,600 image samples. High inter-annotator agreement is reported: 85.25% for emotion (Gwet’s AC1 = 0.785), 95.25% for visual attributes.
- **Distributional Properties**: 87.93% of the corpus has positive valence; low-arousal emotions dominate. Style distribution ensures ≥400 images per painting genre.

### ArtEmoBenchmark (Audio-Visual)

- **Source**: 1,200 expertly curated 30 s movie clips from 1,432 films, each with emotionally motivated music.
- **MCQ Construction**: The benchmark comprises 1,200 multiple-choice questions (MCQs), balanced across three input modalities: audio-only, visual-only, and audio-visual joint, with manual post-processing and verification.
- **Emotion Model**: Continuous Valence-Arousal (V–A) labeling, with scores in [–1, +1] per axis, annotated independently by five experts per condition (audio-only, visual-only, audio-visual), then averaged.

## 2. Benchmark Tasks and Modalities

ArtEmoBenchmark specifies two main classes of tasks:

### A. Image Generation Tasks (Text-to-Image)

- **Emotion-Conditional Generation**: Given a specific painting style ‘s’ and an emotion ‘e’ (from the discrete set), generate an image $\hat{I} = G_A(s, e)$ that optimally conveys the target emotion in the designated style.
- **Full Text-to-Image Generation**: Conditioned on $(s, v, a, d)$ where $v$ is valence, $a$ is arousal, and $d$ is a content-rich scene description; the task is to generate $\hat{I} = G_B(s, v, a, d)$. The formal objective is:
  $$
  L(G) = \lambda_\text{attr}\,L_\text{attr}(G(x)) + \lambda_\text{img}\,L_\text{img}(G(x))
  $$
  with $L_\text{attr}$ as CLIP-based embedding cosine distance and $L_\text{img}$ as Fréchet Inception Distance (FID).

### B. Multimodal Emotion Understanding Tasks

- **Audio-Only, Visual-Only, Audio-Visual Classification**: On the 1,200 MCQs from ArtEmoBenchmark, models must answer questions (overall/specific, content/emotion) restricted to their assigned modality or combination thereof.
- **Types of Questions**:
  - OC/OE: Overall content and emotion (per modality).
  - SC/SE: Specific content and specific emotion (per modality).
  - Sp-A, Sp-V, Sp-AV: Audio-centric, video-centric, and cross-modal questions (for audiovisual tasks).

## 3. Evaluation Protocols and Metrics

### Quantitative Metrics

- **Discrete Emotion Classification**:
  - Accuracy: $\text{Acc} = \frac{1}{N} \sum_{i = 1}^{N} 1\{y_i = \hat{y}_i\}$
  - F1-score (per emotion), Precision, Recall.
- **Continuous V–A Correlation**: Pearson’s $r$ between annotated and predicted arousal/valence vectors.
- **CLIP Alignment Score**: Cosine similarity between image embedding and text prompt embedding.
- **Fréchet Inception Distance (FID)**: For distributional similarity between generated and real artwork.
- **Pixel-Based Metrics**: SSIM, PSNR, LPIPS (standard image quality definitions).
- **Attributes Alignment**: Fine-tune a classifier on visual attributes; score as CLIP similarity between predicted attribute text and generated images.
- **Audio-Visual MCQ Accuracy**: Simple proportion correct per subtask and overall mean.

### Annotation Protocol and Agreement

High consistency is ensured by multiple human annotators, particularly for continuous V–A labeling, with all MCQs manually checked for correctness and clarity. In image tasks, attributes and emotions are verified on random samples with Gwet’s AC1 and percent agreement reported.

## 4. Baseline Models and Comparative Results

### Diffusion Models (Art Generation)

| Model                 | Overall Attribute Alignment | FID ↓    | LPIPS ↓ | SSIM ↑ | PSNR ↑  |
|-----------------------|----------------------------|----------|---------|--------|---------|
| FLUX 1-dev            | 0.6392                     | 21.29    | 0.6706  | 0.2108 | 9.57    |
| PixArt-σ              | 0.6505                     | 36.23    | 0.6754  | 0.1658 | 8.99    |
| SDXL                  | 0.6385                     | 61.93    | 0.7110  | 0.1677 | 9.13    |
| FLUX 1-dev-finetuned  | 0.6604                     | 31.65    | 0.6508  | 0.2102 | 9.66    |

Removal of emotion labels during fine-tuning reduces Attributes Alignment by 11% and increases FID by 4.3; exclusion of “Art Therapy” fields does not degrade standard metrics but does lower subjective emotional intensity.

### Audio, Visual, and Audio-Visual Language Models

| Model                | Audio (A) | Visual (V) | Audio-Visual (AV) | Overall (All) |
|----------------------|-----------|------------|-------------------|---------------|
| Qwen-Audio           | 52.5%     | –          | –                 | –             |
| Qwen2.5-VL           | –         | 72.3%      | –                 | –             |
| VideoLLaMA2          | 60.8%     | 81.3%      | 36.5%             | 59.5%         |
| AffectGPT            | 51.5%     | 60.8%      | 42.0%             | 51.4%         |
| VAEmotionLLM         | 77.0%     | 87.8%      | 53.5%             | 72.8%         |

Ablations confirm that integrated adapters and emotion modules in VAEmotionLLM provide complementary gains: Audio Adapter alone gives no gain; LoRA or single modules give moderate increases; the full VAEmotionLLM achieves the highest accuracy on every modality [2511.12077].

## 5. Methodological Features and Extensions

### Reproducibility

Detailed protocols facilitate reproducibility:

- EmoArt is available at https://huggingface.co/datasets/printblue/EmoArt-130k under CC BY-NC 4.0.
- Preprocessing: standardize all images to 512×512 with provided JSON annotations.
- Training: LoRA fine-tuning on FLUX 1-dev (learning rate 1e–4, batch size 4, 5 epochs on 2,800 images).
- Inference: CFG scale 7.5, 50 DDIM steps, fixed seed, template-based prompts.
- Evaluation scripts for all metrics are provided in the repository.

### Extensions

- Modalities: Benchmark is extensible to video and 3D by mapping arousal–valence dimensions to temporal or volumetric controls.
- Annotation Ablations: Researchers can disable individual attribute fields or emotion dimensions to analyze their impact on generation.
- Emotion Classification: Zero-shot classifiers trained on EmoArt descriptions can be evaluated on generated samples for assessment of transferability.
- New Model Adaptation: Prompt templates and LoRA recipes are compatible with new diffusion architectures (e.g., Imagen, DALL·E).

## 6. Relation to Prior Benchmarks and Multimodality

ArtEmoBenchmark is situated at the convergence of art-focused sentiment datasets such as ArtEmis and ArtELingo, and general multimodal affect understanding benchmarks:

- **Multilingual and Cross-cultural Expansion**: ArtELingo extends prior collections with multilingual (Arabic, Chinese, Spanish) and culturally divergent annotations, highlighting systematic variations in emotion perception tied to culture [2211.10780].
- **Expressive Modalities**: Other datasets, such as MEMO-Bench, inform the dual-axis structure (affect generation and understanding), progressive testing from coarse labels to fine intensity, and human-in-the-loop annotation methods [2411.11235].
- **3D Emotionality**: Related research in 3D facial expression rendering emphasizes axes of expression diversity, fluidity, and content alignment, suggesting plausible future extensions to 3D art-driven avatars [2412.02508].

## 7. Limitations and Future Directions

Despite its scale and methodological rigor, ArtEmoBenchmark retains several limitations:

- Cultural bias: EmoArt and ArtEmoBenchmark are predominantly Western-centric in art corpus selection; cultural nuance among non-Western genres may be underrepresented.
- Annotation model: While continuous V–A annotation is a notable strength, reliance on human verification in a limited language group constrains global emotional diversity.
- Modal expansion: Video, 3D, haptic modalities, and dynamic interactive art are not fully addressed.
- Metric comprehensiveness: Only accuracy is reported for MCQ tasks; future efforts could incorporate F1, calibration, and perceptual alignment metrics.

A plausible implication is that expanding ArtEmoBenchmark to cover additional art forms, languages, affect taxonomies, and modalities will further advance research in comprehensive machine understanding and generation of emotion in artistic media.

---

**References:**  
- "EmoArt: A Multidimensional Dataset for Emotion-Aware Artistic Generation" [2506.03652]  
- "Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Sound" [2511.12077]  
- "MEMO-Bench: A Multiple Benchmark for Text-to-Image and Multimodal Large Language Models on Human Emotion Analysis" [2411.11235]  
- "ArtELingo: A Million Emotion Annotations of WikiArt with Emphasis on Diversity over Language and Culture" [2211.10780]  
- "Towards Rich Emotions in 3D Avatars: A Text-to-3D Avatar Generation Benchmark" [2412.02508]

Source: https://www.emergentmind.com/topics/artemobenchmark