PPAD: Painting Process Assessment Dataset

• PPAD is the first large-scale dataset that captures the dynamic, multi-stage progression of artistic painting processes with expert annotations.
• It integrates 15,000 real and 10,000 synthetic painting process sequences, enabling detailed temporal analysis through nearly 250,000 key-frame images.
• The dataset serves as a comprehensive benchmark for evaluating computational creativity models and art education tools using eight distinct quality metrics.

The Painting Process Assessment Dataset (PPAD) is the first large-scale, expert-annotated dataset specifically constructed to capture the multi-stage, dynamic nature of artistic painting processes. Unlike prior resources that assess only the static final outcome, PPAD enables fine-grained and quantitative evaluation of artistic progression, serving as a foundational benchmark for the study of computational creativity and models of art education (Jiang et al., 12 Jul 2025).

1. Scope and Objectives

PPAD is designed to address limitations in existing artistic image assessment resources by focusing on the temporal evolution of paintings. Its primary objectives are:

• To enable quantitative study of how artists build up a painting across stages (observation, composition, execution).
• To provide human-aligned, fine-grained annotation along eight distinct dimensions of process quality.
• To serve as a comprehensive training and evaluation benchmark for models assessing painting process quality, exemplified by its initial application in the PPJudge framework.

A critical motivation is the recognition that “quality” in art often emerges from the interplay between consistency with intent, process stability, and the progressive deepening of expressive features—factors that cannot be properly captured by analyzing the terminal output alone.

2. Dataset Composition

PPAD consists of a combination of real and synthetic painting process data, capturing both authentic human artistry and algorithmically generated process trajectories.

2.1. Volume and Structure

• 15,000 real painting processes recorded from art students, each broken into ∼10 key frames per process.
• 10,000 synthetic processes, subdivided into:
  • 5,000 prompt-to-painting sequences generated using ProcessPainter (diffusion-based synthesis).
  • 5,000 inversion sequences generated via Paints-Undo.

The total dataset comprises approximately 250,000 key-frame images sampled from video captures or synthetic generators.

2.2. Source Breakdown and Data Splits

Source	# Sequences	Role	Split
Real (video, key-frames)	15,000	Main train/test	80% train / 20% test (12,000/3,000)
Synthetic – ProcessPainter	5,000	Pre-training	90% pre-train / 10% test (4,500/500)
Synthetic – Paints-Undo	5,000	Pre-training	90% pre-train / 10% test (4,500/500)

It is common practice to hold out 10% of the training set for validation: Real: 12,000 train → 10,800 train / 1,200 val; Synthetic: 9,000 pre-train → 8,100 pre-train / 900 val. This suggests that researchers may employ such splits for model development.

3. Annotation Protocol and Attribute Taxonomy

Each painting process in PPAD is scored by 15 domain experts (art or design majors, age 22–30) who independently rate the process across eight attributes, using an integer scale from 1 (poor) to 10 (excellent). Final attribute labels are the average of all experts’ individual ratings:

3.1. Attribute Categories

• Painting Consistency
  • Consis.: Semantic/visual conformity between final frame and reference.
• Painting Stability
  • Style Stability (S.S.): Smoothness of stylistic progression.
  • Color Stability (Col.S.): Harmonic color transitions.
  • Composition Stability (Com.S.): Consistent compositional layout.
  • Process Stability (Proc.S.): Regular progress, minimal regressions.
• Painting Depth
  • Detail Depth (D.D.): Enrichment of expressive fine details.
  • Color Depth (Col.D.): Increasing palette sophistication over time.
  • Composition Depth (Com.D.): Growth in structural richness.

3.2. Illustrative Attribute Statistics (Real Processes)

Attribute	Mean ± Std. Dev.
Consis.	7.6 ± 1.2
S.S.	7.4 ± 1.3
Col.S.	7.1 ± 1.1
Com.S.	7.5 ± 1.0
Proc.S.	7.2 ± 1.2
D.D.	6.9 ± 1.3
Col.D.	7.3 ± 1.0
Com.D.	7.5 ± 1.1

Real process scores cluster in the 7–8 mean range, whereas synthetic data exhibit a broader spread and distinctive attribute distributions—a property exploitable for contrasting human and algorithmic procedure.

4. Data Acquisition and Preprocessing Pipeline

The data collection procedure integrates both direct human painting capture and algorithmic simulation:

• Recruitment: Hundreds of art students participate, assigned random textual prompts or reference images.
• Recording: Painting sessions are video-recorded, capturing the entire creative process.
• Key-frame Extraction:
  • Uniform temporal downsampling.
  • Pixel-wise L2 difference to identify visually substantive changes.
  • Adaptive thresholding to filter incremental progress; minimum/maximum frame counts enforced for sequence coherence.
• Filtering: Sessions deemed incomplete or invalid are excluded.
• Synthetic Augmentation:
  • ProcessPainter synthesizes prompt-based painting sequences.
  • Paints-Undo generates inversion sequences.
• Dataset Division: Synthetic data primarily supports pre-training; real data and unused synthetic data are used for joint training and testing.

5. Statistical Analyses and Metrics

5.1. Overall Distribution

• Real painting processes: Scores are mid-to-high, with minor skew and tight standard deviation.
• Prompt-to-painting synthetic sequences: Dispersed scores with tendency toward lower stability and depth.
• Inversion sequences: Emphasize final-frame consistency but show variability along stability and depth axes.

5.2. Quantitative Metrics and Formulas

The following formulations are evaluated and/or used in model development associated with PPAD:

• Expert Averaged Score per Attribute:

$$y_{\text{final}} = \frac{1}{K} \sum_{k=1}^{K} y_{\text{expert}}^{(k)}$$

• Rotary Positional Encoding (RoPE) with time-aware modulation:

$$\alpha_{ij}^{\mathrm{RoPE}} = q_i^\top R(i-j) k_j, \quad \theta_{\mathrm{time}}(t) = -\beta\,\theta_{\mathrm{base}}\left(\frac{t}{T}\right)^\gamma$$

• Style Loss (Cosine Similarity):

$$L_{\mathrm{style}}^l = 1 - \frac{E_{\mathrm{style}} \cdot E_{se}^l}{\|E_{\mathrm{style}}\| \|E_{se}^l\|},\quad L_{\mathrm{style}} = \sum_{l=1}^L \alpha_l L_{\mathrm{style}}^l$$

• Score and Total Loss:

$$L_{\mathrm{score}} = \frac{1}{n} \sum_{i=1}^n (y_{i} - \hat y_{i})^2,\quad L_{\mathrm{total}} = L_{\mathrm{style}} + \lambda_{\mathrm{score}} L_{\mathrm{score}}$$

A plausible implication is that these metrics provide multi-level supervision—enabling both process-aligned (temporal) and attribute-aligned learning objectives.

6. Access, Licensing, and Ethical Considerations

• Release: Public availability is scheduled via the project website; specific URLs will be announced at publication.
• License: CC BY-NC 4.0, restricting use to non-commercial research.
• Ethics:
  • Informed consent obtained from all human participants.
  • Complete anonymization of video content; personally identifiable information is excluded.
  • Synthetic data is generated using open-source models under permissive licenses.

7. Context and Applications

PPAD is foundational for training and evaluating models such as PPJudge, which utilize its comprehensive, temporally resolved annotations for human-aligned artistic process assessment. It addresses key gaps in computational art research where process quality, not only outcome quality, is critical. Its design facilitates new inquiries in art education and computational creativity, providing rare insight into the progression of visual thought and technique in art-making (Jiang et al., 12 Jul 2025).