SEA: Evaluating Sketch Abstraction Efficiency via Element-level Commonsense Visual Question Answering

Abstract: A sketch is a distilled form of visual abstraction that conveys core concepts through simplified yet purposeful strokes while omitting extraneous detail. Despite its expressive power, quantifying the efficiency of semantic abstraction in sketches remains challenging. Existing evaluation methods that rely on reference images, low-level visual features, or recognition accuracy do not capture abstraction, the defining property of sketches. To address these limitations, we introduce SEA (Sketch Evaluation metric for Abstraction efficiency), a reference-free metric that assesses how economically a sketch represents class-defining visual elements while preserving semantic recognizability. These elements are derived per class from commonsense knowledge about features typically depicted in sketches. SEA leverages a visual question answering model to determine the presence of each element and returns a quantitative score that reflects semantic retention under visual economy. To support this metric, we present CommonSketch, the first semantically annotated sketch dataset, comprising 23,100 human-drawn sketches across 300 classes, each paired with a caption and element-level annotations. Experiments show that SEA aligns closely with human judgments and reliably discriminates levels of abstraction efficiency, while CommonSketch serves as a benchmark providing systematic evaluation of element-level sketch understanding across various vision-LLMs.

• The paper introduces the SEA metric, a reference-free measure combining class recognizability, commonsense elements, and visual element presence.
• It leverages the CommonSketch benchmark with over 23,000 sketches across 300 classes to evaluate abstract representations using element-level annotations.
• Empirical results demonstrate strong alignment with human scores and potential for enhancing abstraction-driven sketch generation and model selection.

A Formal Analysis of "SEA: Evaluating Sketch Abstraction Efficiency via Element-level Commonsense Visual Question Answering"

Introduction and Motivation

The paper "SEA: Evaluating Sketch Abstraction Efficiency via Element-level Commonsense Visual Question Answering" (2603.28363) presents a new paradigm for quantifying semantic abstraction efficiency in hand-drawn sketches. The core focus is the development of SEA, a reference-free metric targeting how economically a sketch conveys canonical class-defining elements while preserving semantic recognizability. This direction addresses fundamental limitations in current sketch understanding benchmarks and evaluation protocols, which largely focus on pixel-, feature-, or label-level correctness without addressing the abstraction process defining sketch semantics.

Central to this is the introduction of CommonSketch, a large-scale, semantically-annotated benchmark comprising 23,100 human sketches across 300 classes with element-level and caption annotations, enabling robust VQA-based and element-aware analysis. The research considers abstraction efficiency under the lens of class-specific commonsense visual elements, systematically extracted using LLMs and verified via human annotation.

Figure 1: Overview of SEA and CommonSketch. High-abstraction sketches maximize recognizability with minimal detail, and CommonSketch enables element-level evaluation.

Methodology: The SEA Metric

SEA Metric Formulation

The SEA metric is explicitly designed for abstraction efficiency, integrating three primary signals:

• Class recognizability $(P)$: The predicted probability of the ground-truth class, typically using a zero-shot classifier (e.g., CLIP).
• Commonsense element set $(E)$: The cardinality of drawable, class-representative visual elements, extracted from LLMs (primarily GPT-4o and open-weights LLMs for open variants).
• Element expression $(V)$: The count of these elements visually present in a given sketch, automatically identified via VLM-based VQA.

Given a sketch, the pipeline proceeds as follows:

1. Class-specific element extraction ($E$): LLMs output a fine-grained, visually labelable set of semantic elements per class.
2. Element presence detection ($V$): VLM-based VQA modules assess binary presence/absence of each semantic element per sketch.
3. Recognizability inference ($P$): Zero-shot classifiers provide the posterior for the ground-truth class.

These signals are combined by:

$$SEA = \tanh(\alpha Z) \quad \text{where} \quad Z = \text{reward}(P, v) - \text{penalty}(P, v)$$

with $v = V/E$. The reward term encourages sketches maximizing $P$ with minimal $v$, gated to amplify reward when recognizability exceeds visual elaboration. The penalty term suppresses the SEA score either when excess detail is present given low recognition probability or when recognizability collapses altogether. This results in a continuous, bounded, and differentiable metric in $(E)$0, with interpretable failure and efficiency regions.

Figure 3: The SEA computation pipeline with case-based analysis highlighting abstraction failures, overdrawn cases, and efficient abstraction regimes.

Theoretical Properties and Design Choices

SEA is strictly monotonic with respect to class recognizability, saturates for unrecognizable sketches regardless of details, and penalizes over-drawn cases—thereby operationalizing a computational definition of abstraction efficiency grounded in visual-semantic economy. The metric enforces invariance to the absolute number of elements and admits robust tuning via a compact controllable hyperparameter space.

Construction and Properties of the CommonSketch Dataset

CommonSketch is constructed to support reference-free, element-aware abstraction evaluation. Each class's representative element set is derived from LLMs, externally validated, and distilled via annotator consensus to ensure strict visual decorum (presence only if visually explicit in the strokes). The dataset includes category-level class taxonomy aligning with THINGS and TU-Berlin, with comprehensive coverage spanning animate and inanimate objects, artifacts, and pictograms, supporting broad generalization studies.

Figure 2: CommonSketch construction pipeline and category/class coverage, visualizing data flow and representative statistics.

Cross-dataset analysis reveals CommonSketch to have the highest concentration of high-recognition-probability samples compared to TU-Berlin and QuickDraw, confirming its fidelity for both classification and abstraction tasks.

Empirical Evaluation and Results

SEA Behavior Across Abstraction Levels

Evaluation on SEVA sketches (with controlled abstraction via drawing time constraints) demonstrates monotonic increases in SEA scores with greater abstraction (as operationalized through increasing allowed drawing time per sketch)—provided recognizability is not sacrificed. This decouples mere stroke count or element quantity from genuine abstraction, and SEA robustly penalizes either incomplete abstraction or over-machined detail.

Figure 4: Qualitative SEA comparison for sketches at multiple abstraction levels (4, 8, 16, 32) for four object classes, with corresponding variation in visual ratio and prediction probability.

Figure 5: Distribution of SEA scores for SEVA sketches, with abstraction-induced translation from low to high-SEA regions.

Robustness and Component Analysis

A modular pipeline ablation—replacing LLMs or VLMs with open-weight alternatives (GPT-OSS, Qwen2.5-VL, etc.)—shows close concordance in both mean/variance and ranking across classes on human annotation, with inter-pipeline consistency exceeding 0.8 in concordance statistics. Quantitative comparisons (Table 1 in the paper) affirm that Qwen2.5-VL exhibits the highest open-source element-level VQA alignment.

Human Alignment

SEA's alignment with aggregated human abstraction scores is evaluated via controlled user studies (37 and 27 participants) across abstraction bins and pairwise comparison orderings, with agreement rates exceeding 87%. Notably, both closed and open source pipelines achieve nearly identical alignment, substantiating the metric's validity beyond specific backbone dependencies.

(Figure 6)

Figure 6: Alignment between SEA metric bins and human-annotated abstraction quality bins.

Implications, Limitations, and Future Directions

The introduction of SEA and CommonSketch establishes the first systematic, reference-free, and element-level abstraction efficiency protocol for visual concept analysis in freehand sketches. This framework exposes strong biases and limitations in both dataset design (e.g., label-centric QuickDraw) and in the abstraction capacity of current VLMs, opening the door for explicit abstraction-aware training and model selection pipelines.

Practically, the framework has direct implications for abstraction-driven sketch generation (diffusion models, stroke-based vectorization), explainable VQA for non-photorealistic images, and as a reward signal in generative RL or adversarial selection.

Theoretically, SEA operationalizes a differentiable, interpretable abstraction efficiency function, bridging semantic recognition and minimal sufficient visual evidence—a step toward formalizing the abstraction process in computational visual cognition.

Limiting factors include the dataset's current focus on single-object sketches, language/culture coverage in commonsense element extraction, and SEA's reliance on the VQA/classification stack. Targeted generalization to symbolic, compositional, and highly schematic domains is needed, as well as further relaxation of dependence on closed-source LLMs/VLMs.

Conclusion

"SEA: Evaluating Sketch Abstraction Efficiency via Element-level Commonsense Visual Question Answering" provides the first formal, reference-free metric and accompanying element-level benchmark connecting semantic abstraction and recognizability in freehand sketches. SEA advances both the analysis and generation of abstract visuals, catalyzing the study of abstraction in vision-language systems. The authors' public release of CommonSketch and all pipeline components sets a new standard for element-level abstraction research in AI and cognitive vision.