SketchyCOCO: Sketch-Conditioned Vision Benchmark

• SketchyCOCO is a composite benchmark that pairs COCO-derived natural images with synthetic and freehand sketches to study image generation and retrieval tasks.
• It employs rich, multi-level supervision—including paired scene sketch/image examples, foreground triplets, background pairs, and segmentation—to capture object and scene details.
• Evaluations using EdgeGAN, AODA, and retrieval methods demonstrate its effectiveness in quantifying the domain gap between abstract sketches and realistic images.

SketchyCOCO is a composite benchmark for sketch-conditioned vision, introduced by Gao et al. to support image generation from scene-level sketches and later adopted for open-domain sketch-to-photo synthesis and scene-level sketch-based image retrieval. It couples COCO-derived natural imagery with sketches drawn from public sketch corpora, and supplements paired scene sketch/image examples with foreground triplets, background pairs, and scene-sketch segmentation. Because it has been used under both paired generation and unpaired cross-modal protocols, it has become a recurrent testbed for the domain gap between line drawings and natural images (Gao et al., 2020, Xiang et al., 2021, Demić et al., 8 Sep 2025).

1. Origin and conceptual status

SketchyCOCO originated in the work that introduced EdgeGAN for image generation from scene-level sketches. In that formulation, the benchmark was built as a large-scale composite dataset from COCO-Stuff and public sketch sources, with the explicit goal of enabling both object-level and scene-level image generation from sketch inputs. The original framing emphasizes controllability: sketches specify object location, pose, and shape, while permitting rough or omitted background structure (Gao et al., 2020).

Later work in scene-level SBIR characterizes SketchyCOCO somewhat differently. It describes the dataset as originating from Gao et al. (CVPR 2020), pairing sketches with MS-COCO images, but also explicitly notes that the scene sketches are synthetic rather than freehand and “exhibit lower visual quality and realism compared to those in FS-COCO.” This has become important in evaluation: SketchyCOCO is often treated as a convenient, widely used scene-level benchmark, but not as a full proxy for human-drawn scene ambiguity (Demić et al., 8 Sep 2025).

This suggests that SketchyCOCO occupies a hybrid position. The constituent sketch elements are drawn from freehand sketch corpora, yet the scene-level benchmark instances are composited and algorithmically organized. That hybrid status explains why the dataset has supported both generation papers that emphasize sketch controllability and retrieval papers that emphasize evaluation caveats around realism and ambiguity.

2. Construction, scale, and supervision structure

The original dataset construction combines 14 foreground classes and 3 background classes. Instance sketches were collected from Sketchy, TU-Berlin, and QuickDraw, with around 700 sketches per foreground class and an 80\%/20\% train/test split. Natural images were drawn from 14,081 COCO-Stuff images containing at least one of 17 categories, split into 11,265 training images and 2,816 test images. Background instance sketches for clouds, grass, and trees were placed at random positions within corresponding background regions, yielding 27,683 background sketch–image pairs, while foreground sketch–image–edge-map triplets were created by retrieving the most similar sketch of the same class for each foreground object (Gao et al., 2020).

Component	Training split	Test split
Scene sketch–image pairs	11,265	2,816
Foreground triplets	18,869	1,329
Background sketch–image pairs	22,171	5,512

The supervision is richer than a single paired scene tuple. The dataset includes five-tuple ground truth covering {foreground image, foreground sketch, foreground edge maps}, {background image, background sketch}, {scene image, foreground image + background sketch}, {scene image, scene sketch}, and sketch segmentation for scene sketches. Instance segmentation in scene sketches is represented as tuples $(C, B, M)$, where $C$ is category, $M$ is a binary mask, and $B = (x_1, y_1, x_2, y_2)$ is the bounding box.

Resolution is task-dependent. All object-level entities are scaled to $64 \times 64$ and $128 \times 128$, while all scene-level entities are scaled to $128 \times 128$ and $256 \times 256$. This multi-level organization is a defining design choice: SketchyCOCO is not merely a flat scene-sketch benchmark, but a layered resource spanning object instances, backgrounds, and full scenes.

3. Original task formulation and EdgeGAN pipeline

The original paper defines two tasks. In object-level generation, a single object sketch is used to generate a realistic object image matching the sketch’s category, pose, and shape. In scene-level generation, a scene sketch containing multiple object instances and background strokes conditions a full image. The method addresses the scene problem through a two-stage decomposition: foreground generation must exactly satisfy the specified objects, while background generation should be loosely aligned yet consistent with the synthesized foreground.

EdgeGAN is the central object-level model. It introduces an attribute-vector bridge between edge maps and images so that freehand sketches can be used at inference even though object-level training uses extracted edge maps rather than freehand sketches. The architecture contains generators $G_E$ and $G_I$, discriminators $C$0, $C$1, and a joint discriminator $C$2, together with an encoder $C$3 and a classifier $C$4. The encoder maps an edge map, or a freehand sketch at inference time, to an attribute vector intended to remain close to the shared latent variable $C$5, enforced by

$C$6

For scene synthesis, the pipeline first segments a scene sketch into instances $C$7, generates each foreground object via the object model, composites the outputs, and then applies a pix2pix background generator to {foreground image + background sketch} \rightarrow \text{scene image}.

Sketch representation is deliberately heterogeneous. The paper uses freehand sketches and several styles of synthesized edge maps—XDoG, FDoG, Photo-Sketching, and Canny—to reduce overfitting to a single line-drawing style. In the reported ablations, the joint discriminator $C$8 is important for pose and shape correctness, and WGAN-GP is more stable than DCGAN or non-GP WGAN. Quantitatively, the full model reports object-level FID $C$9, Accuracy $M$0, and SS $M$1 $M$2; for scene-level synthesis it reports FID $M$3, SSIM $M$4, FID(local) $M$5, Realism $M$6, and Faithfulness $M$7 (Gao et al., 2020).

4. Open-domain sketch-to-photo synthesis benchmark

In later work on open-domain sketch-to-photo translation, SketchyCOCO is repurposed as a benchmark for generalization beyond classes with sketch supervision. In this setting, the benchmark is defined over 14 object categories. Sketches are sourced from Sketchy, TU-Berlin, and Quick, Draw!, and photos are segmented objects from COCO-Stuff natural scenes “under unconstrained conditions.” The supervision is unpaired: sketches and photos are collected separately and are not aligned per instance. The key experimental split removes the sketches of two categories—sheep and giraffe—entirely during training, making them open-domain classes, while photos remain available for all 14 categories (Xiang et al., 2021).

The AODA framework jointly learns a photo-to-sketch generator $M$8 and a multi-class sketch-to-photo generator $M$9, together with PatchGAN discriminators $B = (x_1, y_1, x_2, y_2)$0 and $B = (x_1, y_1, x_2, y_2)$1 and an HRNet-based photo classifier $B = (x_1, y_1, x_2, y_2)$2. The total generator objective is

$B = (x_1, y_1, x_2, y_2)$3

Here $B = (x_1, y_1, x_2, y_2)$4 follows a Johnson-style architecture with Instance Normalization, while $B = (x_1, y_1, x_2, y_2)$5 uses AdaIN in residual blocks and PixelShuffle for upsampling. The operational problem is that, once sketches of open-domain classes are absent, $B = (x_1, y_1, x_2, y_2)$6 loses ordinary adversarial supervision for those classes and is driven mainly by pixel regression.

To address that degeneration, the paper introduces a random-mixed sampling strategy. For each iteration, it computes pseudo sketches $B = (x_1, y_1, x_2, y_2)$7, maintains a buffer of the 50 most recent pseudo sketch-label minibatches, and with probability controlled by threshold $B = (x_1, y_1, x_2, y_2)$8 replaces the real sketch minibatch seen by $B = (x_1, y_1, x_2, y_2)$9 with pooled pseudo sketches. The crucial asymmetry is that the mixing is only applied to $64 \times 64$0: $64 \times 64$1, $64 \times 64$2, and $64 \times 64$3 still receive correctly labeled real-vs-fake data. The intended effect is to “fool” $64 \times 64$4 into treating synthesized sketches as real ones, thereby restoring adversarial and class-conditional pressure for missing classes.

On SketchyCOCO, AODA reports the best FID among the compared methods across the full, in-domain, and open-domain splits: $64 \times 64$5, $64 \times 64$6, and $64 \times 64$7, respectively. It also reports classifier accuracy $64 \times 64$8 on the full set, $64 \times 64$9 on in-domain sketches, and $128 \times 128$0 on open-domain sketches, together with human preference scores of $128 \times 128$1 overall, $128 \times 128$2 in-domain, and $128 \times 128$3 open-domain. Training uses $128 \times 128$4 inputs, Adam, batch size $128 \times 128$5, 100 epochs, an initial learning rate of $128 \times 128$6, and PyTorch on a single NVIDIA V100 GPU (Xiang et al., 2021).

5. Scene-level sketch-based image retrieval

SketchyCOCO has also become a scene-level SBIR benchmark. In that literature, the most common protocol is not the full 14,081-pair dataset but the SceneSketcher subset, which retains only sketches depicting more than a single foreground object. This refinement yields 1,225 sketch–photo pairs, with 1,015 for training and 210 for testing, and is used to evaluate fine-grained retrieval rather than generative fidelity (Demić et al., 8 Sep 2025).

A recent retrieval formulation uses a Siamese, shared-encoder architecture in which a single ConvNeXt-Base visual backbone from OpenCLIP encodes both sketches and photos into a joint embedding space. Cosine similarity is defined as

$128 \times 128$7

and training minimizes the KL divergence between a softmax-normalized similarity distribution over the batch and a debiased supervisory distribution that mixes the ground-truth positive with a uniform mass over the gallery. This replaces strict one-positive, all-negative supervision with a softened target designed to tolerate accidental semantic matches among nominal negatives.

Evaluation follows the standard fine-grained SBIR protocol with Recall@$128 \times 128$8, reported for $128 \times 128$9, where

$128 \times 128$0

On the SceneSketcher subset, the method reports $128 \times 128$1, $128 \times 128$2, and $128 \times 128$3, exceeding the reported SceneSketcherV2 scores of $128 \times 128$4, $128 \times 128$5, and $128 \times 128$6. The implementation uses PyTorch, the OpenCLIP convnext_base visual encoder pretrained on laion400m_s13b_b51k, Adam with learning rate $128 \times 128$7 and weight decay $128 \times 128$8, AMP, batch size $128 \times 128$9, 10 epochs, and a single NVIDIA A100 40GB GPU. No sketch-specific preprocessing or augmentation is reported, and the method does not rely on external annotations, auxiliary detectors, graph heads, or segmentation heads (Demić et al., 8 Sep 2025).

6. Limitations, ambiguities, and research significance

Several limitations recur across uses of SketchyCOCO. In the original generation paper, scene sketch segmentation may fail on very abstract sketches; foreground objects from COCO-Stuff are biased toward dominant side-view poses; background generation must hallucinate large unconstrained blank regions; and end-to-end scene-sketch-to-image baselines such as pix2pix and SketchyGAN are prone to mode collapse at $256 \times 256$0. The paper also notes that color is not explicitly conditioned, and that line complexity can bias the darkness of the generated image (Gao et al., 2020).

Open-domain synthesis exposes a different set of stresses. AODA reports that, as the number of missing classes increases, output quality degrades; complex scenes and severe sketch distortions remain challenging; and very fine-grained structures or substantial abstraction may still produce texture artifacts or imperfect object/background separation. These observations are consistent with the benchmark’s original motivation: SketchyCOCO forces models to bridge substantial geometric distortion and abstraction between sketches and natural images (Xiang et al., 2021).

Retrieval work raises methodological concerns about evaluation itself. Because SketchyCOCO’s scene sketches are synthetic and less realistic than FS-COCO, they may under-represent real-world ambiguity and drawing-style variability. Moreover, evaluation is typically strict Recall@$256 \times 256$1 with a single ground-truth image per sketch, even though multiple images may be semantically reasonable matches. The retrieval literature therefore argues that current scene-level SBIR evaluation should better account for sketch ambiguity, human agreement, and scalability, while still recognizing SketchyCOCO’s utility as a standardized benchmark (Demić et al., 8 Sep 2025).

Taken together, these lines of work show that SketchyCOCO is best understood not as a single fixed task definition but as a benchmark family. It supports paired scene synthesis, object-centric controllable generation, unpaired open-domain translation, and scene-level retrieval. A plausible implication is that its lasting value lies less in any one leaderboard than in its ability to expose how different models handle the same underlying mismatch: sparse, abstract, line-based structure on one side and full-color, textured, cluttered natural imagery on the other.