FS-COCO: Freehand Sketches in Context

• FS-COCO is a dataset of 10,000 freehand vector scene sketches paired with MS-COCO images and captions, emphasizing holistic scene construction from memory.
• It supports fine-grained image and sketch retrieval as well as scene-sketch captioning through hierarchical vector sketch decoding and detailed stroke sequencing.
• Empirical findings reveal a notable domain gap with semi-synthetic datasets and highlight the importance of drawing order in capturing scene salience.

FS-COCO, short for “Freehand Sketches of Common Objects in Context,” is a dataset and benchmark introduced to study scene understanding from genuinely freehand sketches rather than from isolated object drawings or semi-synthetic scene compositions. It comprises 10,000 freehand scene vector sketches with per-point space-time information, each paired with a reference MS-COCO photo and a textual sketch description written by the sketcher. The resource was designed to support fine-grained image retrieval from scene sketches and captions, multimodal sketch-text retrieval, and scene-sketch captioning, while also exposing how people abstract complex scenes when drawing quickly from memory (Chowdhury et al., 2022).

1. Origins and problem setting

FS-COCO was introduced to address two limitations in prior sketch research. First, most freehand sketch datasets were object-centric, emphasizing isolated objects rather than contextual scenes. Second, existing scene-sketch datasets were largely semi-synthetic, typically formed by composing object sketches or clip-art-like elements into scenes. FS-COCO instead targets holistic freehand scene sketching, where a person views a photograph, internalizes its content, and then reconstructs the scene from memory.

The dataset is grounded in 10,000 randomly selected photos from MS-COCO, which gives it direct alignment with a standard scene-understanding benchmark. Because each COCO source image already provides five human-written image captions, every FS-COCO sample forms a multimodal tuple consisting of a reference photo, a freehand scene sketch, a sketch caption, and the inherited COCO captions. This design supports direct comparison between sketch-based and text-based access to scene semantics.

The motivating use case is practical rather than purely taxonomic. The paper argues that when a person wants to retrieve an image or a video frame from memory, it may be easier to draw a quick scene than to verbalize all relevant visual details. FS-COCO therefore treats sketching as a compact, human-centered interface for scene retrieval rather than only as an artistic medium.

2. Collection protocol and sketch representation

FS-COCO was collected from 100 non-artist participants aged 22–44, with average age 27.03, including 72 male and 28 female annotators. Each participant drew sketches for a distinct subset of 100 photos, producing 10,000 unique sketch-photo pairs overall. The collection procedure was explicitly memory-based: each participant viewed a source image for 60 seconds, then sketched the scene on a blank canvas within 3 minutes.

These timings were chosen after a pilot study with 10 participants. The paper reports that 30 seconds viewing + 60 seconds sketching was too difficult and often produced unrecognizable sketches; 30 seconds + 120 seconds still missed important content; and 60 seconds + 180 seconds gave the best tradeoff between recognizability and speed. Participants could repeat the full process if unsatisfied, with an average of 1.7 attempts. After completing a sketch, the same participant wrote a sketch caption, so the textual description reflected the sketcher’s own intended semantics.

A dedicated human judge reviewed sketch quality and flagged sketches judged “too difficult to understand or recognize,” after which the original annotator revised them. This introduced a recognizability constraint without requiring artistic expertise.

The sketch representation is vectorial and temporally ordered. Each point is encoded as

$P_t = (x_t, y_t, q^1_t, q^2_t, q^3_t),$

where $(x_t, y_t)$ are absolute pen coordinates and $(q^1_t, q^2_t, q^3_t)$ is a one-hot pen-state vector for pen down, pen up / end of stroke, and end of sketch. Because FS-COCO preserves pointwise temporal order, it supports analysis of stroke sequencing, scene salience, and coarse-to-fine construction in a way that raster-only scene sketch datasets cannot.

Property	Value	Notes
Reference photos	10,000	Randomly selected from MS-COCO
Sketches	10,000	Freehand vector scene sketches
Annotators	100	Non-expert participants
Sketch duration	3 minutes	After 60-second image viewing
Sketch captions	10,000	Written by the same sketcher
Image captions	5 per photo	Inherited from MS-COCO

FS-COCO sketches are substantially more complex than standard object-sketch datasets. The paper reports a median stroke count of 64, an average of 74.3 strokes, and 41.1 points per stroke on average. Original scene sketches can exceed 3000 stroke points; one analysis cites an average of 2437 total points, reduced to 165 after Ramer-Douglas-Peucker simplification.

3. Dataset statistics and relation to prior scene-sketch corpora

FS-COCO was designed to capture both object-level abstraction and scene-level abstraction. This distinction is central: in freehand scene drawing, objects may share strokes, participants may alternate across scene elements during drawing, and many image regions may be omitted or simplified. The dataset therefore reflects selective, memory-driven scene construction rather than a compositional assembly of pre-existing object icons.

The paper gives two complementary estimates of category coverage. A caption-based estimate $e_c$, derived from sketch-caption vocabulary, yields 92 categories, with 99.42 sketches per category on average, 1.37 categories per sketch on average, and 1 / 5 as the minimum/maximum category count per sketch. A label-based estimate $e_l$, derived from semantic segmentation labels in source images, yields 150 categories, with 413.18 sketches per category on average, 7.17 categories per sketch on average, and 1 / 25 as the minimum/maximum category count per sketch. The paper explicitly treats the first as a lower estimate and the second as an upper estimate: sketch captions do not exhaustively describe the drawing, while source-image segmentation may include objects or regions that were never sketched.

The reported indoor/outdoor coverage is likewise broad: 33 / 71 indoor categories and 59 / 79 outdoor categories under the lower/upper estimates. Frequent categories include natural scene elements such as tree, grass, and clouds, human and animal categories such as person, horse, giraffe, dog, cow, and sheep, and man-made scene elements such as road, car, bench, clock, train, house, and kite.

Relative to earlier scene-sketch datasets, FS-COCO is distinctive less because of absolute pair count than because of its freehand and temporal character.

Dataset	Scale	Reported properties
SketchyScene	7,264 pairs, 45 categories	Object abstraction yes; scene abstraction no; not freehand; no temporal order; no captions
SketchyCOCO	14,081 pairs, 17 categories	Synthetic scene assembly; not freehand; no temporal order; no captions
FS-COCO	10,000 pairs, 92 / 150 categories	Freehand; stroke temporal order; captions; both object- and scene-level abstraction

A structural limitation is that FS-COCO does not include stroke-level or object-level annotations. The paper states that these were deliberately omitted to avoid interrupting natural sketching behavior.

4. Benchmark tasks and hierarchical sketch modeling

FS-COCO defines two principal benchmark families. The first is fine-grained image retrieval from freehand scene sketches, extended to sketch captions and sketch-text fusion. The second is scene sketch captioning. For the main retrieval experiments, the paper uses 70% of each user’s sketches for training and 30% for testing, producing 7,000 training pairs and a 3,000-image test gallery. To compare fairly against smaller galleries used by prior scene-sketch datasets, the paper also reports results on 10 random subsets of 210 pairs. A supplementary user-generalization setting trains on 70 users and tests on 30 unseen users.

Retrieval is evaluated with R@1 and R@10. Scene-sketch captioning is evaluated with BLEU-4, METEOR, ROUGE, CIDEr, and SPICE. Retrieval baselines include Siamese VGG16 variants, HOLEF/HOLEF-VGG16, zero-shot CLIP, and a lightly fine-tuned CLIP* variant in which only layer-normalization modules are updated and a fully connected projection maps features into a shared 512D space. Retrieval training uses a triplet loss with margin 0.2.

The paper’s main architectural contribution is a hierarchical vector sketch decoder (“H-Decoder”), introduced because prior single-RNN sketch decoders were suitable for object sketches of roughly up to 200 points but not for scene sketches with thousands of points. A raster sketch encoder $E(\cdot)$ produces a feature map

$F \in \mathbb{R}^{h' \times w' \times c},$

which is globally pooled into a latent representation

$l_R \in \mathbb{R}^{512}.$

A global stroke-level LSTM then predicts stroke embeddings:

$h_0^G = W_h^G l_R + b_h^G,$

$$h_i^G = \mathrm{RNN}_G\big(h_{i-1}^G, [l_R, S_{i-1}]\big),$$

$(x_t, y_t)$0

For each predicted stroke representation $(x_t, y_t)$1, a local point-level LSTM generates the stroke point sequence:

$(x_t, y_t)$2

$(x_t, y_t)$3

$(x_t, y_t)$4

The decoder predicts absolute coordinates and pen-state outputs, supervised by mean squared error on coordinates and categorical cross-entropy on pen state. This hierarchical decomposition matches the natural structure

$(x_t, y_t)$5

and serves as a sketch-specific pretext task for downstream retrieval and captioning (Chowdhury et al., 2022).

5. Empirical findings

The most immediate empirical result is a large domain gap between freehand scene sketches and earlier semi-synthetic scene-sketch datasets. On FS-COCO retrieval, a Siamese VGG16 model trained and tested on FS-COCO achieves R@1 = 23.3 and R@10 = 52.6. The same architecture trained on SketchyScene and tested on FS-COCO drops to 1.8 / 6.6, while training on SketchyCOCO yields R@1 < 0.1 and R@10 = 0.4. On matched 210-image test subsets, Siam.-VGG16 reaches $(x_t, y_t)$6 top-1 and $(x_t, y_t)$7 top-10, while CLIP* reaches $(x_t, y_t)$8 and $(x_t, y_t)$9, indicating that the main degradation on the full benchmark is not only due to data quality but also to gallery size and problem difficulty (Chowdhury et al., 2022).

The dataset also reveals a structural regularity in how people draw scenes. Average stroke length decreases over time, which the paper interprets as a coarse-to-fine drawing process. A masking experiment further shows that removing early-drawn strokes harms retrieval much more than removing late strokes, indicating that temporal order carries a salience signal: participants tend to sketch the most retrieval-relevant scene content first.

A second central result concerns the relative informativeness of sketches and captions. For image-caption retrieval, the paper reports 11.1 / 31.1 for CNN-RNN, 21.0 / 50.9 for zero-shot CLIP, and 22.1 / 52.3 for CLIP*. For sketch-caption retrieval, the corresponding results are 7.2 / 23.6, 11.5 / 35.3, and 14.8 / 36.6. For scene-sketch retrieval, Siam.-VGG16 reaches 23.3 / 52.6, while zero-shot CLIP on sketches is only 1.17 / 6.07 and CLIP* is 5.5 / 26.5. The striking comparison is that freehand scene sketches with Siam.-VGG16 slightly outperform fine-tuned CLIP* retrieval from image captions on the main FS-COCO test set.

The sketch-text relation is not redundant. Sketch-caption vocabulary matches 81.50% of image-caption vocabulary, and 66.5% of words in sketch captions overlap with image captions for the same instance, compared with 60.8% overlap among the five COCO image captions themselves. At the same time, sketch captions are shorter—6.55 words on average versus 10.46 for image captions—and emphasize nouns and verbs more than adjectives, omitting many color terms. This supports the paper’s claim that sketches preserve structural and fine-grained visual cues, whereas text contributes attributes such as color.

Fusion confirms that the modalities are complementary. Combining scene sketches with image captions yields 25.3 / 55.0 for CNN-RNN with additive fusion, 24.3 / 53.9 for CNN-RNN concatenation, 23.9 / 53.5 for CLIP* additive fusion, and 23.3 / 52.6 for CLIP* concatenation. The best fused result therefore exceeds either sketch-only or caption-only retrieval.

For scene-sketch captioning, the paper reports 13.7 / 17.1 / 44.9 / 69.4 / 14.5 for the Xu et al. baseline, 16.0 / 18.9 / 49.1 / 80.5 / 15.8 for AG-CVAE, 16.7 / 21.0 / 52.9 / 90.1 / 16.0 for LNFMM, and 17.3 / 21.1 / 53.2 / 95.3 / 17.2 for LNFMM with H-Decoder pretraining, demonstrating that the hierarchical sketch pretext task improves not only retrieval but also caption generation.

6. Terminology, limitations, and significance

Within sketch research, FS-COCO refers specifically to Freehand Sketches of Common Objects in Context. A recurring source of confusion is that, in parts of the few-shot semantic segmentation literature, COCO-$(q^1_t, q^2_t, q^3_t)$0 is often called “FS-COCO” in practice, even though it denotes a different benchmark family centered on few-shot segmentation rather than scene-sketch understanding (Moon et al., 2024). The sketch dataset and the few-shot segmentation shorthand are therefore terminologically adjacent but conceptually distinct.

The dataset’s limitations are explicit. Category counts are approximate because caption-derived labels undercount sketched content while image-derived semantic labels overcount available scene content. The resource also lacks object-level and stroke-level labels, which constrains supervised parsing of sketch structure. User-style generalization remains nontrivial: the supplementary protocol that trains on 70 users and tests on 30 unseen users frames personalization and cross-user transfer as open problems rather than solved ones. The paper also shows that zero-shot CLIP transfers poorly to freehand scene sketches, implying a substantial modality gap between large-scale photo-text pretraining and human scene sketching.

Even with those constraints, FS-COCO establishes a new empirical regime. It is the first dataset in the paper’s framing to combine freehand scene sketches, vector stroke sequences, temporal order, and paired captions at scale. This enables research not only on scene-sketch retrieval and captioning, but also on human abstraction, salience encoded by drawing order, sketch-text complementarity, and sketch-specific pretraining for complex sequential visual representations. A plausible implication is that FS-COCO functions as both a benchmark and a measurement instrument for how people compress scene semantics into rapid line drawings.