---
title: 'FS-COCO: Freehand Sketches in Context'
url: https://www.emergentmind.com/topics/fs-coco
type: topic
---

# FS-COCO: Freehand Sketches in Context

FS-COCO, short for **“Freehand Sketches of Common Objects in Context,”** is a dataset and benchmark introduced to study scene understanding from genuinely freehand sketches rather than from isolated object drawings or semi-synthetic scene compositions. It comprises **10,000** freehand scene **vector sketches** with per-point space-time information, each paired with a reference **MS-COCO** photo and a textual sketch description written by the sketcher. The resource was designed to support fine-grained image retrieval from scene sketches and captions, multimodal sketch-text retrieval, and scene-sketch captioning, while also exposing how people abstract complex scenes when drawing quickly from memory [2203.02113].

## 1. Origins and problem setting

FS-COCO was introduced to address two limitations in prior sketch research. First, most freehand sketch datasets were **object-centric**, emphasizing isolated objects rather than contextual scenes. Second, existing scene-sketch datasets were largely **semi-synthetic**, typically formed by composing object sketches or clip-art-like elements into scenes. FS-COCO instead targets **holistic freehand scene sketching**, where a person views a photograph, internalizes its content, and then reconstructs the scene from memory.

The dataset is grounded in **10,000 randomly selected photos from MS-COCO**, which gives it direct alignment with a standard scene-understanding benchmark. Because each COCO source image already provides **five human-written image captions**, every FS-COCO sample forms a multimodal tuple consisting of a reference photo, a freehand scene sketch, a sketch caption, and the inherited COCO captions. This design supports direct comparison between sketch-based and text-based access to scene semantics.

The motivating use case is practical rather than purely taxonomic. The paper argues that when a person wants to retrieve an image or a video frame from memory, it may be easier to draw a quick scene than to verbalize all relevant visual details. FS-COCO therefore treats sketching as a compact, human-centered interface for scene retrieval rather than only as an artistic medium.

## 2. Collection protocol and sketch representation

FS-COCO was collected from **100 non-artist participants** aged **22–44**, with average age **27.03**, including **72 male** and **28 female** annotators. Each participant drew sketches for a **distinct subset of 100 photos**, producing **10,000 unique sketch-photo pairs** overall. The collection procedure was explicitly memory-based: each participant viewed a source image for **60 seconds**, then sketched the scene on a blank canvas within **3 minutes**.

These timings were chosen after a pilot study with **10 participants**. The paper reports that **30 seconds viewing + 60 seconds sketching** was too difficult and often produced unrecognizable sketches; **30 seconds + 120 seconds** still missed important content; and **60 seconds + 180 seconds** gave the best tradeoff between recognizability and speed. Participants could repeat the full process if unsatisfied, with an average of **1.7 attempts**. After completing a sketch, the **same participant** wrote a sketch caption, so the textual description reflected the sketcher’s own intended semantics.

A dedicated human judge reviewed sketch quality and flagged sketches judged “too difficult to understand or recognize,” after which the original annotator revised them. This introduced a recognizability constraint without requiring artistic expertise.

The sketch representation is vectorial and temporally ordered. Each point is encoded as

$$
P_t = (x_t, y_t, q^1_t, q^2_t, q^3_t),
$$

where \((x_t, y_t)\) are absolute pen coordinates and \((q^1_t, q^2_t, q^3_t)\) is a one-hot pen-state vector for pen down, pen up / end of stroke, and end of sketch. Because FS-COCO preserves pointwise temporal order, it supports analysis of **stroke sequencing**, **scene salience**, and **coarse-to-fine construction** in a way that raster-only scene sketch datasets cannot.

| Property | Value | Notes |
|---|---:|---|
| Reference photos | 10,000 | Randomly selected from MS-COCO |
| Sketches | 10,000 | Freehand vector scene sketches |
| Annotators | 100 | Non-expert participants |
| Sketch duration | 3 minutes | After 60-second image viewing |
| Sketch captions | 10,000 | Written by the same sketcher |
| Image captions | 5 per photo | Inherited from MS-COCO |

FS-COCO sketches are substantially more complex than standard object-sketch datasets. The paper reports a **median stroke count of 64**, an **average of 74.3 strokes**, and **41.1 points per stroke** on average. Original scene sketches can exceed **3000 stroke points**; one analysis cites an average of **2437** total points, reduced to **165** after Ramer-Douglas-Peucker simplification.

## 3. Dataset statistics and relation to prior scene-sketch corpora

FS-COCO was designed to capture both **object-level abstraction** and **scene-level abstraction**. This distinction is central: in freehand scene drawing, objects may share strokes, participants may alternate across scene elements during drawing, and many image regions may be omitted or simplified. The dataset therefore reflects selective, memory-driven scene construction rather than a compositional assembly of pre-existing object icons.

The paper gives two complementary estimates of category coverage. A **caption-based estimate** \(e_c\), derived from sketch-caption vocabulary, yields **92 categories**, with **99.42** sketches per category on average, **1.37** categories per sketch on average, and **1 / 5** as the minimum/maximum category count per sketch. A **label-based estimate** \(e_l\), derived from semantic segmentation labels in source images, yields **150 categories**, with **413.18** sketches per category on average, **7.17** categories per sketch on average, and **1 / 25** as the minimum/maximum category count per sketch. The paper explicitly treats the first as a lower estimate and the second as an upper estimate: sketch captions do not exhaustively describe the drawing, while source-image segmentation may include objects or regions that were never sketched.

The reported indoor/outdoor coverage is likewise broad: **33 / 71 indoor categories** and **59 / 79 outdoor categories** under the lower/upper estimates. Frequent categories include natural scene elements such as `tree`, `grass`, and `clouds`, human and animal categories such as `person`, `horse`, `giraffe`, `dog`, `cow`, and `sheep`, and man-made scene elements such as `road`, `car`, `bench`, `clock`, `train`, `house`, and `kite`.

Relative to earlier scene-sketch datasets, FS-COCO is distinctive less because of absolute pair count than because of its **freehand** and **temporal** character.

| Dataset | Scale | Reported properties |
|---|---|---|
| SketchyScene | 7,264 pairs, 45 categories | Object abstraction yes; scene abstraction no; not freehand; no temporal order; no captions |
| SketchyCOCO | 14,081 pairs, 17 categories | Synthetic scene assembly; not freehand; no temporal order; no captions |
| FS-COCO | 10,000 pairs, 92 / 150 categories | Freehand; stroke temporal order; captions; both object- and scene-level abstraction |

A structural limitation is that FS-COCO does **not** include stroke-level or object-level annotations. The paper states that these were deliberately omitted to avoid interrupting natural sketching behavior.

## 4. Benchmark tasks and hierarchical sketch modeling

FS-COCO defines two principal benchmark families. The first is **fine-grained image retrieval from freehand scene sketches**, extended to sketch captions and sketch-text fusion. The second is **scene sketch captioning**. For the main retrieval experiments, the paper uses **70% of each user’s sketches for training** and **30% for testing**, producing **7,000 training pairs** and a **3,000-image test gallery**. To compare fairly against smaller galleries used by prior scene-sketch datasets, the paper also reports results on **10 random subsets of 210 pairs**. A supplementary user-generalization setting trains on **70 users** and tests on **30 unseen users**.

Retrieval is evaluated with **R@1** and **R@10**. Scene-sketch captioning is evaluated with **BLEU-4**, **METEOR**, **ROUGE**, **CIDEr**, and **SPICE**. Retrieval baselines include Siamese VGG16 variants, HOLEF/HOLEF-VGG16, zero-shot CLIP, and a lightly fine-tuned **CLIP\*** variant in which only layer-normalization modules are updated and a fully connected projection maps features into a shared **512D** space. Retrieval training uses a **triplet loss** with margin **0.2**.

The paper’s main architectural contribution is a **hierarchical vector sketch decoder** (“H-Decoder”), introduced because prior single-RNN sketch decoders were suitable for object sketches of roughly up to **200 points** but not for scene sketches with thousands of points. A raster sketch encoder \(E(\cdot)\) produces a feature map

$$
F \in \mathbb{R}^{h' \times w' \times c},
$$

which is globally pooled into a latent representation

$$
l_R \in \mathbb{R}^{512}.
$$

A global stroke-level LSTM then predicts stroke embeddings:

$$
h_0^G = W_h^G l_R + b_h^G,
$$

$$
h_i^G = \mathrm{RNN}_G\big(h_{i-1}^G, [l_R, S_{i-1}]\big),
$$

$$
S_i = W_y^G h_i^G + b_y^G.
$$

For each predicted stroke representation \(S_i\), a local point-level LSTM generates the stroke point sequence:

$$
h_0^L = W_h^L S_i + b_h^L,
$$

$$
h_j^L = \mathrm{RNN}_L\big(h_{j-1}^L, [S_i, P_{t-1}]\big),
$$

$$
P_t = W_y^L h_j^L + b_j^L.
$$

The decoder predicts absolute coordinates and pen-state outputs, supervised by **mean squared error** on coordinates and **categorical cross-entropy** on pen state. This hierarchical decomposition matches the natural structure

$$
\text{scene sketch} \rightarrow \text{strokes} \rightarrow \text{points},
$$

and serves as a sketch-specific pretext task for downstream retrieval and captioning [2203.02113].

## 5. Empirical findings

The most immediate empirical result is a large **domain gap** between freehand scene sketches and earlier semi-synthetic scene-sketch datasets. On FS-COCO retrieval, a Siamese VGG16 model trained and tested on FS-COCO achieves **R@1 = 23.3** and **R@10 = 52.6**. The same architecture trained on **SketchyScene** and tested on FS-COCO drops to **1.8 / 6.6**, while training on **SketchyCOCO** yields **R@1 < 0.1** and **R@10 = 0.4**. On matched **210-image** test subsets, Siam.-VGG16 reaches **\(50.39\% \pm 2.15\%\)** top-1 and **\(89.38\% \pm 2.0\%\)** top-10, while CLIP\* reaches **\(42.53\% \pm 3.16\%\)** and **\(87.93\% \pm 2.14\%\)**, indicating that the main degradation on the full benchmark is not only due to data quality but also to gallery size and problem difficulty [2203.02113].

The dataset also reveals a structural regularity in how people draw scenes. Average stroke length decreases over time, which the paper interprets as a **coarse-to-fine** drawing process. A masking experiment further shows that removing **early-drawn strokes** harms retrieval much more than removing late strokes, indicating that temporal order carries a salience signal: participants tend to sketch the most retrieval-relevant scene content first.

A second central result concerns the relative informativeness of sketches and captions. For **image-caption retrieval**, the paper reports **11.1 / 31.1** for CNN-RNN, **21.0 / 50.9** for zero-shot CLIP, and **22.1 / 52.3** for CLIP\*. For **sketch-caption retrieval**, the corresponding results are **7.2 / 23.6**, **11.5 / 35.3**, and **14.8 / 36.6**. For **scene-sketch retrieval**, Siam.-VGG16 reaches **23.3 / 52.6**, while zero-shot CLIP on sketches is only **1.17 / 6.07** and CLIP\* is **5.5 / 26.5**. The striking comparison is that freehand scene sketches with Siam.-VGG16 slightly outperform fine-tuned CLIP\* retrieval from image captions on the main FS-COCO test set.

The sketch-text relation is not redundant. Sketch-caption vocabulary matches **81.50%** of image-caption vocabulary, and **66.5%** of words in sketch captions overlap with image captions for the same instance, compared with **60.8%** overlap among the five COCO image captions themselves. At the same time, sketch captions are shorter—**6.55** words on average versus **10.46** for image captions—and emphasize nouns and verbs more than adjectives, omitting many color terms. This supports the paper’s claim that sketches preserve structural and fine-grained visual cues, whereas text contributes attributes such as color.

Fusion confirms that the modalities are complementary. Combining scene sketches with image captions yields **25.3 / 55.0** for CNN-RNN with additive fusion, **24.3 / 53.9** for CNN-RNN concatenation, **23.9 / 53.5** for CLIP\* additive fusion, and **23.3 / 52.6** for CLIP\* concatenation. The best fused result therefore exceeds either sketch-only or caption-only retrieval.

For scene-sketch captioning, the paper reports **13.7 / 17.1 / 44.9 / 69.4 / 14.5** for the Xu et al. baseline, **16.0 / 18.9 / 49.1 / 80.5 / 15.8** for AG-CVAE, **16.7 / 21.0 / 52.9 / 90.1 / 16.0** for LNFMM, and **17.3 / 21.1 / 53.2 / 95.3 / 17.2** for **LNFMM with H-Decoder pretraining**, demonstrating that the hierarchical sketch pretext task improves not only retrieval but also caption generation.

## 6. Terminology, limitations, and significance

Within sketch research, **FS-COCO** refers specifically to **Freehand Sketches of Common Objects in Context**. A recurring source of confusion is that, in parts of the few-shot semantic segmentation literature, **COCO-\(20^i\)** is often called **“FS-COCO” in practice**, even though it denotes a different benchmark family centered on few-shot segmentation rather than scene-sketch understanding [2411.11917]. The sketch dataset and the few-shot segmentation shorthand are therefore terminologically adjacent but conceptually distinct.

The dataset’s limitations are explicit. Category counts are approximate because caption-derived labels undercount sketched content while image-derived semantic labels overcount available scene content. The resource also lacks object-level and stroke-level labels, which constrains supervised parsing of sketch structure. User-style generalization remains nontrivial: the supplementary protocol that trains on **70 users** and tests on **30 unseen users** frames personalization and cross-user transfer as open problems rather than solved ones. The paper also shows that zero-shot CLIP transfers poorly to freehand scene sketches, implying a substantial modality gap between large-scale photo-text pretraining and human scene sketching.

Even with those constraints, FS-COCO establishes a new empirical regime. It is the first dataset in the paper’s framing to combine **freehand** scene sketches, **vector stroke sequences**, **temporal order**, and **paired captions** at scale. This enables research not only on scene-sketch retrieval and captioning, but also on human abstraction, salience encoded by drawing order, sketch-text complementarity, and sketch-specific pretraining for complex sequential visual representations. A plausible implication is that FS-COCO functions as both a benchmark and a measurement instrument for how people compress scene semantics into rapid line drawings.

Source: https://www.emergentmind.com/topics/fs-coco