KiloGram: Dataset for Abstract Visual Reasoning
- KiloGram is a dataset comprising 1,016 tangram puzzles with comprehensive annotations at both whole-shape and part levels.
- It features dual-axis design with geometric representations (SVG/PNG) and semantic cues (color vs black) to probe image–text alignment.
- Empirical evaluations show that fine-tuning with explicit part descriptions significantly boosts retrieval performance in multi-modal models.
Searching arXiv for the KiloGram dataset paper and closely related work on abstract visual reasoning with tangram shapes. KiloGram is a deeply annotated dataset for abstract visual reasoning built from tangram stimuli and designed to study how humans and multi-modal models coordinate language and vision under weakly constrained naming conventions. The resource contains over one thousand abstract tangram puzzles, pairs whole-shape descriptions with part-level decompositions in free-form English, and provides both vector and raster visual representations together with segmentation maps and JSON annotations. It was introduced as a benchmark for reference-game evaluation of abstract visual reasoning, with an emphasis on whether explicit part descriptions improve recognition and retrieval for both humans and machines (Ji et al., 2022).
1. Dataset definition and scope
KiloGram was constructed by digitizing 1,004 puzzles from Slocum (2000) and adding 12 canonical shapes used in prior behavioral work, yielding 1,016 unique stimuli (Ji et al., 2022). Each tangram is composed from the standard seven primitives: two large right triangles, one medium triangle, two small triangles, one square, and one parallelogram. The dataset is therefore anchored in a fixed combinatorial basis while varying the global configuration and the linguistic interpretations attached to that configuration.
The corpus was designed to be both visually and linguistically richer than earlier resources. In addition to whole-shape descriptions, it includes segmentation maps and part labels, allowing evaluation at multiple levels of abstraction. This design supports analysis of whole-object naming, part decomposition, and cross-modal alignment between text and image. The paper describes the resource as orders of magnitude larger and more diverse than prior resources, which suggests that KiloGram was intended not merely as an incremental benchmark extension but as a substrate for studying variability in abstract reference itself (Ji et al., 2022).
2. Construction pipeline and visual representations
The underlying tangrams were vectorized into SVG via corner-detection heuristics and manual validation. For model-training purposes, each SVG was also rendered as a PNG in two resolutions tied to specific model families: 224×224 for CLIP and 384×384 for ViLT (Ji et al., 2022). No further geometric augmentation was applied beyond uniform resizing and padding.
Two visual conditions are distributed. In the “black” condition, all seven pieces are uniformly black with thin white outlines. In the “color” condition, each semantically labeled part is tinted in one of seven preset CSS colors. This distinction is central to the benchmark because it operationalizes whether explicit correspondence between linguistic part labels and visual part highlighting improves abstract visual reasoning.
The dataset therefore encodes two distinct representational axes. One axis concerns geometry: the same seven-piece combinatorics rendered as SVG and PNG. The other concerns semantic highlighting: either no explicit part cueing (“black”) or part-level cueing through color (“color”). A plausible implication is that KiloGram was structured to separate the contribution of geometric abstraction from the contribution of cross-modal supervision over parts.
3. Annotation schema, collection procedure, and variability
KiloGram captures both whole-shape references and part-level decompositions in free-form English. Annotators on Amazon Mechanical Turk first answered “This shape, as a whole, looks like _,” then grouped the seven pieces into semantically coherent parts using prompts of the form “The part(s) you selected look(s) like _” until every piece was assigned (Ji et al., 2022). Part labels were unconstrained lemmas, producing 3,110 unique part lemmas and 3,031 whole-shape lemmas. No pre-defined hierarchy was imposed; hierarchical relations such as head body may be inferred post hoc from co-occurrence in descriptions.
Quality control was explicit. Workers were US-based, required at least 98% acceptance, completed an MTurk qualification task and a language-proficiency survey, and were limited to one annotation per tangram, with a maximum of 200 tangrams per worker. The collection process also used manual spot checks and compensation calibrated at approximately $0.14 per complete annotation, intended to approximate $12–15 per hour. In total, 13,404 annotations were collected across 1,016 shapes at a cost of $2,172.94 (Ji et al., 2022).
Each shape has on average 10.11 whole-part annotations in the “Full” set, and a subset of 74 shapes has more than 50 annotations each in the “Dense” set. Basic corpus statistics are reported as a mean whole-shape description length of tokens, a mean of parts per shape, and a mean of pieces per part. These values show that the dataset is not simply a collection of single-label object names; it systematically records decompositional variation.
To quantify annotation variability, the paper defines three measures. Shape Naming Divergence (SND), computed over whole-shape annotations, is given by
The mean SND in the Full set is . Part Naming Divergence (PND) is computed identically but over concatenated part labels, with mean . Part Segmentation Agreement (PSA) is defined by constructing, for each pair of segmentations, a cost matrix whose entries count matching pieces between parts and solving for the maximum-weight matching; the reported mean PSA is out of 7 (Ji et al., 2022).
These statistics formalize a central property of the corpus: naming is highly variable, while part segmentation exhibits substantial but non-trivial agreement. This suggests that KiloGram is suited to studying the interface between semantic flexibility and structural consistency.
4. Organization, splits, and licensing
All data are grouped by stimulus ID and include SVG files, rendered PNGs, segmentation maps, and JSON annotation files (Ji et al., 2022). The JSON schema for each shape contains an array of annotations with fields
{whole_shape_text, parts: [ {piece_ids:[],part_label}], worker_id}.
This organization preserves the relation between global descriptions and piece-level semantic partitions.
The benchmark split is defined over shapes rather than annotations, thereby preventing leakage of the same geometry across partitions. The released partitions are as follows:
| Split | Shapes |
|---|---|
| Training | 692 |
| Development | 125 |
| Held-out test | 125 |
| Test-dense | 74 |
The entire release is available at https://lil.nlp.cornell.edu/kilogram/. The downloadable package contains /svg/, /png/black/, /png/color/, /annotations/*.json, and LICENSE.txt. No registration is required. The data are released under U.S. “fair use” (Title 17 §108) for educational and research purposes, and usage is restricted to non-commercial research and educational projects under the attached agreement (Ji et al., 2022).
From a methodological standpoint, the shape-level split is significant because the evaluation target is abstraction over unseen tangrams rather than memorization of repeated annotations. The licensing terms also delimit the dataset’s intended use: reproducible academic research rather than unrestricted downstream commercialization.
5. Reference-game evaluation and model protocol
KiloGram evaluates abstract visual reasoning through a reference-game setup. Given a description and a set of $0.14 per complete annotation, intended to approximate $0 candidate images $0.14 per complete annotation, intended to approximate $1, the task is to select
$0.14 per complete annotation, intended to approximate $2
where $0.14 per complete annotation, intended to approximate $3 is a similarity score produced by a multi-modal model (Ji et al., 2022). This formulation turns abstract shape recognition into a retrieval problem and allows direct comparison across architectures.
Two Transformer-based models were benchmarked. CLIP (ViT-B/32) is treated as a dual-encoder contrastive model using
$0.14 per complete annotation, intended to approximate $4
ViLT is treated as a single-encoder model with an image–text matching head, using its binary-classification logit for “match” as $0.14 per complete annotation, intended to approximate $5. The evaluation considered four input conditions:
| Condition | Text input | Image input |
|---|---|---|
| whole+black | whole-shape text | black image |
| parts+black | “<whole> with <part_1>, …, <part_m>” | black image |
| whole+color | whole-shape text | color image |
| parts+color | combined text with parts | color image |
In the parts+color condition, text parts are italicized to match image tints. During fine-tuning, the benchmark further generates augmented examples by randomly selecting subsets of part labels and coloring only those parts, while all other parts remain black. Training uses a symmetric cross-entropy loss over $0.14 per complete annotation, intended to approximate $6 matching matrices, and all experiments use an ensemble of three identical models (Ji et al., 2022).
This protocol isolates several factors at once: the availability of global versus part-level language, the presence or absence of explicit visual part cueing, and the effect of joint versus separate image–text encoding. A plausible implication is that KiloGram functions less as a generic image–text benchmark than as a controlled probe of compositional alignment.
6. Empirical findings, human comparison, and limitations
The principal empirical result is that zero-shot pre-trained models perform near chance on the reference game. CLIP achieves 16–18% and ViLT 11–13% across conditions, with chance at 10% (Ji et al., 2022). The paper interprets this as evidence that off-the-shelf multi-modal models do not generalize concepts effectively to abstract tangram shapes.
Fine-tuning on approximately 6,600 reference-game examples changes this substantially. On the development set, whole+black yields 43.3% for CLIP and 40.9% for ViLT. Adding part names in the black condition raises performance to 45.3% for CLIP, a gain of 2.0 points, and 45.7% for ViLT, a gain of 4.8 points. Color alone in whole+color yields no benefit. By contrast, parts+color yields 45.4% for CLIP and 75.2% for ViLT, corresponding to a 34.3-point improvement over whole for ViLT. Data augmentation further raises CLIP to 47.6% and marginally affects ViLT at 72.2%. On the held-out test set, CLIP peaks at 50.2% with augmented parts+color, while ViLT reaches 77.3% (Ji et al., 2022).
Human evaluation provides a complementary baseline. With $0.14 per complete annotation, intended to approximate $7 participants completing 20 trials each, humans achieve 47.7% in whole+black and 63.0% in parts+color, an absolute gain of 15.3 points. A two-component Gaussian mixture fitted to human accuracies suggests sub-populations centered at approximately 52.5% and 83.8%. The paper also reports that model confidence increases monotonically with each additional part label and corresponding color, though gains diminish after approximately four parts, and ViLT consistently outperforms CLIP once part-level alignment is available (Ji et al., 2022).
Several limitations are stated directly. The results indicate that multi-modal pre-training alone does not endow models with human-like abstraction or compositionality, while relatively small amounts of targeted fine-tuning can unlock large improvements, especially in architectures that jointly encode image and text. The benchmark is therefore informative about the importance of structured visual decomposition and explicit part descriptions, but it does not by itself resolve broader questions about pragmatic under- and over-informativity, multilingual generalization, or instruction-following. Future directions proposed for the dataset include interactive reference games, extension to other languages and cultural contexts, and richer production and instruction-following tasks such as “color the head red” and “assemble the puzzle from these parts” (Ji et al., 2022).