- The paper introduces KamonBench, a grammar-based dataset of 20,000 Japanese family crests designed to evaluate how well vision-language models can recover compositional factors such as containers, modifiers, and base motifs, along with specified diagnostics of factorization.
- KamonBench's diagnostics include direct measurements for factor accuracy, counterfactual motif sensitivity, and linear probes of factor accessibility, giving it an edge over conventional caption metrics, and identifying motif setting as the key bottleneck.
- Benchmarks reveal that containers and modifiers are effectively recovered but motif identification, especially under containment and held-out factors, remains a significant challenge for benchmarks and various models, confirming the gap that traditional models struggle to overcome.
KamonBench is a grammar-based image-to-structure benchmark built on Japanese family crests (kamon, 家紋), designed to evaluate compositional factor recovery in vision-LLMs. Each synthetic crest is generated from three known factors—a container C, a modifier R, and a base motif M—and is paired with a description in the kamon description language (KDL), a segmented Japanese analysis, an English translation, and a non-linguistic program code. Because the generative factors are known by construction, evaluation can go beyond caption-level string metrics to direct factor metrics, controlled recombination splits, counterfactual motif-sensitivity tests, and linear probes of factor accessibility.
Motivation: kamon as a constrained compositional domain
Kamon share structural properties with European heraldry, which uses the formal language of blazon; the corresponding Japanese formal language, kamon yĹŤgo, is less tightly constrained but still restricts how motifs and their arrangements are described. A crest decomposes into a container (an outer shell such as a ring or polygon), a modifier (either a spatial arrangement such as "three-stacked" or "bottoms-together," or a modification such as "demon" or "bean"), and a base motif. With 3,513 motifs, 36 containers, and six modifier values, the non-recursive combination space is roughly 770,000 crests, and recursive containment makes it theoretically unbounded.
The authors argue that this domain is a useful test of whether models represent motifs, containers, and modifiers as reusable visual factors rather than memorized whole crest–description pairs. Synthetic generation serves two purposes: it avoids contamination from web-crawled crest examples that proprietary models may have memorized, and it exposes the underlying factors for controlled diagnostics. The benchmark follows the diagnostic tradition of SCAN and CLEVR (Mignot, 2018, Johnson et al., 2016), but casts compositional generalization as visual recognition in a culturally grounded formal language. The authors are explicit that the benchmark does not claim unsupervised disentanglement or identifiable latent dimensions; it provides supervised factor-recovery and linear-accessibility diagnostics consistent with the known limitations of unsupervised disentanglement (Murakonda et al., 2019).
Dataset construction
KamonBench contains 20,000 composite examples generated from a BNF grammar over the factor structure. Each composite example includes a rendered crest, a KDL description, a segmented Japanese analysis, an English translation, and a program label encoding (C,R,M). Auxiliary component examples provide one isolated base motif per composite example and one isolated container per contained composite, yielding 20,000 motif components, 14,116 container components, and 54,116 examples total. Composites are split 0.8/0.1/0.1 into train/development/test sets, with component examples included in the same splits for training.
Three target label spaces are used over identical images and splits: segmented Japanese KDL analyses, English translations, and deterministic program codes derived from generator components. The annotation pipelines for the Japanese and English targets are not released, which is a transparency limitation worth noting. The release also includes three recombination splits—(C,M), (R,M), and (C,R,M)—though only the (C,M) split is evaluated in the paper's experiments. Component images derive from the Rebolforces kamondataset scraped from a now-defunct catalogue website; upstream provenance cannot be tracked further, and the data are released under CC-BY-NC 4.0 for non-commercial research use.
Evaluation design
The benchmark supports four families of diagnostics. Aggregate metrics include exact-match accuracy, accuracy restricted to novel-image/novel-target examples (AccNIT​), and character- or token-level edit distance (CER/TER). Factor metrics decompose program-code predictions into container, modifier, and motif accuracies; natural-language outputs are deliberately left unmapped to factors because the mapping would be ambiguous. Counterfactual motif sensitivity holds the container-modifier context fixed and varies the motif, measuring motif separation, pair motif accuracy, and collapsed motif groups—an intervention-style test aligned with causal representation learning (Schwob et al., 2021). Linear probes on frozen representations test whether each factor is linearly accessible before decoding, with the caveat that probe accuracy shows what a linear readout can recover, not what the decoder actually uses.
Baseline results
Three baseline families are trained separately per label space: a ViT encoder with an autoregressive Transformer decoder, and ImageNet-initialized VGG16 n-gram decoders with and without learned position-dependent masks that read the crest from outside to inside.
| Model |
Label |
CER/TER |
Acc |
AccNIT​ |
| ViT |
Japanese |
0.035 |
0.895 |
0.893 |
| ViT |
English |
0.117 |
0.571 |
0.516 |
| ViT |
Program |
0.022 |
0.941 |
0.945 |
| VGG masks |
Program |
0.068 |
0.821 |
0.841 |
| VGG no masks |
Program |
0.101 |
0.732 |
0.783 |
ViT dominates all label spaces. The factor decomposition shows that aggregate program accuracy is largely a test of motif binding rather than output syntax: all models recover containers at essentially 1.000 accuracy and modifiers above 0.98, while motif accuracy separates the models (ViT 0.946, masked VGG 0.826, no-mask VGG 0.735). Counterfactual tests confirm that all models react to motif changes (motif separation 1.000, zero collapsed groups), but pair motif accuracy preserves the same ranking, indicating that weaker models map motif differences to wrong identities rather than ignoring them.
On the controlled R0 recombination split—which holds out container-motif pairs while both primitives appear elsewhere in training—the ordering persists and the gap remains motif-driven: ViT achieves 0.907 overall accuracy versus 0.711 (masked VGG) and 0.572 (no-mask VGG). This split targets systematicity in Hupkes et al.'s taxonomy: recombination of familiar primitives under novel combinations.
Linear probes reveal a sharper representational gap. Container and modifier labels are nearly perfectly linearly accessible in all three representations (probe accuracy ≥ 0.987), but motif accessibility differs substantially: ViT reaches 0.776 overall motif probe accuracy (motif cross-entropy 1.401), masked VGG 0.552 (MCE 4.905), and no-mask VGG 0.412 (MCE 6.520). Notably, ViT's motif probe accuracy drops from 0.971 on train-present R1 combinations to 0.702 on held-out ones, and masked VGG falls to 0.091 probe accuracy on held-out R2 combinations—evidence that even the strongest representation binds motifs less reliably under recombination.
A reduced-training-data sweep (2,500 / 5,000 / 10,000 composites) shows steep scaling: ViT program accuracy rises from 0.424 to 0.691 to 0.876, while container and modifier accuracies remain near ceiling throughout, again isolating motif binding as the bottleneck.
An appendix documents an instructive failure mode: an initial unregularized masked VGG n-gram-2 program model collapsed the entire contained slice (1,431 examples) onto a single motif token, M:0808, which appears only 13 times in training and never as a dev/test target—yielding 0.000 contained motif accuracy despite near-perfect container and modifier recovery. Regularization and checkpoint selection removed the collapse. This case directly motivates the counterfactual and group-level diagnostics, which would surface such collapse that aggregate TER compresses into a single low score.
Human and multimodal LLM comparisons
Two small studies contextualize task difficulty. In a few-shot transcription test on 20 random synthetic examples, Claude Opus 4.7 Max produced zero correct transcriptions and GPT 5.4 xhigh one, whereas the fine-tuned VGG baseline made one error and ViT none. Both LLMs sometimes recovered individual components, indicating partial but insufficient exposure or capability. Because proprietary training data cannot be audited, this experiment addresses the contamination confound directly: whatever web exposure these models may have had does not translate into few-shot success on synthetic crests.
In a human study, 32 participants received brief instructions on basic motifs, modifications, and spatial arrangements, then analyzed ten synthetic examples. Two participants made no errors; modification identification was the hardest category (64% of errors). Self-reported prior knowledge did not predict performance—the two error-free participants reported no prior knowledge, while the worst participant reported some. Against the same ten-example task, GPT made 3 errors and Claude 10; 44% of participants matched or exceeded GPT, and all participants performed at least as well as Claude. These results indicate that local aspects of the kamon system are learnable from minimal instruction by humans, while closed multimodal models do not reliably solve the synthetic task from a small prompt.
Limitations
The paper concedes several constraints plainly. Synthetic crests are less polished than professionally rendered kamon and differ from real crests found in books or on the web, so results may not transfer to naturalistic crest images. Only a small set of baselines is evaluated; Japanese and English outputs are treated as strings without mapping back to generator factors; probing covers only linear accessibility from a single feature vector per image; and retraining experiments cover only the R3 split, leaving the released R4 and R5 splits unanalyzed. The Japanese and English annotation pipelines are unreleased, and the provenance of source motif images terminates at an archived scrape. An open question the paper leaves explicit is whether stronger architectures or more data close the motif-binding gap under containment and recombination, or whether it reflects a more fundamental limitation of how current encoders bind fine-grained visual identity within structured contexts.
Conclusion
KamonBench contributes a controlled, culturally grounded testbed in which every image carries known symbolic factors, enabling evaluation of sparse compositional visual recognition beyond aggregate caption metrics. Its central empirical finding is consistent across baselines, data scales, and evaluation levels: containers and modifiers are recovered almost perfectly and are linearly accessible, while motif identity—especially under containment and held-out factor combinations—is the dominant bottleneck, in both decoding behavior and frozen representations. The accompanying human and proprietary-LLM studies establish that the task is learnable from limited instruction yet resistant to few-shot solution by large multimodal models, making the benchmark a practical probe of compositional factor recovery that is difficult to satisfy through memorization alone.