PictOBI-20k: Visual Decipherment Benchmark

• PictOBI-20k is a benchmark that pairs oracle bone character images with corresponding real-object images, formulating decipherment as a 4-choice visual alignment task.
• The dataset comprises 20,008 images across 80 categories, curated from diverse sources to represent both standalone and complex pictographic forms.
• The evaluation protocol, which includes 4-choice accuracy and human reference-point alignment, probes whether multimodal models leverage genuine visual reasoning over language priors.

Searching arXiv for the specified paper to ground the response in the current literature. arXiv search query: (Chen et al., 6 Sep 2025) PictOBI-20k is a large pictograph-specific benchmark for evaluating large multimodal models (LMMs) on the visual decipherment of oracle bone characters (OBCs), introduced in "PictOBI-20k: Unveiling Large Multimodal Models in Visual Decipherment for Pictographic Oracle Bone Characters" (Chen et al., 6 Sep 2025). It pairs images of OBC glyphs with images of corresponding real-world objects and formulates decipherment as a 4-choice visual alignment problem rather than a text-only interpretation task. The benchmark also includes human “reference point” annotations for studying whether model visual reasoning aligns with human-marked salient regions. Its stated purpose is to construct a rigorous, large-scale visual benchmark, quantify LMM performance on pictographic OBC decipherment, and probe whether contemporary LMMs actually use visual information or default to language priors (Chen et al., 6 Sep 2025).

1. Historical and methodological setting

Oracle bone characters are described as the oldest attested form of written Chinese, dating to the Shang Dynasty, 1250–1050 BC, and as a key resource for understanding early modes of production, social structures, and the evolution of civilization (Chen et al., 6 Sep 2025). The paper situates OBC decipherment within a long-standing scholarly program in which textual interpretation is constrained by the sporadic nature of archaeological excavations and the limited inscriptional corpus.

The benchmark is motivated by limitations in traditional decipherment methodologies, including archaeological evidence, genealogical comparison, contextual analysis, and the object-referential method. A central difficulty is that textual ground-truth for OBC interpretation often requires expert consensus, which makes quantitative evaluation difficult. PictOBI-20k responds by focusing on visual decipherment: mapping an OBC pictograph to its corresponding object image. This framing is presented as more intuitive to evaluate because it mirrors early character creation while avoiding some of the indeterminacy associated with full philological interpretation (Chen et al., 6 Sep 2025).

A common misconception would be to treat PictOBI-20k as a benchmark for complete OBC decipherment in the textual or epigraphic sense. The benchmark is narrower. It evaluates pictographic visual mapping, not the entirety of scholarly OBC interpretation. This distinction is methodologically important because the dataset is designed to make evaluation more objective precisely by restricting the task to visually grounded correspondences.

2. Dataset composition and curation

PictOBI-20k contains 20,008 images in total: 15,175 OBC images and 4,833 real object images, with sources spanning 12 channels, including 8 for OBCs and 4 for objects (Chen et al., 6 Sep 2025). The reported average resolutions are 512×512 for OBC images and 512×610 for object images.

Component	Value	Note
Total images	20,008	Rounded to “20k”
OBC images	15,175	Rubbing or handprinted form
Object images	4,833	Real object photos/illustrations
Total categories	80	Following Xigui Qiu’s taxonomy
Questions	15,175	4-choice format

The OBC images are curated from OBC-centric sites and open datasets and cover multiple font appearances. The sources and counts reported in Table 1 are YinQiWenYuan (761), XiaoXueTang (1,731), GuoXueDaShi (35), Oracle-241 (1,994), Oracle-50K (5,635), HUST-OBS (4,539), OBI125 (410), and OBIdatasetIJDH (70) (Chen et al., 6 Sep 2025). The object images are sourced from Freepik, Pexels, Pinterest, and the Bronze Ware Database, and are described as manually curated under appropriate CC BY licenses to visually match the pictographic semantics of the OBC categories.

The category system comprises 80 pictographic OBC types following Xigui Qiu’s taxonomy. Of these, 70 are “normal” categories, defined as standalone pictographs representing objects in isolation, and 10 are “complex” categories, defined as composite images requiring contextual elements and not representing a single object in isolation (Chen et al., 6 Sep 2025). This distinction is central to later analyses because the complex categories operationalize context sensitivity rather than simple silhouette matching.

The dataset is positioned relative to earlier OBC resources such as Oracle-241, Oracle-50K, HUST-OBS, OBI125, and OBIdatasetIJDH. Those earlier datasets are characterized as being used primarily for recognition or genealogy, whereas PictOBI-20k is described as unique in pairing OBC images with real object images for pictographic visual decipherment and in combining that formulation with human reference-point annotations (Chen et al., 6 Sep 2025).

3. Task formulation and annotation protocol

The benchmark defines two evaluation tasks. The first is OBC-to-object matching in a 4-choice format. Each question uses an OBC image as input and the prompt: “Which real object corresponds to the given OBC image?” The model must select one correct answer among four options, with distractors randomly sampled from other classes and option order cyclically rotated to mitigate option-position bias. The paper explicitly notes that the random baseline for this setup is 25% (Chen et al., 6 Sep 2025).

The second task is human–LMM reference point alignment. Given a marked “REF point” on the OBC image, the model is asked: “Which point best corresponds to the REF point given in the OBC image?” It then selects among four candidate points on the object image, where two points are human-annotated corresponding regions and two are distractors (Chen et al., 6 Sep 2025).

The human annotation protocol covers a subset of 240 OBC–object pairs, obtained by selecting three pairs per type across 80 types. Five OBC experts with an ancient philology background annotated the pairs. For each pair, annotators marked two corresponding reference points on the OBC image and the object image and added two irrelevant points as interference; the point radius was set to 8 pixels for visibility (Chen et al., 6 Sep 2025). The paper does not report explicit annotation guidelines beyond “mark two points indicating corresponding regions,” nor does it report inter-annotator agreement metrics such as Cohen’s kappa.

This design makes a specific methodological intervention. Instead of asking only whether a model selects the correct object category, it also asks whether the model aligns the same local regions that human experts treat as salient. A plausible implication is that the reference-point task functions as a diagnostic of visual grounding rather than merely end-task accuracy.

4. Evaluation setup and metrics

The benchmark is treated as an evaluation resource rather than a train/validation/test benchmark; the paper does not report train, validation, or test splits, and it does not provide per-category sample counts (Chen et al., 6 Sep 2025). The experiments appear to be single-turn, zero-shot prompts with images, although the paper does not report decoding parameters such as temperature or top-$k$, nor whether any few-shot exemplars were used.

The official metric for the multi-choice task is accuracy:

$$\text{Acc} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\big(\hat{y}_i = y_i\big)$$

This metric is used in the tables and figures for OBC-to-object matching (Chen et al., 6 Sep 2025). For the reference-point task, the paper reports “consistency (%)” on the 240 annotated pairs, but does not provide a formal formula. The dataset description provides a conventional operationalization as accuracy over the point-selection task.

The evaluated LMMs comprise 11 models. The proprietary models are GPT-4o-2024-11-20, Gemini 2.5 Pro, and Claude 4 Sonnet-20250514. The open-source models are GLM-4.5V-106B, Qwen2.5-VL-{3B, 7B, 32B, 72B}, and InternVL3-{8B-Instruct, 38B-Instruct, 78B-Instruct} (Chen et al., 6 Sep 2025). In addition, the paper analyzes standalone vision backbones—DINOv2 L/14, CLIP L/14, and InternViT-300M-448px-V2.5—using patch-level cosine similarity between the annotated reference and candidate options, following Fu et al., 2025. For non-API models, the reported hardware is 4× Nvidia H200 (141 GB) GPUs (Chen et al., 6 Sep 2025).

The methodological contrast between full LMMs and standalone vision backbones is one of the benchmark’s distinctive features. It is not only an application benchmark but also an instrument for testing whether multimodal systems actually exploit visual representations.

5. Empirical results

The main evaluation results are reported as accuracy on normal, complex, and overall subsets (Chen et al., 6 Sep 2025).

Model	Overall accuracy (%)	Note
Random 4-choice baseline	25.00	Normal/Complex/Overall all 25.00
Gemini 2.5 Pro	53.66	Best proprietary; overall best
InternVL3-38B-Instruct	51.40	Best open-source overall
InternVL3-78B-Instruct	50.71	Second-best open-source
Claude 4 Sonnet-20250514	34.94	Proprietary
GLM-4.5V-106B	32.48	Open-source
Qwen2.5-VL-32B	32.56	Reported as such in the paper
GPT-4o-2024-11-20	26.23	Slightly above random
InternVL3-8B-Instruct	26.99	Open-source
Qwen2.5-VL-7B	26.17	Open-source
Qwen2.5-VL-72B	25.36	Open-source
Qwen2.5-VL-3B	24.76	Below random baseline

At the subset level, the paper reports that Gemini 2.5 Pro attains 55.22 on normal categories, 39.44 on complex categories, and 53.66 overall. InternVL3-38B-Instruct attains 52.71 on normal, 39.51 on complex, and 51.40 overall. GPT-4o-2024-11-20 records 26.31 on normal, 25.52 on complex, and 26.23 overall. Across the table, accuracy on complex OBCs is consistently lower than on normal OBCs, which the paper interprets as evidence that context-dependent composite pictographs remain difficult for current systems (Chen et al., 6 Sep 2025).

The inter-class analysis reported in Fig. 3 gives a mean per-class accuracy of 35.29%, a standard deviation of 8.25%, and a range from 19.56% to 54.25%. Classes with higher accuracy are reported to exhibit strong structural similarity between the OBC glyph and the object image, whereas low-accuracy classes lack such resemblance (Chen et al., 6 Sep 2025). This suggests that current performance is strongly tied to overt morphological correspondence.

The paper also reports scaling behavior. Both the InternVL3 and Qwen2.5-VL series exhibit an “increase-first, saturate-later” scaling law. The authors state that this may hint at a potential “golden ratio” between vision and language backbones and at an inherent tension between visual representation and language priors (Chen et al., 6 Sep 2025). Since the paper does not formalize this “golden ratio,” it is best understood as an empirical conjecture rather than an established law.

6. Human–model alignment, interpretation, and limitations

On the 240 annotated OBC–object pairs used for the reference-point task, Gemini 2.5 Pro achieves the highest reported human–LMM consistency at 76.25% (Chen et al., 6 Sep 2025). The paper also reports that GLM-4.5V “exceed[s] GPT-4o-20241120 significantly,” which is taken as evidence of better visual comparison ability between those proprietary and open models in this diagnostic setting. A scaling law trend similar to the object-matching results is also observed for consistency.

The qualitative interpretation is that human annotations reveal high morphological alignment between pictographic OBCs and their object images. Models that better capture local visual correspondences, rather than relying on language priors, align more closely with human-marked salient regions (Chen et al., 6 Sep 2025). This is reinforced by the vision-backbone analysis: DINOv2’s direct visual readout, based on cosine similarity on patch features, can outperform Claude 4 Sonnet and InternVL3-78B in object matching in the reported analysis, and its attention maps are described as more regionally differentiated and better aligned with human reference points than those of CLIP-L and InternViT in some cases (Chen et al., 6 Sep 2025).

The paper’s core interpretive claim is therefore not simply that current LMMs are imperfect, but that they are “not effectively using visual information” and are “limited by language priors” much of the time. The consistently lower performance on complex categories is presented as further evidence for this diagnosis. A plausible implication is that end-to-end multimodal instruction tuning can preserve or even amplify language-dominant heuristics unless visual grounding is explicitly strengthened.

Several limitations are explicitly noted. The corpus reflects sparse, noisy, and stylistically variable inscriptions. Complex categories require contextual elements and are harder for both humans and LMMs. There may be class imbalance, but per-category counts and benchmark splits are not reported. Distractor sampling is random across classes, which means question difficulty can vary; option rotation mitigates but does not eliminate option-position bias (Chen et al., 6 Sep 2025). The paper also does not report controlled ablations such as image removal, noise substitution, or language-only baselines, and does not quantify saliency alignment with metrics such as KL divergence.

From a cultural-heritage perspective, the benchmark is careful in scope. OBC decipherment is a scholarly domain in which textual ground-truth is not always consensual, so the dataset concentrates on pictographic visual mapping as a more objective evaluation target. Real object images are sourced under CC BY, whereas licensing for OBC images depends on the original repositories; the repository linked by the paper is the stated point of access for code and data (Chen et al., 6 Sep 2025).

The future directions stated in the paper emphasize vision-centric tuning, stronger visual attention mechanisms, and tighter alignment with human reference points. The authors also call for investigation of the proposed vision–language “golden ratio,” controlled ablations to disentangle language priors from genuine visual reasoning, and expansion of complex categories and contextual understanding (Chen et al., 6 Sep 2025). In that sense, PictOBI-20k functions both as a benchmark and as a diagnostic framework for measuring whether multimodal models genuinely perform visual decipherment rather than approximate it through textual priors.