Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation

Abstract: Deciphering ancient Chinese Oracle Bone Script (OBS) is a challenging task that offers insights into the beliefs, systems, and culture of the ancient era. Existing approaches treat decipherment as a closed-set image recognition problem, which fails to bridge the ``interpretation gap'': while individual characters are often unique and rare, they are composed of a limited set of recurring, pictographic components that carry transferable semantic meanings. To leverage this structural logic, we propose an agent-driven Vision-LLM (VLM) framework that integrates a VLM for precise visual grounding with an LLM-based agent to automate a reasoning chain of component identification, graph-based knowledge retrieval, and relationship inference for linguistically accurate interpretation. To support this, we also introduce OB-Radix, an expert-annotated dataset providing structural and semantic data absent from prior corpora, comprising 1,022 character images (934 unique characters) and 1,853 fine-grained component images across 478 distinct components with verified explanations. By evaluating our system across three benchmarks of different tasks, we demonstrate that our framework yields more detailed and precise decipherments compared to baseline methods.

• The paper presents a novel agent-driven, multimodal RAG pipeline integrating component recognition, structured knowledge retrieval, and multi-agent inference to tackle Oracle Bone Script challenges.
• It introduces the OB-Radix dataset with high-fidelity, structure-aware annotations of 1,022 images and 1,853 component masks, enhancing semantic analysis.
• Experimental results show significant improvements in retrieval accuracy and interpretation quality, validated by both automatic metrics and expert evaluations.

Component-Grounded Specialization of Large Models for Oracle Bone Script Interpretation

Introduction and Motivation

Oracle Bone Script (OBS), as China’s earliest mature writing system, presents considerable interpretive challenges due to its fragmentary inscriptions and the limited decipherment of its large glyph corpus. OBS characters are composed of recurrent pictographic components (radicals) with distinct semantic values (Figure 1). Traditional AI approaches based on closed-set image recognition have failed to address the interpretive gap because they lack systematic integration of compositional and contextual knowledge. This paper introduces a new paradigm: agent-driven, multimodal, component-grounded interpretation leveraging Vision-LLMs (VLMs), knowledge graphs, and multi-agent retrieval-augmented generation (RAG) (2604.06711).

Figure 1: Oracle Bone Script (OBS) exemplifies a structured pictographic system composed of meaningful semantic components.

OB-Radix Dataset: Structure-Aware Annotation

A new annotated dataset, OB-Radix, is presented to facilitate component-level analysis. OB-Radix comprises 1,022 character images (934 unique characters) and 1,853 component masks spanning 478 distinct radicals. Annotation prioritizes semantic function and paleographic consistency, enabling downstream modeling to exploit component-based decompositions rather than arbitrary visual regions (Figure 2). Experts utilized LabelMe for high-fidelity segmentation, ensuring reliable anchor points for knowledge retrieval and interpretative reasoning.

Figure 2: Component-level annotation showcasing semantic segmentation of OBS characters grounded in paleographic function.

Agentic Retrieval-Augmented Generation Framework

The core methodological innovation is the multistage agentic RAG pipeline (Figure 3), integrating component recognition, structured retrieval, relationship inference, and interpretation synthesis:

• Component Identification: Utilizes ViT-based DINOv2 encodings with prototype-based classification (Prototypical Networks). The structured feature space enables robust generalization in a low-resource regime and is further detailed in vector space construction analyses.
• Knowledge Graph Retrieval: An LLM agent orchestrates queries over a carefully constructed knowledge graph (KG) extracted from OB-Radix. The agent employs dynamic tool invocation for explanation and co-occurrence, supported by semantic-similarity caching for efficiency.
• Component Relationship Inference: VLMs infer structural relationships, including inscription type (ideographic, pictographic, phono-semantic), generating reasoning traces grounded in visual and semantic evidence.
• Interpretation Generation: Supports two inference strategies—direct VLM fusion and multi-agent separation (retrieval vs. reasoning), improving robustness and semantic coherence.

  Figure 3: The pipeline combines visual encoding, knowledge graph retrieval, inter-component relationship inference, and semantic interpretation generation.

Experimental Results and Evaluation

Three progressive tasks benchmark the system: component retrieval, relationship inference, and interpretation generation.

Component Retrieval:

• Top-1/Top-3/Top-5 accuracy: 0.7795/0.8855/0.9157.

Relationship Inference:

• Significant improvement in classification and reasoning: For Qwen3-VL, accuracy increases from 0.350 (baseline) to 0.599 (proposed); GPT-5 achieves BERTScore of 0.670, nearly 35% higher than baseline.
• LLM-as-a-Judge scores reflect marked gains in interpretability.

  Figure 4: Reasoning trace outputs illustrate enhanced interpretability and structural insight in component-based inference.

Interpretation Generation:

• RAG pipelines consistently outperform baselines across BERTScore, MoverScore, ROUGE-1, and LLM-as-a-Judge.
• For GPT-5, BERTScore rises from 0.633 (baseline) to 0.727 (RAG), and LLM-as-a-Judge shows a 0.456 absolute increase.
• Multi-agent separation further improves semantic grounding and aligns outputs with expert reference.

  Figure 5: Comparative outputs demonstrate the superior semantic grounding of agentic multi-agent RAG versus baseline methods.

Human Expert Assessment:

• Likert scale assessment reveals multi-agent outputs scored 3.433 vs. 2.133 (KG-RAG) and 1.367 (baseline).
• Inter-rater reliability metrics (ICC3: 0.71; Krippendorff’s Alpha: 0.74) confirm evaluation consistency.

Supplementary Studies and Ablation Analyses

Cross-Lingual Evaluation:

• English output performance is consistently lower, confirming the primary bias toward Chinese-centric knowledge bases.
• Retrieval-augmented settings mitigate this degradation, indicating robust relative improvements.

Variant Recognition:

• Recognition of visually distinct variants remains intrinsically difficult without explicit structural or semantic mapping.

Ablation Studies:

• Disabling knowledge graph retrieval reduces interpretive performance, particularly on semantic correctness metrics, validating the necessity of agentic evidence retrieval.

Multi-Agent Collaboration:

• Multi-agent configurations (distinct retrieval and reasoning agents) outperform single-agent setups across all metrics, with improved coherence and interpretive precision.

Practical and Theoretical Implications

The results demonstrate that specialized models for ancient script interpretation require systematic integration of multimodal priors and structured external knowledge. The component-grounded, agent-driven paradigm enables semantic transfer for rare and unseen glyphs—the primary theoretical advance in model specialization for low-resource, historically grounded tasks. Practically, this pipeline generates outputs aligned with domain expertise, supporting human-in-the-loop scholarly workflows. The framework generalizes to other ancient scripts or any domain with rich compositional logic and unresolved semantic gaps.

Future Developments

The paper identifies several limitations: incomplete component recognition, reliance on external KGs, and difficulty with variant forms and phono-semantic compounds. Future research should focus on autonomous acquisition of domain expertise, advanced variant mapping, expanded multi-lingual and cross-modal KG construction, and tighter coupling between model inference and paleographic authority.

Conclusion

This study establishes a robust, interpretable framework for Oracle Bone Script interpretation via agentic, retrieval-augmented multimodal augmentation. Empirical results indicate substantial improvements in both semantic accuracy and interpretability, validated by automatic metrics and expert evaluation. The methodological innovations and annotated resources provide a replicable benchmark for further advancement in domain-specialized vision-LLMs and AI-assisted cultural heritage analysis.

Figure 6: Comparative illustration: agentic RAG integrates component-level knowledge for superior semantic augmentation compared to baselines.