LiveOIBench: OBI Benchmark

• LiveOIBench is an online benchmark system designed to evaluate large multimodal models for Oracle-Bone Inscription processing in archaeology.
• It integrates over 5,500 high-fidelity images across five core OBI domains, employing VQA protocols and diverse metrics.
• The platform offers scalable model serving, real-time evaluation, and transparent leaderboards for proprietary and open-source systems.

LiveOIBench is an online, continuously-operating benchmark system designed to evaluate large multimodal models (LMMs) on the expert-level domain of Oracle-Bone Inscription (OBI) processing. Building on the foundation of the OBI-Bench dataset, LiveOIBench encompasses the full spectrum of expert tasks underlying the scholarly study of ancient script, supporting rigorous, transparent comparison across proprietary and open-source LMMs, and facilitating ongoing advances in multimodal understanding for archaeology and paleography (Chen et al., 2024).

1. Dataset Architecture and Scope

LiveOIBench is constructed from an ensemble of 5,523 high-fidelity images from 11 public and newly curated sources. The benchmark spans five core OBI domains—recognition, rejoining, classification, retrieval, and deciphering. Images originate from a diversity of archaeological contexts and represent all processing stages encountered in expert OBI workflows: original unprocessed fragments, inked rubbings, mixed and rejoinable fragments, cropped character images for classification/retrieval, and hand-printed or generative reproductions. This coverage supports model evaluation under both real-world noise and idealized conditions.

Key dataset components include:

• Character recognition: Large-scale sets with precision coordinate annotation, e.g., YinQiWenYuan_detection (2,000 images) and O2BR (800 images, 4,211 bounding boxes).
• Fragment rejoining: Manually separated and mapped adjacency challenges, e.g., OBI-rejoin (200 maps, 483 fragments).
• Single-character classification and retrieval: Balanced splits from HWOBC, Oracle-50K, OBI125, OBC306, OBI-IJDH.
• Deciphering: Expert-annotated, open-ended ground truth, e.g., EVOBC, OBI Component 20, HUST-OBS.

This breadth enables multifactorial probing of model capabilities, from perceptual localization through to OBI-specific semantic inference.

2. Benchmark Task Design and Evaluation Metrics

LiveOIBench frames all tasks using the Visual Question Answering (VQA) protocol. Each sample forms a tuple (I, Q, A), with I as an image, Q as a natural language query, and A the ground-truth answer. Tasks include:

• Recognition: Textual content description, binary detection (“Are these OBIs?”), character counting, bounding box localization.
• Rejoining: Identification and ranking of fragment pairings for adjacency reconstruction.
• Classification: N-way labeling of single-character crops, with optional confidence scoring.
• Retrieval: Searching gallery images for class-matched items, evaluated with Recall@k.
• Deciphering: Free-form meaning assignment, seeking maximal alignment with expert descriptions.

A comprehensive set of metrics is employed:

• Top-k accuracy (Acc@k): For classification and retrieval, $$Acc@k = \frac{1}{N} \sum_{i=1}^N 1\{y_i \in \hat{Y}_i^{(k)}\}$$.
• Mean Relative Error (MRE): For counting tasks.
• Mean IoU (mIoU): For bounding box localization.
• Mean Average Precision (MAP): For retrieval ranking.
• Textual similarity and BERTScore: For semantic alignment in recognition and deciphering.

All tasks are evaluated in a zero-shot setting; models have no access to held-out test data during training.

3. Evaluation Protocol and Statistical Rigor

LiveOIBench operates as a fully zero-shot, no-training split testbed. Each task is assessed on held-out test images, and model outputs are compared to “golden” expert-validated answers. To smooth model stochasticity, each LMM is queried five times per task, and scores averaged. Intermodel comparisons may employ paired t-tests for metrics such as mIoU and BERTScore, and agreement on deciphering or other categorical outputs is assessed with Cohen’s κ. Public human baselines are established by non-expert annotators, enabling fine-grained assessment of LMM-human parity across task types.

4. System and Infrastructure Design

LiveOIBench comprises multiple high-level system components:

• Data Storage: PostgreSQL or MongoDB instances archive image data, annotations, question/answer tuples, and version metadata.
• Model Serving Layer: Containerized (Docker/Kubernetes) deployment of both proprietary models (e.g., OpenAI/Gemini APIs) and locally hosted open-source LMMs supporting GPU/CPU scaling.
• Inference Pipeline: Microservices dispatch queries to model endpoints, process predictions, and compute evaluation metrics.
• Web/API Backend: FastAPI or Flask endpoints enable submission, status tracking, and result retrieval, supporting integration with external systems or user-facing front ends.
• Dashboard: React or Vue.js interfaces provide real-time leaderboard access, metric visualization, and model upload/prompt editing tools.

Model integration requires standardization of API schemas (for answer text, probabilities, bounding boxes). The pipeline supports batch assignments, query broadcasting, and real-time updating of per-model performance with confidence intervals.

Infrastructure design incorporates scalability (Kubernetes, horizontal GPU autoscaling), resilience (persistent volumes and logs), decoupling (RabbitMQ/Kafka message queueing), and CI-triggered auto-evaluation for new submissions and benchmark updates.

5. Baseline Model Results and Analysis

Baseline evaluations span 6 major proprietary LMMs (e.g., GPT-4o, Gemini 1.5 Pro, Qwen-VL-Max) and 17 open-source LMMs (e.g., InternVL2-76B, LLaVA-NeXT 72B, InternLM-XComposer2). Results demonstrate consistent superiority of proprietary models in perceptual localization (e.g., mIoU: Gemini 1.5 Pro 0.1126 vs. InternVL2-76B 0.0976; human: 0.9272), though all LMMs lag far behind expert human performance in fine-grained annotation. Proprietary LMMs also approach or surpass naive human baselines in deciphering common characters (BERTScore: GPT-4o 0.3876 vs. naive human 0.3872), with open-source leaders trailing but showing progress. All evaluated systems exhibit significant failures on noisy rubbings and complex fragment rejoining tasks.

Summary of selected benchmarks:

Task	Top Proprietary (Score)	Top Open-Source (Score)	Human (Score)
Recognition	Gemini 1.5 Pro (0.1126)	InternVL2-76B (0.0976)	0.9272 (mIoU)
Counting	GLM-4V (0.3681)	InternVL2-76B (0.4328)	0.0033 (MRE)
Rejoining	GPT-4o (78.47% Acc@10)	InternVL2-76B (31.68%)	–
Classification	GPT-4o (72.75% Acc@1)	InternVL2-76B (47.50%)	–
Retrieval	GPT-4o (0.686/0.706 R@1)	LLaVA-NeXT (0.468/0.669)	–
Deciphering	GPT-4o (0.3876)	InternLM-XComposer2 (0.2304)	0.3872 (BERTScore)

This establishes a high-precision, multi-task baseline for ongoing longitudinal comparison (Chen et al., 2024).

6. Deployment, Live Evaluation, and User Interfaces

The operational workflow for LiveOIBench involves preloading all 5,523 (image, query, answer) tuples into a central “test bus” for batch processing. Additional private test images or question types can be uploaded for extended evaluation. API endpoints are provisioned for model registration (/submit_model), evaluation job dispatch (/run_evaluation), job status polling (/job_status), and retrieval of results or filtered leaderboards. Interface components enable task toggling, image preview with overlayed predictions, fine-grained metric tables and visualizations, and custom prompt engineering. Continuous integration ensures reproducibility and system integrity by auto-evaluating any new model submission or benchmark update.

LiveOIBench is designed for real-time scalability and transparent, version-controlled benchmarking, supporting both routine and cutting-edge model assessment.

7. Future Directions

Planned extensions include domain-specific pre-training/fine-tuning of LMMs with additional expert-annotated OBI corpora, significantly expanding to rarer characters, underrepresented sites, and multilingual context. Interactive, human-in-the-loop workflows will be implemented to allow direct expert correction and active feedback for retraining. Enhanced metrics, such as expert-judged credibility for deciphering, and new OBI tasks—including multi-fragment spatial reassembly—are under consideration. Insights gained from OBI-specific noise and annotation challenges are to be transferred for historical document OCR in other domains (e.g., bronzes, bamboo slips). The long-term goal is to establish LiveOIBench as a robust, continuously-tracked platform for domain-adaptive LMMs, rigorously informing model development and methodological innovation in the computational humanities (Chen et al., 2024).