---
title: 'LiveOIBench: OBI Benchmark'
url: https://www.emergentmind.com/topics/liveoibench
type: topic
---

# LiveOIBench: OBI Benchmark

LiveOIBench is an online, continuously-operating benchmark system designed to evaluate large multimodal models (LMMs) on the expert-level domain of Oracle-Bone Inscription (OBI) processing. Building on the foundation of the OBI-Bench dataset, LiveOIBench encompasses the full spectrum of expert tasks underlying the scholarly study of ancient script, supporting rigorous, transparent comparison across proprietary and open-source LMMs, and facilitating ongoing advances in multimodal understanding for archaeology and paleography [2412.01175].

## 1. Dataset Architecture and Scope

LiveOIBench is constructed from an ensemble of 5,523 high-fidelity images from 11 public and newly curated sources. The benchmark spans five core OBI domains—recognition, rejoining, classification, retrieval, and deciphering. Images originate from a diversity of archaeological contexts and represent all processing stages encountered in expert OBI workflows: original unprocessed fragments, inked rubbings, mixed and rejoinable fragments, cropped character images for classification/retrieval, and hand-printed or generative reproductions. This coverage supports model evaluation under both real-world noise and idealized conditions.

Key dataset components include:
- **Character recognition:** Large-scale sets with precision coordinate annotation, e.g., YinQiWenYuan_detection (2,000 images) and O2BR (800 images, 4,211 bounding boxes).
- **Fragment rejoining:** Manually separated and mapped adjacency challenges, e.g., OBI-rejoin (200 maps, 483 fragments).
- **Single-character classification and retrieval:** Balanced splits from HWOBC, Oracle-50K, OBI125, OBC306, OBI-IJDH.
- **Deciphering:** Expert-annotated, open-ended ground truth, e.g., EVOBC, OBI Component 20, HUST-OBS.

This breadth enables multifactorial probing of model capabilities, from perceptual localization through to OBI-specific semantic inference.

## 2. Benchmark Task Design and Evaluation Metrics

LiveOIBench frames all tasks using the Visual Question Answering (VQA) protocol. Each sample forms a tuple (I, Q, A), with I as an image, Q as a natural language query, and A the ground-truth answer. Tasks include:

- **Recognition:** Textual content description, binary detection (“Are these OBIs?”), character counting, bounding box localization.
- **Rejoining:** Identification and ranking of fragment pairings for adjacency reconstruction.
- **Classification:** N-way labeling of single-character crops, with optional confidence scoring.
- **Retrieval:** Searching gallery images for class-matched items, evaluated with Recall@k.
- **Deciphering:** Free-form meaning assignment, seeking maximal alignment with expert descriptions.

A comprehensive set of metrics is employed:
- **Top-k accuracy (Acc@k):** For classification and retrieval, $Acc@k = \frac{1}{N} \sum_{i=1}^N 1\{y_i \in \hat{Y}_i^{(k)}\}$.
- **Mean Relative Error (MRE):** For counting tasks.
- **Mean IoU (mIoU):** For bounding box localization.
- **Mean Average Precision (MAP):** For retrieval ranking.
- **Textual similarity and BERTScore:** For semantic alignment in recognition and deciphering.

All tasks are evaluated in a zero-shot setting; models have no access to held-out test data during training.

## 3. Evaluation Protocol and Statistical Rigor

LiveOIBench operates as a fully zero-shot, no-training split testbed. Each task is assessed on held-out test images, and model outputs are compared to “golden” expert-validated answers. To smooth model stochasticity, each LMM is queried five times per task, and scores averaged. Intermodel comparisons may employ paired t-tests for metrics such as mIoU and BERTScore, and agreement on deciphering or other categorical outputs is assessed with Cohen’s κ. Public human baselines are established by non-expert annotators, enabling fine-grained assessment of LMM-human parity across task types.

## 4. System and Infrastructure Design

LiveOIBench comprises multiple high-level system components:
- **Data Storage:** PostgreSQL or MongoDB instances archive image data, annotations, question/answer tuples, and version metadata.
- **Model Serving Layer:** Containerized (Docker/Kubernetes) deployment of both proprietary models (e.g., OpenAI/Gemini APIs) and locally hosted open-source LMMs supporting GPU/CPU scaling.
- **Inference Pipeline:** Microservices dispatch queries to model endpoints, process predictions, and compute evaluation metrics.
- **Web/API Backend:** FastAPI or Flask endpoints enable submission, status tracking, and result retrieval, supporting integration with external systems or user-facing front ends.
- **Dashboard:** React or Vue.js interfaces provide real-time leaderboard access, metric visualization, and model upload/prompt editing tools.

Model integration requires standardization of API schemas (for answer text, probabilities, bounding boxes). The pipeline supports batch assignments, query broadcasting, and real-time updating of per-model performance with confidence intervals.

Infrastructure design incorporates scalability (Kubernetes, horizontal GPU autoscaling), resilience (persistent volumes and logs), decoupling (RabbitMQ/Kafka message queueing), and CI-triggered auto-evaluation for new submissions and benchmark updates.

## 5. Baseline Model Results and Analysis

Baseline evaluations span 6 major proprietary LMMs (e.g., GPT-4o, Gemini 1.5 Pro, Qwen-VL-Max) and 17 open-source LMMs (e.g., InternVL2-76B, LLaVA-NeXT 72B, InternLM-XComposer2). Results demonstrate consistent superiority of proprietary models in perceptual localization (e.g., mIoU: Gemini 1.5 Pro 0.1126 vs. InternVL2-76B 0.0976; human: 0.9272), though all LMMs lag far behind expert human performance in fine-grained annotation. Proprietary LMMs also approach or surpass naive human baselines in deciphering common characters (BERTScore: GPT-4o 0.3876 vs. naive human 0.3872), with open-source leaders trailing but showing progress. All evaluated systems exhibit significant failures on noisy rubbings and complex fragment rejoining tasks.

Summary of selected benchmarks:

| Task            | Top Proprietary (Score)     | Top Open-Source (Score)   | Human (Score)     |
|-----------------|----------------------------|---------------------------|-------------------|
| Recognition     | Gemini 1.5 Pro (0.1126)    | InternVL2-76B (0.0976)    | 0.9272 (mIoU)     |
| Counting        | GLM-4V (0.3681)            | InternVL2-76B (0.4328)    | 0.0033 (MRE)      |
| Rejoining       | GPT-4o (78.47% Acc@10)     | InternVL2-76B (31.68%)    | –                 |
| Classification  | GPT-4o (72.75% Acc@1)      | InternVL2-76B (47.50%)    | –                 |
| Retrieval       | GPT-4o (0.686/0.706 R@1)   | LLaVA-NeXT (0.468/0.669)  | –                 |
| Deciphering     | GPT-4o (0.3876)            | InternLM-XComposer2 (0.2304) | 0.3872 (BERTScore) |

This establishes a high-precision, multi-task baseline for ongoing longitudinal comparison [2412.01175].

## 6. Deployment, Live Evaluation, and User Interfaces

The operational workflow for LiveOIBench involves preloading all 5,523 (image, query, answer) tuples into a central “test bus” for batch processing. Additional private test images or question types can be uploaded for extended evaluation. API endpoints are provisioned for model registration (`/submit_model`), evaluation job dispatch (`/run_evaluation`), job status polling (`/job_status`), and retrieval of results or filtered leaderboards. Interface components enable task toggling, image preview with overlayed predictions, fine-grained metric tables and visualizations, and custom prompt engineering. Continuous integration ensures reproducibility and system integrity by auto-evaluating any new model submission or benchmark update.

LiveOIBench is designed for real-time scalability and transparent, version-controlled benchmarking, supporting both routine and cutting-edge model assessment.

## 7. Future Directions

Planned extensions include domain-specific pre-training/fine-tuning of LMMs with additional expert-annotated OBI corpora, significantly expanding to rarer characters, underrepresented sites, and multilingual context. Interactive, human-in-the-loop workflows will be implemented to allow direct expert correction and active feedback for retraining. Enhanced metrics, such as expert-judged credibility for deciphering, and new OBI tasks—including multi-fragment spatial reassembly—are under consideration. Insights gained from OBI-specific noise and annotation challenges are to be transferred for historical document OCR in other domains (e.g., bronzes, bamboo slips). The long-term goal is to establish LiveOIBench as a robust, continuously-tracked platform for domain-adaptive LMMs, rigorously informing model development and methodological innovation in the computational humanities [2412.01175].

Source: https://www.emergentmind.com/topics/liveoibench