HistCaps: Historical Dataset for Temporal Reasoning
- HistCaps is a curated dataset of 515 bilingual historical images designed to evaluate temporal inference in vision-language models.
- It provides rich annotations including historical tags and long/short bilingual captions to support event-level supervision.
- Integrated into CompareTemporalBench, HistCaps challenges models to combine visual cues with world knowledge for chronology tasks.
Searching arXiv for the cited paper and related HistCaps references. HistCaps is a curated dataset of historical visual content introduced as one of the two auxiliary datasets used to build CompareBench, a benchmark for visual comparison reasoning in vision-LLMs (VLMs). It consists of 515 historical images with bilingual English–Chinese annotations and serves specifically as the source of CompareBench’s temporal comparison task, CompareTemporalBench. In this role, HistCaps provides historically grounded imagery, event-level tags, and caption-based supervision for evaluating whether VLMs can infer chronology from visual evidence and background knowledge rather than from purely perceptual cues alone (Cai et al., 25 Sep 2025).
1. Definition and provenance
HistCaps is defined in the source paper as a curated dataset of 515 historical images with bilingual annotations in English and Chinese. It is one of the two auxiliary datasets underlying CompareBench, alongside TallyBench, and it is the component used to construct the benchmark’s temporal comparison subset, CompareTemporalBench (Cai et al., 25 Sep 2025).
The dataset was created to support chronological understanding and temporal reasoning in VLMs. The motivating premise is that visual comparison reasoning remains a weak capability in current multimodal systems, especially when the comparison depends on historical ordering rather than on directly observable object properties. HistCaps therefore functions as a historical-image resource for testing whether a model can connect scene content to temporal sequence.
Within CompareBench’s four-way task decomposition—quantity, temporal, geometric, and spatial—HistCaps anchors the temporal dimension. Its role is consequently narrower than that of a general-purpose image-caption dataset: it is designed to enable controlled evaluation of event chronology.
2. Annotation schema and dataset structure
HistCaps does not store question–answer pairs. Instead, each image entry is enriched with a compact annotation structure intended to preserve historical semantics while remaining reusable for later benchmark construction (Cai et al., 25 Sep 2025).
| Field | Content |
|---|---|
| Historical tag | Summary of the key event |
| Long caption | English and Chinese |
| Short caption | English and Chinese |
The paper explicitly contrasts HistCaps with TallyBench on this point: unlike TallyBench, HistCaps contains no QA pairs. The QA-style formulation is introduced only later when HistCaps entries are assembled into CompareTemporalBench. This separation between descriptive annotation and benchmark instantiation is central to how HistCaps is used.
The historical tag provides event-level supervision, while the long and short captions provide two granularities of bilingual textual context. The long caption supplies detailed visual and historical context; the short caption provides concise reference. This design supports temporal comparison by coupling image content with event identity and bilingual description.
3. Historical coverage and temporal semantics
HistCaps is described as covering a wide temporal span of historical events. The paper lists political treaties, wars, revolutions, inaugurations, disasters, and cultural milestones among its event types (Cai et al., 25 Sep 2025). Examples explicitly mentioned include the 1707 Articles of Union, 18th-century naval battles, the 1776 Declaration of Independence, the 1789 Washington inauguration, the 1945 WWII surrender, and the 1969 Moon Landing.
This breadth matters because temporal inference in HistCaps is not reducible to detecting whether an image merely appears “old.” The benchmark construction assumes that chronology may need to be inferred from clothing styles, architecture, technology, environmental context, and world knowledge about historical events or eras. The source paper states that these cues must be integrated rather than treated in isolation.
A common misunderstanding is to treat such a dataset as a visual age-estimation resource. The paper instead frames HistCaps as a resource for knowledge-augmented temporal reasoning. In that framing, a historically earlier event may have to be identified by combining scene-level visual signals with learned knowledge of political, military, cultural, or technological chronology.
4. Use in CompareTemporalBench
HistCaps is the foundation of CompareTemporalBench, the temporal sub-benchmark within CompareBench. CompareTemporalBench contains 100 samples, and each sample is a four-image grid composed of HistCaps entries. The task is to identify which scene corresponds to the earliest historical event (Cai et al., 25 Sep 2025).
The benchmark’s instruction protocol is standardized. For each four-image grid, the model must:
- use visible visual cues such as clothing, architecture, technology, and environment
- combine these cues with relevant world knowledge
- determine which event occurred earliest
- account for possible resizing distortions across sub-images
- answer with a single letter only
The output format is strictly constrained to a single letter, , , , or , with no extra text. This converts HistCaps from a descriptive historical-image corpus into a diagnostic multiple-choice temporal ordering task.
CompareTemporalBench is one of four sub-benchmarks in CompareBench:
- CompareTallyBench — quantity
- CompareTemporalBench — temporal ordering
- CompareGeometryBench — geometry
- CompareSpatialBench — spatial relations
The paper summarizes CompareBench as being “built upon TallyBench, HistCaps, and additional human annotations.” HistCaps is therefore not an isolated dataset artifact; it is part of a modular benchmark architecture in which auxiliary corpora are transformed into targeted comparison tasks.
5. Reported performance on temporal comparison
The source paper reports results for CompareTemporalBench rather than per-category statistics internal to HistCaps. Beyond the corpus size of 515 samples, no finer-grained statistical breakdown for HistCaps itself is provided (Cai et al., 25 Sep 2025).
| System | CompareTemporalBench |
|---|---|
| Human | 30.00% |
| OpenAI GPT-5 | 74.00% |
| OpenAI o3-pro | 72.00% |
| Gemini 2.5 Pro | 64.00% |
| Qwen3-VL-235B-A22B | 32.00% |
| Qwen2.5-VL-72B | 29.00% |
The paper highlights a notable result: humans perform very poorly on temporal comparison in this benchmark, and some large VLMs outperform humans. The stated explanation is that these models can likely leverage background knowledge from pretraining. On that interpretation, CompareTemporalBench is not only a perception test but also a test of knowledge-augmented temporal reasoning.
At the same time, the abstract of CompareBench states that even the strongest models consistently fail at temporal ordering and spatial relations. HistCaps thus underwrites a benchmark slice that is difficult even for top-performing systems, despite the apparent advantage that large pretrained models may derive from historical world knowledge.
6. Significance, interpretation, and limits
HistCaps matters because it enables a real-world, historically grounded benchmark for temporal comparison reasoning, one of the four core dimensions of visual comparison reasoning identified by CompareBench (Cai et al., 25 Sep 2025). Its significance lies in shifting evaluation away from object recognition or caption matching and toward chronological discrimination across scenes.
Several clarifications follow directly from the source material. First, HistCaps is not itself a QA benchmark; it is an annotated historical-image dataset whose entries are later assembled into a QA-style multiple-choice task. Second, the temporal task is not purely perceptual; it requires combining visible cues with world knowledge. Third, high model performance relative to humans on CompareTemporalBench should not be read as evidence that the task is easy. The paper instead presents this pattern as evidence that the task probes historical knowledge acquired during pretraining.
The dataset also has explicit limits in the published description. The paper does not provide per-category statistics for HistCaps beyond its size of 515 samples. Nor does it describe a standalone evaluation protocol for HistCaps independent of CompareTemporalBench. A plausible implication is that HistCaps was designed primarily as an enabling resource for benchmark construction rather than as a self-contained benchmark.
In the architecture of CompareBench, HistCaps therefore occupies a specific and consequential niche: it operationalizes temporal comparison by pairing historical imagery with bilingual event-level annotation, then using that structure to test whether VLMs can infer which of several scenes occurred earliest in history. The paper’s broader conclusion is that this capability remains a systematic blind spot for current VLMs, and HistCaps is the resource that makes that blind spot measurable in the temporal domain.