---
title: 'VIR-Bench: Dual Domain Benchmarks'
url: https://www.emergentmind.com/topics/vir-bench
type: topic
---

# VIR-Bench: Dual Domain Benchmarks

“VIR-Bench” is a reused benchmark name applied to two unrelated research artifacts. In computer architecture, it denotes the **RISC-V Vectorized Benchmark Suite**, a seven-application collection designed for micro-architectural evaluation of vector machines using a gem5-based RISC-V vector architecture model [2111.01949]. In multimodal video understanding, it denotes a benchmark for **travel video itinerary reconstruction**, built from 200 Japan travel videos to evaluate geospatial and temporal reasoning in multimodal large language models (MLLMs) [2509.19002]. The shared label therefore refers not to a single benchmark lineage, but to two distinct benchmarking programs with different objects of study, data modalities, and evaluation protocols.

## 1. Name, scope, and disambiguation

The two uses of the name differ at the level of domain, benchmark target, and methodology.

| VIR-Bench variant | Research domain | Primary evaluation target |
|---|---|---|
| RISC-V Vectorized Benchmark Suite | Computer architecture | Vector microarchitectures |
| Travel video itinerary benchmark | Multimodal video understanding | Geospatial-temporal reasoning in MLLMs |

The architectural VIR-Bench was created because researchers lacked both a flexible, customizable simulator for vector-architecture design and a benchmark suite that stresses the relevant subsystems of a vector machine across different vector-length regimes [2111.01949]. The video-oriented VIR-Bench was created because existing video benchmarks focus largely on indoor scenes or short-range outdoor activities and do not test whether models can reconstruct extended journeys across multiple locations and timescales [2509.19002].

The name can also be confused with similarly titled viral benchmark resources. **ViroGym** is a benchmark for evaluating protein language models on viral proteins, including mutational effect prediction, antigenic diversity prediction, and pandemic prediction [2603.06740]. **ViroBench** is a benchmark for nucleotide foundation models on viral genomics tasks, organized around biological understanding and latent biosecurity risk [2605.25388]. These are separate resources and are not variants of VIR-Bench.

## 2. VIR-Bench as the RISC-V Vectorized Benchmark Suite

In the architectural usage, VIR-Bench is a **collection composed of seven data-parallel applications from different domains**, intended to benchmark vector microarchitectures jointly with a new **gem5-based RISC-V vector architecture model** [2111.01949]. The benchmark and simulator are presented together because they are meant to be used jointly: the paper extends gem5 with a **parameterizable decoupled vector architecture** that supports **RISC-V V-extension instructions**, and the suite is written in a **vector-length agnostic** style using **RISC-V intrinsics**.

This coupling to the simulator is central. The model is designed around the flexibility of the RISC-V vector extension, especially its **unbounded MVL concept**, and can be configured across vector lengths ranging roughly from **512-bit** to **16,384-bit**. The model is **decoupled**, so scalar execution and vector execution are separated; vector memory operations are buffered separately from vector arithmetic operations; vector instructions are renamed and queued before issue; and the vector unit can tolerate memory latency better than a tightly coupled SIMD pipeline. The simulator is configurable in the number of physical vector registers, MVL, issue scheme, number of queues, number of lanes, functional-unit latencies, lane-interconnect topology, number of memory ports, and MSHR-related behavior.

The stated purpose of the suite is diagnostic rather than merely comparative. It is intended to expose how a vector architecture behaves under different application patterns and vector-length regimes, including **embedded-style designs**, which mainly target **short vectors**, and **HPC-style designs**, which are usually designed for **large vectors**. This framing makes the suite a tool for design-space exploration rather than a fixed leaderboard benchmark.

## 3. Composition, workload taxonomy, and findings of the vector-architecture suite

The suite contains seven applications: **Blackscholes**, **Canneal**, **Jacobi-2D**, **Particle Filter**, **Pathfinder**, **Streamcluster**, and **Swaptions** [2111.01949]. These were selected using three criteria: coverage of different application domains, coverage of different data-level parallelism patterns, and coverage of most of the vector ISA through the set as a whole.

| Application | DLP pattern | Main stressed subsystem |
|---|---|---|
| Blackscholes | Regular | Vector arithmetic lanes |
| Canneal | Irregular | Vector memory unit |
| Jacobi-2D | Regular | Lanes, interconnect, memory |
| Particle Filter | Mix | Masks, scalar-vector communication |
| Pathfinder | Regular | Lane interconnection network |
| Streamcluster | Mix | Vector memory subsystem |
| Swaptions | Regular | Arithmetic, memory, MVL/cache scaling |

The reported baseline configuration uses a **64-bit RISC-V superscalar in-order pipeline** at **2 GHz** for the scalar core and **1 GHz** for the vector engine, with vector lengths of **512-bit**, **1024-bit**, **2048-bit**, **4096-bit**, **8192-bit**, and **16384-bit**. Other reported details include **40 physical registers**, VRF sizes from **2.5 KB** to **80 KB**, **1 pipelined arithmetic unit per lane**, a **512-bit memory interface**, **ring topology** interconnect, **32 KB** L1I and L1D caches, a **256 KB** L2, and **2 GB DDR3** memory.

The empirical findings emphasize that no single vector configuration is universally optimal. **Regular, high-DLP applications** benefit most from larger MVL and more lanes, while **irregular or mixed-DLP applications** are often limited by indexed memory latency, scalar-core communication, tail handling, spill code, vector move instructions, or reduction and mask overhead. **Large MVL is not always better**: for some short-vector workloads, a very large MVL reduces performance. Canneal is the clearest example of an embedded-style preference, with best performance around **MVL=16** and scalar outperforming the vectorized version at **MVL=128** and **256** because of spill code, vector move instructions, tail zeroing, and growing vector-operation overhead. By contrast, Jacobi-2D and Swaptions illustrate scenarios in which larger vectors can be effective, although Swaptions also shows that **optimal MVL depends on cache configuration**.

The paper’s conclusion is that VIR-Bench is not just a set of programs but a way to identify what part of a vector machine is limiting performance. This suggests that its primary value lies in architectural diagnosis: lane count, VRF size and ports, memory hierarchy, interconnect topology, and scalar-vector synchronization can be studied under workloads that expose different failure modes.

## 4. VIR-Bench as a travel video itinerary reconstruction benchmark

In the multimodal usage, VIR-Bench is a benchmark for evaluating whether MLLMs can understand long-horizon geospatial and temporal structure in travel videos by reconstructing itineraries [2509.19002]. Its central motivation is that current video benchmarks emphasize indoor scenes, short clips, or local movement, whereas practical applications such as embodied-AI planning, navigation, and autonomous travel assistance require reasoning over extended journeys across multiple locations and timescales.

The benchmark consists of **200 travel videos** filmed across Japan, paired with manually annotated **visiting order graphs**. The dataset is balanced by language, with **100 English-narrated** and **100 Japanese-narrated** videos. It contains **3,689 POIs** and covers **43 of Japan’s 47 prefectures**. Videos span **1 to 5 unique prefectures**, with a mean of about **1.5**, and most videos cover **1–2 prefectures**. The average video duration is about **1,060 seconds**, the average POI duration is about **50.7 seconds**, and the POI count per video ranges from **7 to 75**, with a mean of **18.4**.

The core annotation object is a hierarchical directed graph with four node levels: **root**, **prefecture**, **city**, and **POI**. It contains two principal edge types. **Inclusion edges** encode containment relations such as prefecture → city and city → POI, while **transition edges** encode chronological movement between distinct locations at the same hierarchical level. A special **“Overlap”** edge exists for overlapping POIs but is omitted from the edge-prediction task because such cases are rare. Repeated visits are represented as distinct nodes to avoid cycles.

Construction of the dataset is explicitly human-centric. Ten Japan-based annotators each collected **20 YouTube travel vlogs** and annotated all visited POIs with timestamps and Google Maps URLs. A visit was defined as a location clearly appearing in the video and being clearly visited by the creator. When a venue could not be uniquely identified, annotators used an **UNKNOWN** placeholder plus a category. The authors then queried the Google Places API for detailed metadata and manually checked and corrected errors. The paper states that annotation involved **GeoGuessr-like inference from visual cues**, which makes the benchmark partly a test of geographic recognition from sparse and noisy multimodal evidence.

## 5. Task decomposition, evaluation protocol, and empirical difficulty in the video benchmark

The benchmark frames itinerary reconstruction as a graph problem but decomposes it into two subtasks because end-to-end graph generation was found to be too difficult for current models [2509.19002]. The first is **node prediction**, in which the model must output all visited prefectures, cities, and POIs, with POIs also labeled by category. The second is **edge prediction**, in which the model receives the video and the gold set of visited locations in shuffled order and must recover all inclusion and transition edges. Outputs are required in JSON format.

Evaluation uses macro-averaged **precision**, **recall**, and **F1**, reported separately for each node and edge type and also as an overall weighted average. Prefecture and city nodes are scored by exact string match. POIs are scored with a lightweight sequence-matching rule: a similarity score above **0.7** is accepted as correct; a score above **0.5** is also accepted if the predicted POI category matches the gold category. Inclusion and transition edges are correct only if the entire tuple exactly matches the gold tuple.

The authors evaluate open-weight models including **VideoLLaMA3-7B**, **LLaVA-Video-7B/72B**, **InternVL3-8B/38B/78B**, and **Qwen2.5-VL-7B/32B/72B**, as well as proprietary models **GPT-4.1**, **o4-mini**, **Gemini-2.5-Flash**, and **Gemini-2.5-Pro**. The results indicate that the benchmark is difficult for current MLLMs. On **node prediction**, the best open model is **Qwen2.5-VL-72B** with overall **F1 = 38.1**, while **Gemini-2.5-Pro** and **GPT-4.1** reach **57.4** and **57.0** respectively. On **edge prediction**, **Qwen2.5-VL-72B** reaches **52.4** overall F1, while **Gemini-2.5-Pro** reaches **80.7** and **o4-mini** **64.9**.

The hardest category is **transition edge prediction**. Several open models are near random on this task, with **VideoLLaMA3-7B** at **1.4 F1**, **LLaVA-Video-7B** at **1.1**, and **Qwen2.5-VL-7B** at **1.3**. Even stronger open models remain low, whereas proprietary reasoning-capable models improve much more substantially. The paper further reports that “thinking” at inference helps mainly for edge prediction rather than node prediction, and that audio materially improves temporal reconstruction for Gemini models: removing audio from **Gemini-2.5-Flash** reduces transition-edge F1 from **41.7** to **22.3**. Another ablation shows that increasing frames from **64** to **128** to **256** improves **GPT-4.1**, especially for POI and transition-edge prediction.

These findings support the paper’s claim that the benchmark probes three distinct capabilities: **geographic knowledge**, **temporal reasoning**, and **long-context holistic itinerary understanding**. Inclusion edges are substantially easier than transition edges, which suggests that hierarchical containment is easier than reconstructing travel order across a full video trajectory.

## 6. Benchmark philosophy, significance, and recurring misconceptions

Despite the shared name, the two VIR-Bench resources instantiate different benchmark philosophies. The RISC-V Vectorized Benchmark Suite is a **micro-architectural stress suite** designed to expose bottlenecks in vector lanes, memory systems, mask handling, reductions, and scalar-vector interaction [2111.01949]. The travel-video VIR-Bench is a **reasoning benchmark** designed to test whether MLLMs can integrate geographically grounded recognition with temporal sequence reconstruction over long video contexts [2509.19002].

A common misconception would be to treat either resource as a generic “benchmark suite” without regard to its target abstraction level. In the architectural setting, VIR-Bench is not a compiler benchmark, an application throughput suite, or a pure ISA-conformance test; it is tied to a **gem5-based vector architecture model** and is meant to support design-space studies across **short-, medium-, and large-vector** configurations. In the video setting, VIR-Bench is not simply a place-recognition dataset; the paper explicitly distinguishes **node prediction** from **edge prediction**, and the hardest problem is not naming locations but reconstructing coherent **geospatial-temporal trajectories**.

The practical implications are similarly different. In the vector-architecture paper, the benchmark is presented as a way to study how lane count, MVL, cache capacity, and scalar-vector synchronization affect performance, with the broader conclusion that there is no single best vector configuration. In the video paper, the benchmark is connected to downstream travel-planning systems through a prototype multi-agent planner built around **Gemini-2.5-Pro**; combining both **POI list and video** yields the best travel plans, with an attractiveness score of **3.73 out of 5**, compared with **3.58** for POI-only and **3.46** for video-only. This suggests that the itinerary representation learned through benchmarking can serve as an intermediate structure for real planning tasks.

Taken together, the two works show that “VIR-Bench” has become a label for highly diagnostic benchmarks rather than broad aggregate leaderboards. In one case the diagnosis concerns vector-machine bottlenecks; in the other it concerns failures of MLLMs to connect locations into a coherent journey across space and time.

Source: https://www.emergentmind.com/topics/vir-bench