VIR-Bench: Dual Domain Benchmarks

• The paper introduces a RISC-V vectorized benchmark suite with a gem5-based simulator to diagnose vector microarchitecture performance across diverse application patterns.
• VIR-Bench is a multimodal benchmark using 200 Japan travel videos to evaluate MLLMs’ ability to reconstruct accurate geospatial-temporal itineraries.
• Both benchmarks exemplify diagnostic evaluation: one targets architectural bottlenecks and the other probes long-horizon reasoning in multimodal contexts.

“VIR-Bench” is a reused benchmark name applied to two unrelated research artifacts. In computer architecture, it denotes the RISC-V Vectorized Benchmark Suite, a seven-application collection designed for micro-architectural evaluation of vector machines using a gem5-based RISC-V vector architecture model (Lazo et al., 2021). In multimodal video understanding, it denotes a benchmark for travel video itinerary reconstruction, built from 200 Japan travel videos to evaluate geospatial and temporal reasoning in multimodal LLMs (MLLMs) (Wang et al., 23 Sep 2025). The shared label therefore refers not to a single benchmark lineage, but to two distinct benchmarking programs with different objects of study, data modalities, and evaluation protocols.

1. Name, scope, and disambiguation

The two uses of the name differ at the level of domain, benchmark target, and methodology.

VIR-Bench variant	Research domain	Primary evaluation target
RISC-V Vectorized Benchmark Suite	Computer architecture	Vector microarchitectures
Travel video itinerary benchmark	Multimodal video understanding	Geospatial-temporal reasoning in MLLMs

The architectural VIR-Bench was created because researchers lacked both a flexible, customizable simulator for vector-architecture design and a benchmark suite that stresses the relevant subsystems of a vector machine across different vector-length regimes (Lazo et al., 2021). The video-oriented VIR-Bench was created because existing video benchmarks focus largely on indoor scenes or short-range outdoor activities and do not test whether models can reconstruct extended journeys across multiple locations and timescales (Wang et al., 23 Sep 2025).

The name can also be confused with similarly titled viral benchmark resources. ViroGym is a benchmark for evaluating protein LLMs on viral proteins, including mutational effect prediction, antigenic diversity prediction, and pandemic prediction (Zhou et al., 6 Mar 2026). ViroBench is a benchmark for nucleotide foundation models on viral genomics tasks, organized around biological understanding and latent biosecurity risk (Ye et al., 25 May 2026). These are separate resources and are not variants of VIR-Bench.

2. VIR-Bench as the RISC-V Vectorized Benchmark Suite

In the architectural usage, VIR-Bench is a collection composed of seven data-parallel applications from different domains, intended to benchmark vector microarchitectures jointly with a new gem5-based RISC-V vector architecture model (Lazo et al., 2021). The benchmark and simulator are presented together because they are meant to be used jointly: the paper extends gem5 with a parameterizable decoupled vector architecture that supports RISC-V V-extension instructions, and the suite is written in a vector-length agnostic style using RISC-V intrinsics.

This coupling to the simulator is central. The model is designed around the flexibility of the RISC-V vector extension, especially its unbounded MVL concept, and can be configured across vector lengths ranging roughly from 512-bit to 16,384-bit. The model is decoupled, so scalar execution and vector execution are separated; vector memory operations are buffered separately from vector arithmetic operations; vector instructions are renamed and queued before issue; and the vector unit can tolerate memory latency better than a tightly coupled SIMD pipeline. The simulator is configurable in the number of physical vector registers, MVL, issue scheme, number of queues, number of lanes, functional-unit latencies, lane-interconnect topology, number of memory ports, and MSHR-related behavior.

The stated purpose of the suite is diagnostic rather than merely comparative. It is intended to expose how a vector architecture behaves under different application patterns and vector-length regimes, including embedded-style designs, which mainly target short vectors, and HPC-style designs, which are usually designed for large vectors. This framing makes the suite a tool for design-space exploration rather than a fixed leaderboard benchmark.

3. Composition, workload taxonomy, and findings of the vector-architecture suite

The suite contains seven applications: Blackscholes, Canneal, Jacobi-2D, Particle Filter, Pathfinder, Streamcluster, and Swaptions (Lazo et al., 2021). These were selected using three criteria: coverage of different application domains, coverage of different data-level parallelism patterns, and coverage of most of the vector ISA through the set as a whole.

Application	DLP pattern	Main stressed subsystem
Blackscholes	Regular	Vector arithmetic lanes
Canneal	Irregular	Vector memory unit
Jacobi-2D	Regular	Lanes, interconnect, memory
Particle Filter	Mix	Masks, scalar-vector communication
Pathfinder	Regular	Lane interconnection network
Streamcluster	Mix	Vector memory subsystem
Swaptions	Regular	Arithmetic, memory, MVL/cache scaling

The reported baseline configuration uses a 64-bit RISC-V superscalar in-order pipeline at 2 GHz for the scalar core and 1 GHz for the vector engine, with vector lengths of 512-bit, 1024-bit, 2048-bit, 4096-bit, 8192-bit, and 16384-bit. Other reported details include 40 physical registers, VRF sizes from 2.5 KB to 80 KB, 1 pipelined arithmetic unit per lane, a 512-bit memory interface, ring topology interconnect, 32 KB L1I and L1D caches, a 256 KB L2, and 2 GB DDR3 memory.

The empirical findings emphasize that no single vector configuration is universally optimal. Regular, high-DLP applications benefit most from larger MVL and more lanes, while irregular or mixed-DLP applications are often limited by indexed memory latency, scalar-core communication, tail handling, spill code, vector move instructions, or reduction and mask overhead. Large MVL is not always better: for some short-vector workloads, a very large MVL reduces performance. Canneal is the clearest example of an embedded-style preference, with best performance around MVL=16 and scalar outperforming the vectorized version at MVL=128 and 256 because of spill code, vector move instructions, tail zeroing, and growing vector-operation overhead. By contrast, Jacobi-2D and Swaptions illustrate scenarios in which larger vectors can be effective, although Swaptions also shows that optimal MVL depends on cache configuration.

The paper’s conclusion is that VIR-Bench is not just a set of programs but a way to identify what part of a vector machine is limiting performance. This suggests that its primary value lies in architectural diagnosis: lane count, VRF size and ports, memory hierarchy, interconnect topology, and scalar-vector synchronization can be studied under workloads that expose different failure modes.

4. VIR-Bench as a travel video itinerary reconstruction benchmark

In the multimodal usage, VIR-Bench is a benchmark for evaluating whether MLLMs can understand long-horizon geospatial and temporal structure in travel videos by reconstructing itineraries (Wang et al., 23 Sep 2025). Its central motivation is that current video benchmarks emphasize indoor scenes, short clips, or local movement, whereas practical applications such as embodied-AI planning, navigation, and autonomous travel assistance require reasoning over extended journeys across multiple locations and timescales.

The benchmark consists of 200 travel videos filmed across Japan, paired with manually annotated visiting order graphs. The dataset is balanced by language, with 100 English-narrated and 100 Japanese-narrated videos. It contains 3,689 POIs and covers 43 of Japan’s 47 prefectures. Videos span 1 to 5 unique prefectures, with a mean of about 1.5, and most videos cover 1–2 prefectures. The average video duration is about 1,060 seconds, the average POI duration is about 50.7 seconds, and the POI count per video ranges from 7 to 75, with a mean of 18.4.

The core annotation object is a hierarchical directed graph with four node levels: root, prefecture, city, and POI. It contains two principal edge types. Inclusion edges encode containment relations such as prefecture → city and city → POI, while transition edges encode chronological movement between distinct locations at the same hierarchical level. A special “Overlap” edge exists for overlapping POIs but is omitted from the edge-prediction task because such cases are rare. Repeated visits are represented as distinct nodes to avoid cycles.

Construction of the dataset is explicitly human-centric. Ten Japan-based annotators each collected 20 YouTube travel vlogs and annotated all visited POIs with timestamps and Google Maps URLs. A visit was defined as a location clearly appearing in the video and being clearly visited by the creator. When a venue could not be uniquely identified, annotators used an UNKNOWN placeholder plus a category. The authors then queried the Google Places API for detailed metadata and manually checked and corrected errors. The paper states that annotation involved GeoGuessr-like inference from visual cues, which makes the benchmark partly a test of geographic recognition from sparse and noisy multimodal evidence.

5. Task decomposition, evaluation protocol, and empirical difficulty in the video benchmark

The benchmark frames itinerary reconstruction as a graph problem but decomposes it into two subtasks because end-to-end graph generation was found to be too difficult for current models (Wang et al., 23 Sep 2025). The first is node prediction, in which the model must output all visited prefectures, cities, and POIs, with POIs also labeled by category. The second is edge prediction, in which the model receives the video and the gold set of visited locations in shuffled order and must recover all inclusion and transition edges. Outputs are required in JSON format.

Evaluation uses macro-averaged precision, recall, and F1, reported separately for each node and edge type and also as an overall weighted average. Prefecture and city nodes are scored by exact string match. POIs are scored with a lightweight sequence-matching rule: a similarity score above 0.7 is accepted as correct; a score above 0.5 is also accepted if the predicted POI category matches the gold category. Inclusion and transition edges are correct only if the entire tuple exactly matches the gold tuple.

The authors evaluate open-weight models including VideoLLaMA3-7B, LLaVA-Video-7B/72B, InternVL3-8B/38B/78B, and Qwen2.5-VL-7B/32B/72B, as well as proprietary models GPT-4.1, o4-mini, Gemini-2.5-Flash, and Gemini-2.5-Pro. The results indicate that the benchmark is difficult for current MLLMs. On node prediction, the best open model is Qwen2.5-VL-72B with overall F1 = 38.1, while Gemini-2.5-Pro and GPT-4.1 reach 57.4 and 57.0 respectively. On edge prediction, Qwen2.5-VL-72B reaches 52.4 overall F1, while Gemini-2.5-Pro reaches 80.7 and o4-mini 64.9.

The hardest category is transition edge prediction. Several open models are near random on this task, with VideoLLaMA3-7B at 1.4 F1, LLaVA-Video-7B at 1.1, and Qwen2.5-VL-7B at 1.3. Even stronger open models remain low, whereas proprietary reasoning-capable models improve much more substantially. The paper further reports that “thinking” at inference helps mainly for edge prediction rather than node prediction, and that audio materially improves temporal reconstruction for Gemini models: removing audio from Gemini-2.5-Flash reduces transition-edge F1 from 41.7 to 22.3. Another ablation shows that increasing frames from 64 to 128 to 256 improves GPT-4.1, especially for POI and transition-edge prediction.

These findings support the paper’s claim that the benchmark probes three distinct capabilities: geographic knowledge, temporal reasoning, and long-context holistic itinerary understanding. Inclusion edges are substantially easier than transition edges, which suggests that hierarchical containment is easier than reconstructing travel order across a full video trajectory.

6. Benchmark philosophy, significance, and recurring misconceptions

Despite the shared name, the two VIR-Bench resources instantiate different benchmark philosophies. The RISC-V Vectorized Benchmark Suite is a micro-architectural stress suite designed to expose bottlenecks in vector lanes, memory systems, mask handling, reductions, and scalar-vector interaction (Lazo et al., 2021). The travel-video VIR-Bench is a reasoning benchmark designed to test whether MLLMs can integrate geographically grounded recognition with temporal sequence reconstruction over long video contexts (Wang et al., 23 Sep 2025).

A common misconception would be to treat either resource as a generic “benchmark suite” without regard to its target abstraction level. In the architectural setting, VIR-Bench is not a compiler benchmark, an application throughput suite, or a pure ISA-conformance test; it is tied to a gem5-based vector architecture model and is meant to support design-space studies across short-, medium-, and large-vector configurations. In the video setting, VIR-Bench is not simply a place-recognition dataset; the paper explicitly distinguishes node prediction from edge prediction, and the hardest problem is not naming locations but reconstructing coherent geospatial-temporal trajectories.

The practical implications are similarly different. In the vector-architecture paper, the benchmark is presented as a way to study how lane count, MVL, cache capacity, and scalar-vector synchronization affect performance, with the broader conclusion that there is no single best vector configuration. In the video paper, the benchmark is connected to downstream travel-planning systems through a prototype multi-agent planner built around Gemini-2.5-Pro; combining both POI list and video yields the best travel plans, with an attractiveness score of 3.73 out of 5, compared with 3.58 for POI-only and 3.46 for video-only. This suggests that the itinerary representation learned through benchmarking can serve as an intermediate structure for real planning tasks.

Taken together, the two works show that “VIR-Bench” has become a label for highly diagnostic benchmarks rather than broad aggregate leaderboards. In one case the diagnosis concerns vector-machine bottlenecks; in the other it concerns failures of MLLMs to connect locations into a coherent journey across space and time.