HOIVG-Bench: Multimodal HOI Video Benchmark

• HOIVG-Bench is a benchmark for multimodal HOI video generation that jointly uses text, human and object images, audio, and pose for realistic synthesis.
• It comprises 135 curated, standardized 5-second, 720p clips and evaluates models on text alignment, reference consistency, synchronization, and pose accuracy.
• The benchmark supports detailed ablation studies on unified conditioning methods, guiding improvements in end-to-end models like OmniShow.

HOIVG-Bench is a benchmark for Human-Object Interaction Video Generation (HOIVG), introduced alongside "OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation" (Zhou et al., 13 Apr 2026). It is designed to evaluate generation of human-object interaction videos conditioned jointly on text, a human reference image, an object reference image, audio, and pose. In the supplied literature, its defining role is to measure the full multimodal composition problem rather than partial control settings such as text+pose or text+image alone. The benchmark contains 135 carefully curated samples, and all quantitative and qualitative comparisons are standardized to 5-second, 720p, portrait-mode clips (Zhou et al., 13 Apr 2026).

1. Problem formulation and motivation

HOIVG-Bench was created to address what the OmniShow paper describes as a lack of a dedicated and comprehensive benchmark for HOIVG (Zhou et al., 13 Apr 2026). In that formulation, HOIVG denotes generation of realistic human-object interaction videos conditioned on five inputs: text, a human reference image, an object reference image, audio, and pose. The benchmark therefore targets the combined use of modalities that play distinct roles: text provides global semantics, the human image constrains identity, the object image constrains object appearance, audio constrains synchronization, and pose constrains motion and interaction structure.

The motivation is explicitly comparative. Existing resources are described as focusing on limited control settings, including text + pose, text + images, or single-modality customization tasks, and thus as failing to evaluate the unified multimodal setting required by HOIVG (Zhou et al., 13 Apr 2026). HOIVG-Bench is intended to supply that missing evaluation layer. The benchmark is therefore not merely a video-quality suite; it is structured around the question of whether a model can jointly preserve identity, object appearance, motion structure, and audio-visual synchronization in a coherent interaction sequence.

The application domain is also specified. The benchmark is oriented toward realistic HOI scenarios relevant to e-commerce demonstrations, short video production, and interactive entertainment. This practical framing is important because it distinguishes the benchmark from narrowly constrained academic setups that test only one or two conditioning channels.

2. Benchmark composition and curation pipeline

HOIVG-Bench contains 135 carefully curated samples (Zhou et al., 13 Apr 2026). Each sample includes five modalities: a text caption, a human reference image, an object reference image, audio, and a pose sequence. The source videos are selected from an in-house video library, with three stated selection criteria: duration greater than 4 seconds, clear human-object interactions, and diversity in human attributes and object categories.

The curation pipeline is multimodal by construction. The object reference image is not obtained by simply cropping the source video. Instead, the pipeline uses Nano Banana to modify object textures and colors and to add detailed appearance, with the stated goal of producing higher-quality object reference images that better resemble real-world generation inputs. The human reference image is also generated from video screenshots using Nano Banana. The paper states that this is done for privacy and de-identification, while preserving stylistic similarity to the original subject and changing identity features (Zhou et al., 13 Apr 2026).

Pose supervision is extracted per frame from the original videos using DWPose, yielding the motion-control signal used in evaluation. Audio is created in a separate two-stage process: GPT-4o generates a speech script focused on describing the target object, and GPT-4o also analyzes the human reference image’s gender and age, after which ElevenLabs synthesizes speech with matching timbre. The paper emphasizes that the audio is semantically consistent with the benchmark scenario rather than simply reused from the original clip.

The benchmark is also curated for diversity. The selected videos are required to cover human attributes such as gender, age, and ethnicity; object categories including daily necessities, tools, and other objects; and motion dynamics spanning a broad range of pose intensities. The paper further notes that manual checks are used to filter out images with noticeable “AI-ness,” which functions as a quality-control step for the publicly released reference images (Zhou et al., 13 Apr 2026).

3. Evaluation protocol and task settings

A central property of HOIVG-Bench is evaluation standardization. Although OmniShow can generate videos up to 10 seconds, the benchmark fixes comparison to 5-second clips, 720p resolution, and portrait mode for fairness across methods (Zhou et al., 13 Apr 2026). This creates a controlled evaluation regime in which differences are less likely to be confounded by clip length or output format.

Automatic evaluation is organized into five dimensions. Text alignment is measured by TA using VideoReward. Reference consistency is measured by FaceSim and NexusScore based on OpenS2V. Pose accuracy is measured by AKD and PCK computed from DWPose, with PCK using a 5% error threshold. Audio-visual synchronization is measured by Sync-C and Sync-D using a SyncNet-style protocol following Chung and Zisserman (2016). Video quality is measured by AES and IQA from VBench, together with VQ and MQ from VideoReward (Zhou et al., 13 Apr 2026).

The benchmark is evaluated in several conditioning regimes because existing baselines do not support the same input sets:

Setting	Conditions	Compared methods
R2V	Text + Reference image	HunyuanCustom, HuMo-1.7B, HuMo-17B, VACE, Phantom-1.3B, Phantom-14B, OmniShow
RA2V	Text + Reference + Audio	HunyuanCustom, HuMo-1.7B, HuMo-17B, OmniShow
RP2V	Text + Reference + Pose	AnchorCrafter, VACE, OmniShow
RAP2V	Reference + Audio + Pose to Video	OmniShow; compared separately with a cascaded baseline, VACE + LatentSync

This setting structure is integral to the benchmark design. It makes explicit which subsets of the full HOIVG problem current models can handle, and it reserves the fully multimodal setting, RAP2V, for models that actually support simultaneous reference, audio, and pose conditioning.

4. Benchmark use in comparative evaluation

HOIVG-Bench is the main testbed through which the OmniShow paper reports comparative performance across multimodal generation settings (Zhou et al., 13 Apr 2026). In the R2V setting, OmniShow is reported as best on NexusScore (0.389), AES (0.468), VQ (11.12), and MQ (5.885), and near-best on FaceSim (0.874), slightly below Phantom-14B’s 0.876. The paper notes that OmniShow is not always the strongest on text alignment in this setting, since VACE and Phantom-14B score higher on TA.

In the RA2V setting, OmniShow is reported as best or near-best on most metrics, including Sync-C = 8.612, Sync-D = 7.608, NexusScore = 0.369, AES = 0.465, IQA = 0.742, VQ = 10.86, and MQ = 5.554. The main significance of this result within the benchmark is that audio-conditioned HOIVG is evaluated not only by visual fidelity but also by synchronization-specific metrics.

In the RP2V setting, OmniShow achieves NexusScore = 0.418, PCK = 0.460, IQA = 0.722, VQ = 10.28, and MQ = 4.937, while VACE remains strong on some metrics such as TA and FaceSim. This split is useful because it isolates pose adherence from audio synchronization and reveals how models behave when motion control is present without audio conditioning.

The RAP2V setting is structurally distinctive because the paper states that OmniShow is the only model supporting RAP2V generation. Direct benchmarking against alternative unified models is therefore unavailable, and the comparison is instead made against a cascaded baseline, VACE + LatentSync. On HOIVG-Bench, OmniShow outperforms that baseline on all listed metrics, including TA: 7.134 vs 6.885, FaceSim: 0.645 vs 0.591, NexusScore: 0.353 vs 0.341, Sync-C: 7.699 vs 7.016, AKD: 0.172 vs 0.198, PCK: 0.478 vs 0.340, VQ: 11.06 vs 10.05, and MQ: 5.880 vs 3.911 (Zhou et al., 13 Apr 2026). Within the paper’s framing, these results support the claim that end-to-end multimodal unification is preferable to a sequential pipeline for the full HOIVG setting.

Human evaluation is also reported for HOIVG-Bench-related comparisons. The study uses side-by-side preference judgments with 30 participants for RA2V, 33 participants for RP2V, and 20 randomly selected samples, with human evaluators preferring OmniShow in the majority of cases (Zhou et al., 13 Apr 2026).

5. Analytical and diagnostic role

HOIVG-Bench is used not only for headline comparison but also for ablation and diagnostic analysis (Zhou et al., 13 Apr 2026). One set of experiments studies Unified Channel-wise Conditioning against token concatenation and a variant without reference reconstruction loss. The reported outcome is improvement in FaceSim, NexusScore, and AES, indicating that benchmark metrics are sensitive to how reference conditioning is injected into the generator.

Another group of ablations examines Gated Local-Context Attention. The reported variants are without audio context, without attention-map constraints, without adaptive gating, and the full method. The paper states that audio context improves temporal coherence, attention-map constraints are crucial for synchronization, and adaptive gating helps training stability and improves quality. Because HOIVG-Bench includes both synchronization and quality metrics, it can expose these trade-offs directly.

The benchmark is also used to assess the Decoupled-Then-Joint Training strategy. Compared variants include single-stage training on RA2V only, R2V → RA2V, A2V → RA2V, and the proposed strategy. The paper reports that the proposed procedure gives the best trade-off between reference consistency, audio-visual synchronization, and quality. Additional analyses include a RoPE strategy ablation for pseudo-frames and a context window size ablation for audio packing.

This pattern of use suggests that HOIVG-Bench functions as a structured analysis environment as well as an evaluation leaderboard. Its metric suite is broad enough to reveal distinct failure modes in semantics, identity preservation, pose adherence, synchronization, and overall visual quality.

6. Limitations, biases, and position within HOI benchmarking

The OmniShow paper explicitly identifies several limitations of HOIVG-Bench (Zhou et al., 13 Apr 2026). First, the human reference images are AI-generated rather than real photos, and the paper acknowledges that this may introduce a slight distribution bias relative to fully real-world images. Second, although the curation pipeline includes manual filtering to remove images with obvious AI artifacts, the benchmark may still not perfectly match natural image distributions. Third, because the benchmark is built from an in-house video library and carefully curated synthetic reference assets, it may not capture the full open-world diversity of unconstrained internet data. Fourth, benchmark evaluation is restricted to 5-second clips, even though the underlying model can generate up to 10 seconds, so longer-horizon behavior is not tested. Finally, the paper notes that in extreme scenarios with overly intense motion or conflicting multimodal inputs, the model may produce artifacts or blur.

Within the broader literature supplied here, HOIVG-Bench occupies a distinct niche. Ego-HOIBench targets egocentric HOI detection from first-person images using explicit hand-verb-object triplets and localization (Deng et al., 17 Jun 2025). RoHOI is a corruption-oriented robustness benchmark for HOI detection built on HICO-DET and V-COCO (Wen et al., 12 Jul 2025). The benchmark introduced in "Rethinking Human-Object Interaction Evaluation for both Vision-LLMs and HOI-Specific Methods" reformulates static-image HOI evaluation as a multiple-answer, multiple-choice task for standalone VLMs and HOI-specific detectors (Lei et al., 26 Aug 2025). By contrast, HOIVG-Bench is specifically concerned with generation, not detection or recognition, and with joint multimodal conditioning, not single-image labeling or robustness under corruption.

In that sense, HOIVG-Bench marks a shift in HOI benchmarking from recognizing interactions in images to synthesizing interaction videos under coordinated semantic, visual, auditory, and kinematic control. Its scope is narrower than open-world video evaluation but more specialized with respect to multimodal human-object interaction synthesis.