OmniShow: HOIVG Multimodal Video Generation

• OmniShow is a specialized framework for Human-Object Interaction Video Generation that integrates text, reference images, audio, and pose for joint controllability.
• It extends a 12B MMDiT-based Waver 1.0 backbone with Unified Channel-wise Conditioning and Gated Local-Context Attention to ensure identity preservation and audio-visual synchronization.
• The system employs a decoupled-then-joint training strategy with HOIVG-Bench evaluation to overcome data scarcity and achieve high-fidelity video synthesis.

Searching arXiv for the primary paper and closely related context. OmniShow is an end-to-end framework for Human-Object Interaction Video Generation (HOIVG), a task defined as synthesizing high-quality videos of humans interacting with objects under multimodal conditioning by text, reference images, audio, and pose. The system is built on Waver 1.0, a 12B MMDiT-based video generation backbone, and extends it with Unified Channel-wise Conditioning, Gated Local-Context Attention, and Decoupled-Then-Joint Training. It is accompanied by HOIVG-Bench, a benchmark intended to evaluate alignment, identity preservation, pose accuracy, audio-visual synchronization, and video quality in this setting (Zhou et al., 13 Apr 2026).

1. Task definition and problem setting

The paper defines HOIVG as a multimodal video-generation problem in which a model must jointly satisfy four control sources. Text provides a global semantic description, reference images specify the appearance of the human and object, audio governs synchronized motion and lip movement, and pose provides explicit structural control over the human body (Zhou et al., 13 Apr 2026). The target use cases include e-commerce product demonstrations, short video production, interactive entertainment, and avatar-style interaction videos.

A central requirement is not merely photorealistic synthesis but joint controllability. The generated sequence should preserve identity and object appearance, follow intended motion, and remain synchronized with audio. This makes HOIVG distinct from narrower video-generation settings that optimize only one or two control signals.

The paper situates prior work in several categories and argues that none of them supports the full problem end to end. Reference-to-Video (R2V) methods are effective at preserving subject identity from reference images but typically lack audio responsiveness. Audio-to-Video (A2V) methods are effective at synchronized lip and motion generation but usually accept only an initial frame, limiting subject and object customization. Methods that combine audio and reference images generally ignore pose and are often not designed for HOIVG. HOI-specific systems such as AnchorCrafter, HunyuanVideo-HOMA, and DreamActor-H1 depend on restrictive structural inputs such as masks, meshes, trajectories, or bounding boxes and often cannot use audio cues. The paper therefore identifies three core obstacles: the controllability-versus-quality trade-off, data scarcity, and an evaluation gap (Zhou et al., 13 Apr 2026).

A common misconception is to treat OmniShow as a general omnimodal foundation model. The paper instead presents it as a task-specialized framework for HOIVG. This contrasts with broader omnimodal systems that unify text, image, speech, and video across understanding and generation tasks in a single backbone, such as Dynin-Omni (Kim et al., 9 Mar 2026).

2. Backbone, conditioning interface, and objective

OmniShow inherits its generative substrate from Waver 1.0, which uses a latent-diffusion formulation with Flow Matching supervision (Zhou et al., 13 Apr 2026). The objective is written as

$\mathcal{L}_{\text{FM} = \mathbb{E}_{t, \mathbf{x}_0, (\mathbf{x}_1, \mathbf{e})} \left[ \| v_\theta(t, \mathbf{x}_{\text{in}, \mathbf{e}) - u(\mathbf{x}_t | \mathbf{x}_1) \|^2 \right],$

where $t$ is the timestep, $\mathbf{x}_0 \sim \mathcal{N}(0,\mathbf{I})$ is Gaussian noise, $\mathbf{x}_1$ denotes clean video tokens, $\mathbf{e}$ is the text embedding, $\mathbf{x}_t$ denotes noisy video tokens, and $$u(\mathbf{x}_t \mid \mathbf{x}_1)=\mathbf{x}_1-\mathbf{x}_0$$ is the ground-truth flow velocity.

The underlying backbone is already task-unified for T2V and I2V through channel concatenation:

$\mathbf{x}_{\text{in} = \text{Concat}(\mathbf{x}_t, \mathbf{c}, \mathbf{m}),$

with $\mathbf{c} \in \mathbb{R}^{N \times D}$ denoting condition tokens and $\mathbf{m} \in {[0,1]}^{N \times 4}$ the condition mask. For T2V, the condition tokens are black-image tokens and the masks are zero. For I2V, the first frame is inserted into $t$0 and the mask entries are set to 1. OmniShow’s architectural modifications are designed to preserve this native conditioning pattern rather than replacing it, which the paper argues reduces disruption to the pretrained generative prior (Zhou et al., 13 Apr 2026).

The framework supports several multimodal conditioning configurations.

Setting	Conditions
R2V	text + reference image
RA2V	text + reference image + audio
RP2V	text + reference image + pose
RAP2V	text + reference image + audio + pose

This flexible interface is important because the HOIVG problem is not reducible to a single fixed-input regime. The same system must accommodate identity-preserving synthesis, audio-driven motion, pose-controlled animation, and full multimodal composition.

3. Unified Channel-wise Conditioning

Unified Channel-wise Conditioning is the mechanism used to inject reference images and pose while retaining compatibility with the backbone’s original token structure (Zhou et al., 13 Apr 2026). Rather than adding a separate cross-modal branch, OmniShow extends the backbone’s existing channel-wise conditioning scheme to additional temporal slots.

The design begins by rendering pose as an RGB video, producing pose-video tokens $t$1, while the reference image is encoded as tokens $t$2. To make image conditioning temporally compatible with video generation, the model augments the noisy video tokens with pseudo-frame tokens $t$3. The unified injection is formulated as

$t$4

where $t$5 denotes concatenation along the temporal axis and $t$6 is the augmented mask.

The pseudo-frame construction serves a specific purpose: it gives the model explicit temporal slots into which image-reference information can be inserted without altering the native video-token layout. The paper argues that this reduces the adaptation gap relative to token-concatenation alternatives and better preserves the base model’s generative prior.

The method is further stabilized by a Reference Reconstruction Loss. Leaving pseudo-frames as zeros is described as uninformative, so the pseudo-frames are initialized with noisy reference-image tokens perturbed at the same timestep $t$7, and a Flow Matching loss is applied on them as reference reconstruction loss $t$8. According to the paper, this encourages reconstruction of the reference image and improves retention of semantic detail, especially human identity (Zhou et al., 13 Apr 2026).

The ablations are consistent with this interpretation. Compared with token concatenation, Unified Channel-wise Conditioning yields better FaceSim, better NexusScore, and better AES, while removing the reference reconstruction loss reduces fidelity, particularly in identity preservation.

4. Gated Local-Context Attention

Gated Local-Context Attention is the subsystem responsible for audio-visual synchronization (Zhou et al., 13 Apr 2026). It consists of three components: audio context packing, attention map constraints, and adaptive gating.

Audio is encoded with Wav2Vec 2.0, and representations from multiple layers are merged to capture both semantic and rhythmic information. These features are then linearly interpolated to the video frame rate. A sliding window with size $t$9 stacks neighboring audio features, while stride $\mathbf{x}_0 \sim \mathcal{N}(0,\mathbf{I})$0 is used to align with VAE temporal compression. The resulting packed features are flattened chronologically into dense 2D contextual audio features. The stated purpose is to provide local temporal context that is adequate for phoneme-level and motion-level consistency.

Cross-modal interaction is constrained through a masked attention mechanism:

$\mathbf{x}_0 \sim \mathcal{N}(0,\mathbf{I})$1

where $\mathbf{x}_0 \sim \mathcal{N}(0,\mathbf{I})$2 denotes queries from video tokens, $\mathbf{x}_0 \sim \mathcal{N}(0,\mathbf{I})$3 and $\mathbf{x}_0 \sim \mathcal{N}(0,\mathbf{I})$4 are keys and values from audio tokens, and $\mathbf{x}_0 \sim \mathcal{N}(0,\mathbf{I})$5 is a binary attention mask. A value of 1 allows interaction and 0 blocks it, with $\mathbf{x}_0 \sim \mathcal{N}(0,\mathbf{I})$6 implemented as a large negative constant. The paper also states that audio tokens are zero-padded to align with pseudo-frame tokens. This local masking restricts each video frame to its corresponding neighborhood in the audio stream, thereby reducing interference from irrelevant audio regions.

To stabilize optimization, the module uses a learnable gate $\mathbf{x}_0 \sim \mathcal{N}(0,\mathbf{I})$7, initialized to a near-zero value $\mathbf{x}_0 \sim \mathcal{N}(0,\mathbf{I})$8. Audio injection is written as

$\mathbf{x}_0 \sim \mathcal{N}(0,\mathbf{I})$9

where $\mathbf{x}_1$0, $\mathbf{x}_1$1, $\mathbf{x}_1$2 denotes the full attention block, and $\mathbf{x}_1$3 is element-wise multiplication. The gate is described as reducing feature-distribution disruption, acting as an explicit indicator of audio impact, and leading the authors to place audio attention only in dual-stream blocks. This modification increases parameter count by only about 0.3B (Zhou et al., 13 Apr 2026).

Ablations support the module’s role. Removing audio context harms temporal coherence and Sync-D; removing attention-map constraints significantly harms synchronization; and removing adaptive gating degrades final visual quality. The best values reported in the ablation table are Sync-C: 9.023, Sync-D: 7.419, and AES: 0.540 (Zhou et al., 13 Apr 2026).

5. Decoupled-Then-Joint Training and data construction

The paper identifies data scarcity as a structural problem for HOIVG because no large paired dataset exists with full text-image-audio-pose quintuplets (Zhou et al., 13 Apr 2026). OmniShow addresses this by constructing multiple datasets from a large in-house human-centric video pool and by training in stages.

The appendix-described construction pipeline includes shot segmentation, multi-dimensional filtering, and specialized processing to derive R2V, A2V, RA2V, and RAP2V datasets from approximately $\mathbf{x}_1$4 clips totaling around 3500 hours. This reflects a heterogeneous-data regime rather than a single fully paired corpus.

The first phase is decoupled training. A specialized R2V model is trained on the R2V dataset with audio modules removed to match the base architecture, while a specialized A2V model is trained on A2V data, following the common practice of using the first frame as input. This stage is intended to let each model master its own control modality without interference.

The second phase is joint training via model merging. Audio modules are inherited from the A2V model, and the remaining weights are linearly interpolated with coefficients 0.6 for A2V and 0.4 for R2V. The paper motivates this asymmetry by stating that audio synchronization is more sensitive to weight disruption, whereas visual identity is more global and more tolerant. Notably, the merged model exhibits emergent RA2V capability before explicit RA2V training (Zhou et al., 13 Apr 2026).

After merging, the model is trained on the full RA2V dataset, then fine-tuned on a high-quality RA2V subset. Pose is introduced only in the final stage to avoid over-reliance on this strong signal. In the ablations, this staged strategy provides the best trade-off among the compared curricula, with NexusScore: 0.364, Sync-D: 8.14, and AES: 0.474 (Zhou et al., 13 Apr 2026).

This training design suggests a broader methodological point. Rather than requiring a fully aligned, fully multimodal dataset at scale, OmniShow treats HOIVG as a composition problem over partially overlapping sub-task datasets. The paper presents weight merging not only as an initialization trick but as a mechanism that itself can induce useful multimodal capability.

6. HOIVG-Bench, empirical results, and broader context

To address the evaluation gap, the paper introduces HOIVG-Bench, a benchmark with 135 curated samples (Zhou et al., 13 Apr 2026). Each sample contains a detailed text caption, a human reference image, an object reference image, semantically aligned audio, and a coherent pose sequence. The construction process includes video curation from an in-house library, object-image acquisition using Nano Banana, AI-generated but identity-deidentified human images based on video screenshots, pose extraction with DWPose, and audio synthesis in which GPT-4o generates an object-focused speech script and infers human attributes, while ElevenLabs synthesizes matching speech. The benchmark uses AI-generated human and object images to satisfy privacy and legal constraints, with manual filtering to reduce “AI-ness.”

The benchmark evaluates five dimensions: Text alignment using VideoReward TA; Reference consistency using FaceSim and NexusScore; Pose accuracy using AKD and PCK at 5% threshold; Audio-visual synchronization using Sync-C and Sync-D; and Video quality using VBench AES and IQA together with VideoReward VQ and MQ. For fairness, all outputs are standardized to 5-second clips at 720p portrait mode (Zhou et al., 13 Apr 2026).

The principal quantitative findings are organized by conditioning regime. In R2V, OmniShow reports FaceSim: 0.874, NexusScore: 0.389, TA: 7.746, VQ: 11.12, and MQ: 5.885. In RA2V, it reports Sync-C: 8.612, Sync-D: 7.608, NexusScore: 0.369, VQ: 10.86, and MQ: 5.554, while HuMo-17B slightly leads on TA and FaceSim. In RP2V, it achieves PCK: 0.460, NexusScore: 0.418, VQ: 10.28, and MQ: 4.937; the paper notes that FaceSim may decline in this regime because large pose changes alter viewpoint and facial morphology. In RAP2V, where no direct baseline exists, OmniShow is compared against a cascaded VACE + LatentSync pipeline and reports TA: 7.134 vs 6.885, FaceSim: 0.645 vs 0.591, NexusScore: 0.353 vs 0.341, Sync-C: 7.699 vs 7.016, Sync-D: 7.674 vs 7.823, AKD: 0.172 vs 0.198, PCK: 0.478 vs 0.340, VQ: 11.06 vs 10.05, and MQ: 5.880 vs 3.911 (Zhou et al., 13 Apr 2026).

Human evaluators preferred OmniShow in side-by-side comparisons on RA2V and RP2V. The paper attributes this preference to smoother temporal dynamics, richer visual details, and better perceived realism, even when some objective scores were similar (Zhou et al., 13 Apr 2026).

Within the broader literature, OmniShow occupies a specific niche. It is not an omni-modal understanding benchmark such as OmniEval, which evaluates whether models can jointly reason over synchronized video, audio, and text across perception, understanding, reasoning, and temporal grounding tasks (Zhang et al., 26 Jun 2025). Nor is it a general 3D/4D camera-control model such as OmniView, which targets view synthesis, camera-controlled generation, and temporal extrapolation (Fan et al., 11 Dec 2025). OmniShow instead addresses a narrower but practically consequential synthesis problem: end-to-end HOIVG under four simultaneous control modalities. A plausible implication is that models making broader claims about omni-modal competence may require complementary evaluation on both HOIVG-Bench and general multimodal suites such as OmniEval (Zhang et al., 26 Jun 2025).

The paper also states that the model can generate videos up to 10 seconds, although evaluation is standardized to 5-second clips. This distinction is methodologically relevant: the reported benchmark results establish performance under a fixed comparison protocol, while longer-horizon generation remains a capability claim rather than the primary basis of comparison (Zhou et al., 13 Apr 2026).