UniVerse-1: Unified Audiovisual Generation

• UniVerse-1 is an open-source unified audio-video generation model designed to generate synchronized and coherent audiovisual content by deeply fusing pre-trained audio and video experts.
• The model utilizes a novel Stitching of Experts strategy that integrates Wan2.1 and Ace-step models, ensuring precise temporal alignment and cross-modal interaction.
• Advanced techniques like conditional flow matching, independent noise sampling, and an online annotation pipeline enable robust training on 7,600 hours of curated data.

Searching arXiv for papers directly relevant to "UniVerse-1" and closely related unified audio-video generation work. Searching query "UniVerse-1" audio video generation to=arxiv_search.search 皇轩 code {"query":"UniVerse-1 audio video generation","max_results":5} UniVerse-1 is an open-source, Veo-3-like unified audio-video generation model designed to generate coherent video and synchronized audio jointly rather than treating video generation as a “silent movie” stage followed by post-hoc audio synthesis. Its central objective is synchronous audiovisual generation for settings in which temporal coupling is structurally important, including lip-synchronized speech, ambient sound generation, speech-plus-video generation, and scenes in which sound must match specific visual events in time (Wang et al., 7 Sep 2025).

1. Scope and problem formulation

UniVerse-1 is motivated by the observation that audiovisual content is inherently multimodal: a barking dog should coincide with a visible dog, chopping vegetables should produce chopping sounds at the right moments, speech should align with mouth movement, and environmental sounds should track the visual scene. The model is therefore constructed as a joint generative system rather than as a pipeline in which audio is conditioned only on a completed video sample (Wang et al., 7 Sep 2025).

The paper argues that post-hoc video-to-audio generation cannot enforce true temporal coupling at training and inference time. In particular, it does not solve the reverse direction, cannot guarantee lip sync during text-conditioned generation, and does not naturally support synchronous “Veo-3-like” audiovisual synthesis. UniVerse-1 addresses this by generating audio and video together in one model.

Within the paper’s framing, the system is also intended as an open-source baseline for closing the gap with proprietary unified audiovisual systems. At the same time, it does not claim parity with Veo-3 in absolute quality, and it does not claim to outperform every task-specialized baseline on every metric. That positioning is important to its role in the literature: UniVerse-1 is presented as a unified framework with strong audiovisual coupling rather than as a replacement for all specialist models.

2. Architectural design and Stitching of Experts

The central architectural idea is Stitching of Experts (SoE), in which UniVerse-1 fuses two pre-trained foundation models instead of training a unified audiovisual model from scratch (Wang et al., 7 Sep 2025). The two experts are:

• Wan2.1 (1.3B) as the video expert
• Ace-step (3.5B) as the music/audio expert

The paper attributes three functions to SoE: training efficiency, reuse of unimodal priors, and deep cross-modal interaction. The fusion is not limited to input- or output-level concatenation. Instead, corresponding transformer blocks in the video and audio models are deeply fused in a dual-stream architecture so that each modality can affect the other during generation.

The Wan2.1 video model includes a 3D VAE, a umT5 text encoder, and a Diffusion Transformer (DiT). Video of shape $(3, T, H, W)$ is compressed to a latent of shape $(16, T/t, H/h, W/w)$ with downsampling factors $t=4$ and $h=w=8$. The latent is patchified with kernel $(1,2,2)$ and fed to the DiT, which predicts velocity under a Flow Matching objective.

The Ace-step audio model includes a Music-DCAE, a umT5 text encoder, a lyric encoder, a speaker encoder, and a DiT. Audio is converted to a mel spectrogram and encoded into a latent of shape $(8, T/t, F/f)$ with $t=f=8$, then patchified with kernel $(16,1)$. Conditioning comes from the music style prompt, lyrics, and speaker identity. UniVerse-1 removes the speaker encoder because the target is general speech generation rather than speaker-specific synthesis.

Cross-modal communication occurs inside each block. In the video branch, after self-attention, hidden states from the audio branch are injected; in the audio branch, audio hidden states are injected into the video branch’s self-attention and cross-attention. To stabilize scale, incoming hidden states are normalized by a shared LayerNorm before injection into the video stream, and a two-layer linear adapter aligns the incoming features. Figure-level details reported in the paper specify that video self-attention receives additional mel tokens, video cross-attention also receives mel-derived context, and mel/audio cross-attention receives video tokens as additional context. For the video stream, audio features are projected through dedicated key and value projections, followed by frame-by-frame cross-attention using video queries so that temporal alignment is preserved.

A practical complication is that Wan2.1 and Ace-step have different numbers of transformer blocks. UniVerse-1 resolves this with a layer interpolation strategy: it computes the depth difference, inserts new blocks into the shallower model at uniform intervals, and initializes the inserted block parameters by linear interpolation of neighboring weights. This is described as a smooth bridge between the two pretrained backbones rather than hard architectural mismatch.

3. Objectives, alignment losses, and stabilization mechanisms

Both experts are trained with Conditional Flow Matching, and UniVerse-1 retains that formulation in the fused setting (Wang et al., 7 Sep 2025). Given noise $x_0 \sim p_0$ and data sample $x_1 \sim p_1$, the interpolation path and target velocity are

$(16, T/t, H/h, W/w)$0

The model $(16, T/t, H/h, W/w)$1 is trained with

$(16, T/t, H/h, W/w)$2

UniVerse-1 uses a weighted combination of video flow matching loss, audio flow matching loss, and a semantic alignment loss for audio:

$(16, T/t, H/h, W/w)$3

The Semantic Alignment Loss (SSL) uses an intermediate feature $(16, T/t, H/h, W/w)$4 from layer $(16, T/t, H/h, W/w)$5 of the fused block, with the configuration $(16, T/t, H/h, W/w)$6. That feature is aligned against target features from two pretrained audio encoders: MERT, described as a general music representation, and mHuBERT, described as speech-centric. The intermediate feature is projected by two heads, $(16, T/t, H/h, W/w)$7 and $(16, T/t, H/h, W/w)$8, temporally interpolated to match target sequence lengths, and optimized with a negative cosine similarity criterion.

A separate training mechanism is the Low Quality Data Loss Strategy (LQLS). The paper identifies VGGSound and AudioSet as sources with good audio diversity but poor visual quality. To prevent low-quality visuals from damaging video learning, UniVerse-1 applies normal video flow matching on cleaner data, but for these low-quality datasets computes video loss only when the diffusion timestep $(16, T/t, H/h, W/w)$9 is high. The threshold is set to $t=4$0 out of 1000 steps, so low-quality datasets contribute only in the noisy, coarse regime where the model learns broad structure rather than high-frequency visual detail.

Another stabilization mechanism is the independent noise sampling strategy. The paper observes that if a single PRNG seed is used sequentially for both modalities, the generated noise tensors for video and audio may become spuriously correlated, creating an accidental dependency of the form $t=4$1. This can be exploited as a shortcut during training and can degrade audio quality at inference if video sampling changes alter the PRNG state. UniVerse-1 therefore uses separate PRNG instances and independently seeded video and audio noise. The ablation study reports that removing this strategy significantly hurts FD, KL, CLAP score, CE/CU, WER, lip-sync, and AV alignment.

4. Data curation and online annotation

A major contribution of UniVerse-1 is its online annotation pipeline, introduced to address temporal and semantic mismatch produced by static offline captions (Wang et al., 7 Sep 2025). The problem described in the paper is that training samples are short clips randomly extracted from longer videos; a caption generated for the full video may not describe the sampled time span and may even refer to events outside the clip. In unified audio-video generation, even small temporal shifts can invalidate the sound description.

The solution is a server-side pipeline that annotates the exact clip used for training. The process is: randomly sample a fixed-length clip, for example 5 seconds; immediately annotate that clip; place the aligned result into a shared buffer; and allow the training process to consume these prepared tuples. The pipeline uses Whisper for ASR and Qwen2.5-Omni for multimodal captioning. Whisper produces speech transcription, and a structured prompt containing the transcribed speech is passed יחד with the audio-video clip to Qwen2.5-Omni, which outputs verified speech content, a video caption, and an ambient audio caption. The reported purpose is to improve temporal alignment, annotation accuracy, speech content correctness, ambient sound descriptions, and overall training consistency.

The final training dataset totals about 7,685 hours, described in the paper as approximately 7,600 hours. It draws from YouTube, movie clips, Pexels stock footage, VGGSound, and AudioSet. The YouTube collection includes music variety shows, classical performances, cooking tutorials, public speeches, interviews, vlogs, and tool-usage demonstrations.

For YouTube, Pexels, and movie clips, the filtering pipeline removes clips without audio; keeps only resolution $t=4$2; requires bitrate-to-resolution ratio $t=4$3; requires DOVER aesthetic score $t=4$4; segments videos with PySceneDetect; discards clips shorter than 5 seconds; removes silence using audio activity detection; detects speech with Whisper; and, when speech is detected, requires a face via RetinaFace and uses SyncNet to verify lip sync, retaining only clips with SyncNet confidence $t=4$5 as speech data.

Split	Hours	Description
Verified speech-centric content	1,187	Speech and lip-sync focused
General-purpose audio-video content	3,074	Broad audiovisual scenes
VGGSound and AudioSet	3,422	Audio strengthening

This composition is intended to support ambient sound generation, speech-lip synchronization, broader environmental sound coverage, and robustness across real-world audiovisual scenes.

5. Verse-Bench and empirical evaluation

To evaluate unified audiovisual generation systematically, the paper introduces Verse-Bench, a benchmark intended to assess video generation quality, audio generation quality, text-to-speech, audio-to-video, video-to-audio, and fully joint audio-video generation (Wang et al., 7 Sep 2025). Its purpose is explicitly not only to test each modality in isolation but also to evaluate whether generated audio and video are consistent with one another.

Verse-Bench contains 600 samples from diverse sources including YouTube frames, BiliBili frames, TikTok clips, movie frames, anime frames, AI-generated images, and public web images. It is divided into three subsets:

• Set 1-I: 205 samples of image-text pairs, with captions and speech content annotated using LLMs and manual review
• Set 2-V: 295 video clips from YouTube and BiliBili, with captions generated by LLMs, Whisper ASR transcripts, and human verification
• Set 3-Ted: 100 TED Talk samples from September 2025, annotated using the same pipeline as Set 2

The benchmark covers sound categories including human speech, birds, cats and dogs, music, rain and thunder, typing, cooking and chopping, machinery, cars, planes, trains, glass breaking, and explosions.

Its evaluation protocol is task-specific. Video generation uses Motion Score (MS) based on normalized optical flow magnitude from RAFT, Aesthetic Score (AS) aggregated from MANIQA, aesthetic-predictor-v2-5, and MUSIQ, and ID Consistency (ID) based on DINOv3 similarity between a reference image and generated frames. Audio generation uses FD, KL, CS via CLAP similarity, CE/CU from AudioBox-Aesthetics, PC, PQ, and a protocol that also mentions IS. TTS uses WER from Whisper-large-v3. Audio-to-video adds SyncNet confidence for lip sync. Video-to-audio adds AV-A alignment using Synchformer. Joint audio-video generation uses the relevant metrics from both modalities.

The reported quantitative findings emphasize strong unified performance rather than universal dominance over specialist systems. UniVerse-1 reports ID = 0.89, described as the best identity preservation among the video metrics compared in the paper; PC = 2.49; and, for joint generation, AV-A = 0.23 and CS = 0.16, exceeding SVG’s AV-A = 0.09 and CS = 0.08. For fully generated audio-to-video lip sync, UniVerse-1 reports LSE-C = 1.34. The paper notes that this is lower than ground-truth-audio-based audio-to-video systems such as Wan-S2V, but also notes that those systems are evaluated in an easier setting because they are given the target audio.

Two ablations are emphasized. Removing LQLS reduces motion score, aesthetic score, ID consistency, audio metrics, and AV-A. Removing independent noise sampling substantially degrades audio generation quality, CLAP alignment, CE/CU, WER, lip sync, and AV-A. These ablations support the claim that both the selective use of low-quality visual data and the decoupling of modality noise are operationally important.

6. Position in the literature, implementation, and limitations

UniVerse-1 is presented in relation to prior joint audio-video models such as MM-Diffusion, SyncFlow, UniForm, R-FLAV, and SVG, which the paper characterizes as having one or more limitations including small training datasets, low video quality, weak disentanglement, class-conditional settings, or limited synchronization quality (Wang et al., 7 Sep 2025). Against that backdrop, UniVerse-1 is defined by four distinguishing ingredients: SoE-based fusion of pretrained experts, online clip-level annotation, independent noise sampling, and training on approximately 7,600 hours of curated audio-video data.

The implementation reported in the paper uses 50k steps, effective batch size 128, AdamW, learning rate $t=4$6, gradient accumulation = 4, and distributed training with FSDP. The model and code are made publicly available, and the paper explicitly frames this openness as part of an effort to advance research in unified audio-video generation.

The paper is also explicit about limitations. UniVerse-1 is constrained by the Wan2.1-1.3B video backbone; larger video backbones are identified as a future direction. The authors also call for more refined data curation. In comparative terms, the model is said still to trail top proprietary systems such as Veo-3. This suggests that the main significance of UniVerse-1 lies not in claiming closure of the performance gap, but in providing an open-source and technically concrete route toward unified, temporally aligned audiovisual generation.

A common misunderstanding would be to treat UniVerse-1 as simply a video model with an attached audio decoder. The architecture and training protocol indicate something stronger: the model is designed as a deeply fused, dual-stream diffusion system in which audio and video hidden states interact throughout the network depth. Another misunderstanding would be to read the model as a specialist state-of-the-art replacement for all unimodal generators. The paper does not make that claim. Its contribution is the construction of an open unified framework that is especially strong in ambient sound generation and speech-video alignment while establishing a reproducible baseline for future open-source audiovisual systems.