Lumos-Nexus: Efficient Frequency Bridging with Homogeneous Latent Space for Video Unified Models

Published 29 May 2026 in cs.CV and cs.AI | (2605.31603v1)

Abstract: Connector-based video unified models have demonstrated strong capability in instruction-grounded video synthesis, but integrating a large high-fidelity generator into the unified training loop is computationally prohibitive, limiting achievable visual quality. We therefore propose Lumos-Nexus, a training-efficient unified video generation framework that facilitates the development of strong reasoning-driven generation capabilities while significantly enhancing visual fidelity. Lumos-Nexus adopts a two-stage design: 1) During training, only a lightweight generator is aligned with the understanding block to learn to take in reasoning-driven semantic control. 2) During inference, we introduce Unified Progressive Frequency Bridging (UPFB) to progressively hand off generation to a high-capacity pretrained generator in the shared latent space, enabling coarse-to-fine refinement and producing high-fidelity videos without compromising reasoning quality. To fill the gap in reasoning-driven video generation benchmarks, we introduce VR-Bench, which assesses a model's capability to translate inferred intent into coherent and semantically aligned video content. Extensive experiments demonstrate that Lumos-Nexus achieves substantial gains in visual realism and temporal coherence on VBench, while exhibiting strong reasoning-based generative performance on VR-Bench. Code and models are available at https://jiazheng-xing.github.io/nexus-lumos-home/.

Abstract PDF Upgrade to Chat

Authors (12)

Summary

The paper introduces a two-stage training framework that leverages a lightweight generator for semantic control and a large generator for high-fidelity detail via UPFB.
It employs a homogeneous latent space with a cosine-shaped temporal handoff to fuse low- and high-frequency components for robust semantic and visual alignment.
Experimental results on VBench and VR-Bench show state-of-the-art performance in fidelity, temporal coherence, and multi-dimensional reasoning evaluation.

Lumos-Nexus: Training-Efficient Unified Video Generation via Progressive Frequency Bridging

Introduction and Motivation

Lumos-Nexus addresses a core limitation in unified video generation: the prohibitive cost of end-to-end training with large, high-fidelity diffusion generators, which restricts practical scaling and impacts visual quality. Traditional connector-based video unified models leverage a multimodal understanding block for semantic priors and a diffusion generator for visual synthesis. However, aligning a massive generator with the semantic head requires extensive compute, leading to models that either sacrifice reasoning for fidelity or vice versa.

Lumos-Nexus introduces a two-stage, computationally efficient framework. First, a lightweight generator is aligned with the understanding block during training for cost-efficient acquisition of reasoning-driven semantic control. Second, Unified Progressive Frequency Bridging (UPFB) enables inference-stage collaboration between the small (semantic) and large (fidelity) generators in a homogeneous latent space, ensuring high-quality, semantically faithful video synthesis. Additionally, the paper proposes VR-Bench, a systematic benchmark to evaluate reasoning alignment in text-to-video generation.

Figure 1: Examples from Lumos-Nexus illustrating high-fidelity, reasoning-aware text-to-image and text-to-video generation outputs.

Methodology

Connector-Based Unified Models and Latent Homogeneity

Lumos-Nexus builds on the connector-based paradigm, in which a pretrained visual-LLM (VLM) understanding block produces semantic embeddings that are transformed via a connector into generator-compatible conditioning. Crucially, both the lightweight and high-capacity generators share a homogeneous latent space, enabling direct inheritance of structure semantics from small to large generator during inference.

Unified Progressive Frequency Bridging (UPFB)

Directly mixing or averaging the outputs of small (semantic) and large (fidelity) generators during denoising yields semantic instability and structural artifacts due to architectural bias mismatches. UPFB resolves this by introducing a temporally progressive, frequency-aware fusion strategy:

The small generator dominates early denoising steps, emphasizing global semantics and layout, while the large generator incrementally refines high-frequency details in later stages.
Temporal handoff is governed by a cosine-shaped weight $w_t$ , modulated by hyperparameter $\gamma_w$ , which allows for controlled semantic-to-detail transitions.
Frequency decomposition is performed on velocity fields at each sampling step, splitting them into low- (LF) and high-frequency (HF) components via Gaussian smoothing with time-decaying bandwidth. The small generator provides LF guidance (semantic coherence), while the large generator contributes HF refinement (details).
Dual-frequency bridging produces the final velocity prediction through asymmetric fusion, suppressing the noisy high-frequency content from the small generator.
Root-mean-square (RMS) alignment and energy re-balancing ensure numerically stable fusion across denoising steps.
Figure 2: Lumos-Nexus architecture. (a) Training aligns the connector and small generator. (b) Inference employs UPFB, fusing semantic and high-fidelity cues for video synthesis.

Reasoning-Driven Video Generation and VR-Bench

Standard video generation benchmarks (e.g., VBench) evaluate visual fidelity and temporal coherence but overlook reasoning: the capacity to map inferred semantic intent to coherent video content. VR-Bench is introduced to fill this gap, systematically evaluating models across eight reasoning dimensions:

High-Level Physical World Reasoning: Dynamic Reference Frame, Energy Transfer Visualization, Material Memory Consistency.
High-Level Commonsense Reasoning: Conceptual Action Reasoning, Cultural Commonsense Reasoning, Preventive Causal Reasoning.
Embodied Physical Reasoning: Biological Behavior Reasoning, Concurrent Action Coordination.

Evaluation leverages an MLLM (Qwen3-VL-30B) to answer metric-specific questions based on generated video analysis.

Figure 3: Overview of VR-Bench evaluation structure and process.

Figure 4: T2V qualitative comparison on VR-Bench, highlighting differences across key reasoning categories.

Experimental Results

Quantitative and Qualitative Analysis

Lumos-Nexus sets a new state-of-the-art on VBench-T2V, achieving 84.12 in overall score and outperforming competitive open- and closed-source baselines in both fidelity and temporal metrics. On VR-Bench, it achieves a strong 79.28 in overall reasoning, surpassing previously leading open-source solutions (e.g., Wan2.1-14B), and demonstrates robust alignment across all reasoning axes.

Figure 5: Visual comparison of video samples under different $\gamma_w$ handoff sharpness, illustrating the controllable tradeoff between semantic and detail dominance.

Ablative experiments validate that the UPFB fusion strategy yields substantial gains over naïve bridging schemes (e.g., direct average). Cost analysis indicates that Lumos-Nexus achieves these gains at a marginal inference overhead relative to large generator-only baselines, while incurring only the training cost of the small generator.

Ablation Studies

Key findings from the extensive ablation studies:

Optimal $\gamma_w = 0.3$ in the temporal handoff achieves the best semantic-detail tradeoff and maximizes both VBench and VR-Bench scores.
Intermediate frequency bandwidth $(\sigma_{\min}, \sigma_{\max})=(0.35, 0.70)$ yields strongest performance; excessively narrow or wide bands degrade alignment or fidelity.
RMS normalization and energy re-balancing are critical for stable fusion and prevent divergence in denoising trajectories.
Direct bridging or upscaling the generator backbone without UPFB produces duplicate structures, inferior realism, and cannot exploit semantic priors from the VLM.
Figure 6: Comparison of bridging strategies; UPFB outperforms naïve direct averaging strategies, reducing artifacts and duplications.

Theoretical and Practical Implications

Lumos-Nexus re-architects the cost structure of unified video generation: high-capacity generators are only required in inference, enabling scalable deployment without retraining costs. The framework decouples the learning of semantic alignment and visual quality, allowing strong reasoning and high-fidelity generations without demands for end-to-end large-model optimization. The homogeneous latent space assumption is validated and shown to be a practical, transferable constraint for state-of-the-art model families.

VR-Bench represents an essential diagnostic tool for the next generation of multimodal generative models, shifting evaluation toward semantic, physical, and embodied reasoning.

Figure 7: VR-Bench qualitative comparison; Lumos-Nexus demonstrates clear improvements in complex reasoning dimensions, including dynamic reference frames and causal action sequences.

Future Directions

Building on Lumos-Nexus, promising directions include:

Extending UPFB to scenarios with heterogeneous latent spaces via lightweight domain adaptation.
Model distillation to amortize inference costs, compressing UPFB outcomes into a single generator.
Integrating more powerful VLMs or structured symbolic reasoning modules for even richer semantic control.
Expanding VR-Bench to encapsulate multi-agent, long-horizon, and abstract reasoning scenarios.

Conclusion

Lumos-Nexus delivers a training-efficient, high-fidelity, and reasoning-preserving framework for unified video generation. The UPFB mechanism is both theoretically sound—respecting the inductive biases of different generator capacities—and empirically validated to yield superior semantic and visual outcomes over alternative strategies. The introduction of VR-Bench moves the evaluation standard toward alignment with real-world, intent-grounded video synthesis.