SenseNova-U1: Unified Vision–Language Model

• SenseNova-U1 is a unified multimodal model that integrates vision and language tasks through a native mixture-of-transformers architecture.
• It employs the NEO-unify backbone with dual-objective training to jointly learn analytic and creative capabilities without separate encoders or decoders.
• The model achieves state-of-the-art performance across vision-language benchmarks, image generation, and agentic decision-making with efficient scaling.

SenseNova-U1 is a unified vision–language foundation model developed to dissolve the entrenched dichotomy between multimodal understanding (perception and reasoning) and generation (synthesis and editing). It implements a "native" mixture-of-transformers (MoT) architecture, termed NEO-unify, in which both understanding and generation are synergistic outcomes of a single model backbone. SenseNova-U1 introduces a streamlined, end-to-end system that learns a joint representation space for both analytic and creative capabilities, excelling at tasks spanning vision–language perception, knowledge reasoning, spatial intelligence, agentic decision-making, image generation, editing, and early vision–language–action and world modeling benchmarks. The model eliminates frozen vision encoders, variational autoencoders, and cascaded pipelines, presenting a genuinely unified approach to multimodal intelligence (Diao et al., 12 May 2026).

1. Motivation and Foundational Principles

Traditional large vision–LLMs (VLMs) separate "understanding" (tasks such as VQA, classification, and reasoning) and "generation" (tasks such as text-to-image or interleaved outputs) into distinct modules. This results in duplicate architectures (separate encoders/decoders), fragmented latent spaces, and misaligned pipelines. Such structural separation precludes the emergence of native multimodal intelligence where analytic and generative processes share mutually supportive representations.

SenseNova-U1 fundamentally reconceptualizes this paradigm through:

• Direct operation on raw pixels and tokens, eschewing frozen VEs or VAEs.
• End-to-end learning of joint objectives for both understanding and generation.
• Synergistic integration of autoregressive text modeling with pixel-space flow matching within a unified backbone.
• Scaling from dense (8B parameters) to mixture-of-experts (30B parameters, ~3B active at inference) configurations.

SenseNova-U1 demonstrates performance that matches or surpasses leading open-source understanding-only and generation models, while maintaining a simplified, computationally efficient architecture (Diao et al., 12 May 2026).

2. NEO-unify Architecture

The core of SenseNova-U1 is the NEO-unify backbone, a mixture-of-transformers (MoT) framework with the following structural features:

• Patch Encoding and Decoding: Images are partitioned into 32×32 patches, encoded with two convolutional layers (stride 16, 2), GELU nonlinearity, and 2D sinusoidal rotary embeddings (RoPE), producing "visual tokens." These tokens co-inhabit a shared stream with word tokens, demarcated by <img>/</img> delimiters. Text is tokenized via BPE or WordPiece, then projected into the vision–language joint embedding space. For generation, an MLP head directly regresses pixel patches, while understanding uses a conventional linear vocab decoder.
• Dual-Objective Training: The model is trained with a composite objective:

$$\mathcal{L}_\mathrm{total} = \lambda_1\,\mathcal{L}_\mathrm{Und} + \lambda_2\,\mathcal{L}_\mathrm{Gen}$$

$\mathcal{L}_\mathrm{Und}$ is standard causal cross-entropy for text, and $\mathcal{L}_\mathrm{Gen}$ implements pixel-space flow matching:

$$z_t = t\,x + (1-t)\,\sigma_R\,\varepsilon, \quad t \in [0,1] \ v^\star = \frac{x-z_t}{1-t} \ v_\theta(z_t, t) = \frac{\hat{x}_\theta(z_t, t)-z_t}{1-t} \ \mathcal{L}_\mathrm{Gen} = \mathbb{E}_{t,x,\varepsilon,(H,W)} \| v_\theta(z_t, t) - v^\star \|_2^2$$

• Mixture-of-Transformers (MoT) Backbone: Each Transformer block processes two parallel streams (understanding and generation) with shared self-attention but separate Q/K/V projections, normalization, and FFN, dynamically routed by token type. Text tokens utilize causal attention; image tokens attend bidirectionally to both prefix text and full image span; noise tokens (generation) are separated from their clean counterparts during the backward pass.
• Native RoPE: Rotary position embeddings are extended across temporal (T, for text) and spatial (H, W, for vision) axes, each with independent sinusoidal encoding.
• Stream-wise Mixture-of-Experts (MoE): In the A3B (30B) variant, the understanding stream uses 128 experts (30B total), the generation stream uses 32 experts (8B), with top-8 routing yielding approximately 3B active parameters per inference step.

3. Model Variants, Scaling, and Training Pipeline

Two primary variants are offered:

Variant	Layers	Hidden	Attention Heads (Q/K)	MoE Experts (U/G)	Total Params	Active Params
SenseNova-U1-8B-MoT	42	4096	32 / 8	dense/dense	8.2B each	8.2B
SenseNova-U1-A3B-MoT	48	2048	32 / 8	128/32 (U/G)	30B/8B	~3B

The training pipeline comprises:

1. Understanding Warmup: Start from pretrained NEO, fuse Q/K projections, continue on a mid-sized corpus (~0.75T tokens).
2. Generation Pretraining: Freeze understanding; train generation stream on text-to-image data (low-res, high-res, editing/interleaved; ~1.38T tokens).
3. Unified Mid-Training: Joint training on a mixture of text-only, T2I, editing, and interleaved data (84K steps, target mixture 33%/37%/24%/6%).
4. Supervised Fine-Tuning: 9K steps on high-quality instructions, same mixture as above.
5. RL and Distillation for T2I: Flow-GRPO RL using OCR, style, and HPSv3 aesthetic rewards, and Distribution Matching Distillation (DMD2) to reduce generation steps (100→8).

4. Inference and System Optimizations

SenseNova-U1 is deployed through a unified API, abstracting two distinct underlying engines:

• LightLLM: Handles multimodal prefill, autoregressive text decoding, streaming, and flow control; uses tensor parallelism.
• LightX2V: Manages iterative pixel-space flow denoising; uses classifier-free guidance or sequence parallelism.

A hybrid attention kernel implements the “text causal/image bidirectional” mask using a block-level flag (b_image_token_end). Blocks containing only text take the fast causal path, while mixed or image-centric blocks enable full bidirectional span attention. This system design ensures maximal efficiency during inference across varying task requirements (Diao et al., 12 May 2026).

5. Empirical Performance Across Benchmarks

SenseNova-U1 demonstrates state-of-the-art or competitive results across a comprehensive range of multimodal benchmarks:

• Vision–Language Understanding & Reasoning:
  • MathVista: 85.5% (A3B), surpassing Qwen3.5.
  • MMBench-EN (VQA): 91.6% (A3B) vs. 90.1% (Qwen3.5-9B).
  • AI2D (OCR): 92.23%, OCRBench: 91.9%.
• Agentic Decision-Making:
  • τ²-Bench: 75.39% (A3B 8B) vs. Qwen3.5's 81.20% (9B).
  • ClawEval: 58.50% vs. Qwen3.5's 36.50% (8B/9B).
• Text-to-Image and Interleaved Generation:
  • GenEval: 0.91 overall (object composition), outscoring Qwen-Image and Lumina-DiMOO.
  • TIIF-Bench: 89.74/89.17 vs. DALL-E 3's 74.96/70.81.
  • CVTG-2K: 0.940 word accuracy (multi-region text) vs. 0.829 (Qwen-Image).
  • OpenING (interleaved): 9.16/9.07 vs. GPT-4o + DALL-E 3's 8.20.
• Infographics and Commercial Content:
  • IGenBench Q-ACC: 0.51 (8B) & 0.42 (A3B); BizGenEval layout/text/attribute (hard split): 61.6%/46.3%/47.5%, leading all open-source baselines.
• Image Editing:
  • ImgEdit: 4.45/5; GEdit-Bench: G_SC 8.27, G_PQ 7.49; RISEBench (reasoning edits): 30.0 with chain-of-thought.
• Semantic Consistency & Visual Fidelity:
  • PSNR 31.56 dB, SSIM 0.85 on COCO (NEO-unify 2B), rivaling high-quality VAE-based models.
  • RL-based tuning with OCR/IoU, style, and HPS v3 improves text and aesthetic quality.
• Action and World Modeling:
  • Ingests sparse manipulation frames and generates action descriptions; for push-object instructions, synthesizes next state, indicating emergent predictive reasoning in the generation stream.

6. Unified Representation, Modality Handling, and Guidance

The senseNova-U1 architecture is characterized by true modality fusion and native guidance strategies:

• Unified Representation Space: All perception and generation tasks derive from shared embeddings; no translation or re-encoding between modalities.
• Classifier-Free Guidance: For generation, the model uses

$$\nabla_x\log p(x|c_\mathrm{img},c_\mathrm{txt}) = \gamma\bigl[\nabla_x\log p(x|c_{\mathrm{img}},c_{\mathrm{txt}}) -\nabla_x\log p(x|c_{\mathrm{img}})\bigr] +\gamma_{\mathrm{img}}\bigl[\nabla_x\log p(x|c_{\mathrm{img}}) -\nabla_x\log p(x)\bigr] +\nabla_x\log p(x)$$

with $\gamma=4$, $\gamma_\mathrm{img}=1$, maintaining alignment without excessive amplification of visual conditioning.

• Native RoPE: Enables position encoding across all axes: temporal (text), height, and width (vision) without parameter overhead.

7. Roadmap and Theoretical Implications

SenseNova-U1 exemplifies a transition from pipeline-based, stitched-together multimodal systems toward a holistic, end-to-end paradigm where perception, reasoning, and generation co-evolve within a single architecture. This approach is intended to support the emergence of long-horizon planning, embodied interaction, and holistic world-modeling within future models.

A plausible implication is that as models following this design principle scale, richer cognitive capacities may arise natively, eliminating the necessity to explicitly align or coordinate discrete processing pipelines. SenseNova-U1 serves both as a performant foundation model and an initial step toward "native multimodal intelligence" characterized by unified "think-and-act" modeling (Diao et al., 12 May 2026).