SenseNova-U1-8B-MoT: Unified Multimodal Model

• SenseNova-U1-8B-MoT is a unified multimodal model architecture built on the NEO-unify framework that combines text and image understanding with generation capabilities.
• It employs a unified Mixture-of-Transformers backbone with dedicated per-stream parameters to enable both bidirectional and causal attention across modalities.
• The model leverages a multi-stage pretraining curriculum, classifier-free guidance, and a Memory-of-Thought mechanism to enhance reasoning, factual accuracy, and multimodal performance.

SenseNova-U1-8B-MoT is a native unified multimodal model architecture developed within the NEO-unify framework. It combines text and image understanding with generation capabilities in a single model, eliminating the typical separation between perception and synthesis seen in conventional vision-LLMs (VLMs). The design targets robust multimodal intelligence, supporting advanced reasoning, linguistic competence, visual fidelity, and emergent agentic behaviors across tasks involving text, vision, and their integration (Diao et al., 12 May 2026).

1. Model Architecture

SenseNova-U1-8B-MoT utilizes the NEO-unify paradigm, featuring a unified Mixture-of-Transformers (MoT) backbone that processes image, noise (generation), and text tokens within a joint self-attention stream while employing separate per-stream parameters.

The architecture comprises:

• A lightweight patch encoding layer using two convolutional (conv+GELU) layers (strides 16 and 2) and sinusoidal 2D position encoding, mapping raw images into sequences of 32×32-pixel tokens.
• Tokens are delimited using specific start/end tokens (<img>…</img>).
• A unified MoT backbone routes tokens into “understanding” or “generation” streams at each transformer layer. Each stream uses its own Q/K/V projections, LayerNorm, and feed-forward layers, while self-attention is computed over the entire sequence using hybrid masking.
• Outputs include text logits (understanding stream) and pixel-space predictions (generation stream), with a shallow patch decoding layer (MLP head) regressing pixel patches for image synthesis and a linear projection head for text prediction.
• Native rotary positional encoding (RoPE) is extended across temporal and spatial (T, H, W) axes.
• For the 8B-MoT variant, routing is “dense” with each token always selecting the single expert; by contrast, the A3B variant supports per-token softmax expert selection and top-k routing.

SenseNova-U1-8B-MoT Configuration

Configuration	Value
Patch size	32×32 pixels
Pre-Buffer (conv enc.)	Yes
Transformer layers	42
Attention heads (Q/KV)	32 / 8
Head dim (T/H/W)	64 / 32 / 32
Hidden size	4096
MoT experts	1 (dense, per stream)
Total params	8.2B (und/gen)

This architecture enables bidirectional and causal attention patterns depending on token type. The patch-based and RoPE-extended encoding supports precise spatial and temporal relationships.

2. Pretraining Data and Objectives

A multi-stage pretraining curriculum is employed:

• Stage 1 (Understanding Warmup): 0.75T tokens from image–text pairs (32%), captions (17%), infographic QA (14%), and pure text (37%).
• Stage 2 (Generation): 1.38T tokens from text-to-image data (progressive resolutions 256²→2048²).
• Stage 3 (Unified Mid-Training): 1.19T tokens, mixing multimodal understanding (33%), text-to-image (37%), image editing (24%), and interleaved image–text (6%).
• Stage 4 (SFT): 0.13T tokens, same mixture.

Data processing includes cross-source deduplication, CLIP-ratio re-captioning, distribution-balanced sampling, prompt augmentation, and quality filtering.

The joint training objective is:

$$\mathcal{L}_{\text{total}} = \lambda_1 \mathcal{L}_{\mathrm{Und}} + \lambda_2 \mathcal{L}_{\mathrm{Gen}}$$

with

• Autoregressive text loss (CE): $\mathcal{L}_{\mathrm{Und}}$
• Pixel-space flow matching for image generation: $\mathcal{L}_{\mathrm{Gen}}$

Classifier-Free Guidance (CFG) is used for generation with guidance weights $\gamma=4$ for text and $\gamma_{\mathrm{img}}=1$.

3. Training and Inference Strategies

Training is staged:

• Stage 1: 120k steps at 2×10⁻⁵ learning rate (constant).
• Stage 2: 120k, 60k, and 120k steps across phases, with cosine decay from 1×10⁻⁴ to 2×10⁻⁵.
• Stage 3: 84k steps at 2×10⁻⁵, with loss weights λ₁=0.1, λ₂=1.0.
• Stage 4: 9k steps with cosine decay.

Data preprocessing incorporates:

• Distribution-balanced sampling (CLIP-based clustering, attribute stratification).
• Prompt augmentation (semantic, structural, role, complexity).
• Multi-criteria filtering (correctness, hallucination, instruction compliance).
• Four-stage flow for generation data and consistency checks for interleaved sequences.

Inference utilizes disaggregated runtime engines:

• “LightLLM” for multimodal prefill and autoregressive text decoding.
• “LightX2V” for iterative pixel-space denoising (CFG2/sequence parallelism).
• Shared key/value caches and hybrid attention kernels (Triton/FlashAttention3). A unified API supports task switching and employs token-type routing.

4. Integration of Memory-of-Thought (MoT)

SenseNova-U1-8B-MoT operationalizes the Memory-of-Thought mechanism (Li et al., 2023) to enhance reasoning and factual accuracy without altering model parameters. MoT consists of:

• Pre-thinking Stage: The model generates multiple chain-of-thought samples for each unlabeled input, computes answer distributions, and stores high-confidence traces in an external memory bank.
• Test-time Retrieval: Given a test query, relevant memories are retrieved (using embedding-based semantic clustering and LLM-based re-ranking) and prepended as demonstrations to guide inference.

Key details for adaptation:

• Supports up to 8k token contexts; demonstration count adjusts to fit this limit.
• Embeddings for memory retrieval can use the built-in SenseNova-U1-8B-embed API or external SBERT models.
• Memory storage leverages vector stores such as FAISS.

This approach enables the model to benefit from episodic memory-like retrieval, injecting relevant high-confidence reasoning traces as context, which improves accuracy—particularly in arithmetic, commonsense, and factual reasoning.

5. Evaluation and Performance

SenseNova-U1-8B-MoT achieves strong results across a spectrum of multimodal and text-only benchmarks.

Task/Benchmark	8B-MoT Score	Reference Model
MMMU	74.78%	Qwen3VL-8B: 74.10%
MMBench-EN	90.25%	Qwen3VL-8B: 87.50%
OCRBench-v2	61.30%	Qwen3VL-8B: 61.55%
VSI-Bench (32 frames)	62.66%	Qwen3VL*: 56.61%
MMLU-Pro	81.44%	Qwen3VL: 77.30%
IFBench	67.01%	Qwen3VL: 29.93%
τ²-Bench	71.70%	Qwen3VL: 31.65%

In image generation, SenseNova-U1-8B-MoT scores 0.91 on GenEval (compositional), 89.74 on TIIF-short, and 0.940 on CVTG-2K (text regions). In image editing, metrics include 3.90 (ImgEdit), 8.27 (GEdit-Bench), and 26.9 on RISEBench with chain-of-thought support. Interleaved and unified reasoning benchmarks demonstrate 9.07 on OpenING (with CoT) and 68.8 on VBVR-Image.

Ablations indicate native encoder-free patching supports near-lossless pixel reconstruction (PSNR 31.56, SSIM 0.85 for 2B variant), while dense co-training of understanding and generation streams yields minimal loss conflict and log-linear returns with data scaling.

6. Limitations and Future Directions

SenseNova-U1-8B-MoT demonstrates high operational efficiency, robust multimodal unification, and scalability. Notable limitations are:

• Editing performance below specialized models on highly complex hybrid edits, with visible grid artifacts at patch boundaries (32×32 pixels).
• Latent interference between understanding and generation streams at extreme model scales.
• Pretraining data remains predominantly web-centric, with limited coverage of rare scenarios.

Potential improvements include replacing the MLP decoder with PixelShuffle and conv layers to address boundary artifacts, extending the MoT backbone to support stream-wise MoE for further efficiency, and incorporating 3D or temporal priors for embodied and world-modeling tasks.

Planned research includes integration of action tokens and state embeddings for vision-language-action (VLA) and world modeling (WM), leveraging video–instruction pairs, joint RL fine-tuning, and internal world-model pretraining based on state-transition simulation in pixel-space (Diao et al., 12 May 2026).

7. Significance and Research Context

By unifying understanding and generation streams in a single transformer architecture and applying memory-augmented guidance via MoT, SenseNova-U1-8B-MoT represents a substantive advance over previous modular VLMs. It achieves competitive or superior results to specialist models across perception, generation, reasoning, editing, and emerging agentic tasks. This supports the paradigm shift from pipelined, modular VLM systems to native unified networks capable of “thinking and acting across modalities in a native manner” (Diao et al., 12 May 2026, Li et al., 2023).