NEO-unify: Unified Multimodal Transformer

• NEO-unify architecture is a native multimodal paradigm that unifies language, vision, and generation using a single transformer backbone.
• The model eliminates modality-specific encoders by jointly processing tokens with integrated patch encoding and a Mixture-of-Transformers backbone.
• Its unified design yields state-of-the-art performance and emergent capabilities such as vision-language reasoning and interleaved multimodal generation.

The NEO-unify architecture is a native multimodal paradigm unifying language, vision, and generation within a single end-to-end sequence model. By eliminating modality-specific encoders or cascaded pipelines, NEO-unify establishes a transformer-based framework in which multimodal understanding and generation emerge as synergistic, parameter-sharing processes. This approach is exemplified in large-scale models such as SenseNova-U1, which demonstrate state-of-the-art performance across visual perception, language–vision reasoning, high-fidelity generation, and interleaved modalities without the need for explicit translation or bridging modules (Diao et al., 12 May 2026).

1. Foundational Principles and Motivation

NEO-unify was developed to address the entrenched dichotomy in conventional Vision-LLM (VLM) design: understanding (e.g., VQA, captioning) and generation (e.g., T2I, image editing) are traditionally handled as separate sub-problems with distinct architectures and misaligned latent representations. This persistent split is regarded as a structural bottleneck, hindering the emergence of true, “native” multimodal intelligence (Diao et al., 12 May 2026).

The core premise of NEO-unify is that understanding and generation are duals—a single process parameterized by one transformer backbone, in which tokens corresponding to language, vision, and noise (in diffusion-style generation) are processed together. This yields a model that “thinks and acts across modalities in a native manner,” endowing it with consistent representation learning and eliminating piecemeal integration (Diao et al., 12 May 2026).

2. Architectural Blueprint

2.1 Patch Encoding and Decoding

• Patch-Encoding Layer: Input images are decomposed by two 2D convolutional layers (GELU-activated; strides 16 then 2), yielding a flattened sequence of visual tokens, each encoding a $32\times32$ patch. Special delimiters <img>, </img> bracket the patch stream, enabling clear separation from text tokens. Both images and text tokens are projected into a shared $d$-dimensional embedding space.
• Patch-Decoding Layer: The model comprises two distinct output heads:
  • The understanding stream projects token embeddings via a linear head to logits over the vocabulary.
  • The generation stream employs a lightweight MLP to predict pixel-space patches directly—eschewing VAEs or diffusion U-Nets (Diao et al., 12 May 2026).

2.2 Mixture-of-Transformers (MoT) Backbone

• The backbone uses an improved NEO primitive: a native transformer block with extended rotary position embeddings (RoPE) across temporal (T), height (H), and width (W) axes, with independent frequency bases for each. This enables seamless processing of interleaved text, image, and noise tokens, maintaining spatial, temporal, and sequential integrity.
• Processing Streams:
  • Understanding (“clean”): Ingests raw text and image tokens, employing autoregressive decoding for text and bidirectional attention for image.
  • Generation (“noisy”): Handles noise-conditioned image tokens and context, applying pixel-space flow matching for denoising-based generation.
• Unified self-attention masks and complete parameter decoupling per token type (distinct projections, LayerNorms, FFNs) further harmonize modality-specific computation within a shared structure.

2.3 Model Variants

• SenseNova-U1-8B-MoT (Dense): 42 MoT layers (with 4 pre-buffer layers), hidden size 4096, 32/8 heads, single expert per stream.
• SenseNova-U1-30B-A3B-MoT (MoE): 48 MoT layers, hidden size 2048, 32/4 heads, mixture-of-experts routing: 128 experts for understanding, 32 for generation, with top-8 per-token routing (Diao et al., 12 May 2026).

3. Formal Definitions and Training Objectives

3.1 Unified Objective

The process is governed by a joint loss that integrates understanding and generation tasks: $$\mathcal{L}_{\rm total} = \lambda_1\,\mathcal{L}_{\rm Und} + \lambda_2\,\mathcal{L}_{\rm Gen}\quad(\lambda_1=0.1,\;\lambda_2=1.0),$$ where

$$\mathcal{L}_{\rm Und} =-\frac1N\sum_{n=1}^N\log p_\theta(x_n\mid x_{<n},\mathbf{c})$$

is the cross-entropy for next-token prediction, and

$$\mathcal{L}_{\rm Gen} =\mathbb{E}_{t,\mathbf{x},\epsilon}\bigl\|\mathbf{v}_\theta(\mathbf{z}_t,t)-\mathbf{v}^\star\bigr\|_2^2$$

is a pixel-space flow-matching loss parameterized by stochastic sampling over denoising timesteps $t$, with velocity and signal predictions as detailed in the blueprint (Diao et al., 12 May 2026).

3.2 Resolution Adaptation and Guidance

Noise scaling and classifier-free guidance are resolution adaptive:

• Noise schedule: $\sigma_R(H,W) = \sigma_0\sqrt{N(H,W)/N_0}$ with $N(H,W) = (HW) / 32^2$, $N_0=64$.
• During inference, the predicted flow velocity is modified by classifier-free guidance gradients, typically with settings $\gamma=4$, $d$0.

3.3 MoE Gating

For mixture-of-experts (A3B variant), gating per token is based on softmax-normalized scores, with top-8 selection and a balance loss to encourage uniform expert usage: $d$1 with $d$2 in early stages, $d$3 in later training (Diao et al., 12 May 2026).

4. Data Curation and Training Protocol

4.1 Data Sources

NEO-unify relies on a broad, quality-filtered corpus:

• Understanding pretrain (2.5T tokens): includes web text, image–text pairs, document and infographic data.
• Generation pretrain (0.88T image tokens): covers natural images, people, design, and synthetic/infographics, rigorously checked against “what changes / what remains” pairings.
• Interleaved and video: video frames framed as image tokens, plus diverse real-world examples.

4.2 Multi-Stage Training

The pipeline comprises six stages:

1. Understanding Warmup: Freeze generation, fuse Q/K projections, train full model.
2. Generation Pre-train: On frozen understanding branch; multiresolution.
3. Unified Mid-Training: Joint und/gen objectives.
4. Supervised Fine-Tuning: Instruction tuning over joint tasks.
5. RL for T2I: Flow-GRPO with OCR, style, aesthetic rewards.
6. Few-Step Distillation: DMD2 reduces NFE (ODE steps) from 100 to 8 (Diao et al., 12 May 2026).

Optimization uses AdamW, gradient clipping, EMA, and dynamic learning rate schedules.

5. Inference and Sequence Decoding Strategies

• Disaggregated Deployment: LightLLM engine handles multimodal prefill and text decoding, while LightX2V performs pixel-space flow inference; engines share a key/value cache for efficiency.
• Hybrid Attention Kernels: M×T blocks check for image tokens; if present, attention is expanded across modalities. If not, standard causal masks apply.
• Any-to-Image Decoding: Iterative ODE (Euler or DPM-solver) steps, with classifier-free guidance and adaptive renorm.
• Interleaved Generation: Alternating between text autoregression and image denoising within a single generation loop; supports “think patterns” (internal chain-of-thought) for improved coherence and vision-language reasoning.

6. Rationale and Emergent Capabilities

Unified design eliminates semantic–pixel bottlenecks, typically present in models relying on external vision encoders or VAEs. By sharing a transformer backbone across all modalities, NEO-unify ensures that generative and perceptual gradients jointly shape a single representation space. This leads to several key emergent behaviors:

• Vision–Language–Action (VLA): Enables zero-shot robotic manipulation by treating video frames as image tokens, integrated with text instructions.
• World Modeling (WM): Predicts post-action scenes via pixel-space denoising conditioned on action instructions.
• Interleaved Synergy: Benchmarks such as Uni-MMMU and RealUnify demonstrate that generation can assist understanding and vice versa, within a unified pass (Diao et al., 12 May 2026).

A plausible implication is that architectures constructed under the NEO-unify paradigm will accelerate progress toward embodied, goal-directed multimodal intelligence, as future models increasingly avoid modality-bridging artifacts by construction.

7. Comparative Impact and Ecosystem

NEO-unify, as operationalized in SenseNova-U1, matches or exceeds the strongest modular VLMs in both understanding and high-fidelity generation, while offering a compact, extensible codebase suited to diverse research directions. Parameter scales, data regimes, and hyperparameter schedules are detailed in the open-access releases. Modules—including native patch embedding, MoT attention, hybrid attention kernels, and inference engines—are reusable for further model development and downstream application variants.

Within the broader research landscape, NEO-unify supersedes previous "native" vision-LLMs ["From Pixels to Words" (Diao et al., 16 Oct 2025)] by integrating lossless image generation, understanding, and flexible sequence handling, without fragmentation into submodules or reliance on translation between latent spaces.

The paradigm shift advocated by NEO-unify—from bridging between separately trained modality-specific models, to unification via minimal inductive biases and full parameter sharing—redefines the boundary for native, extensible, and interpretable multimodal AI systems (Diao et al., 12 May 2026).