LongCat-Next: Native Multimodal LLM

• LongCat-Next is a native multimodal system that integrates text, vision, and audio as discrete tokens under a unified autoregressive framework.
• It leverages the DiNA approach with dNaViT and RVQ tokenizers to enable any-resolution, high-quality processing for image, audio, and text tasks.
• The unified backbone achieves competitive performance across OCR, ASR, and generation tasks, illustrating practical applications and scalable multimodal learning.

LongCat-Next is a large native multimodal foundation model that brings text, vision, and audio under the same next-token prediction paradigm by lexicalizing all three modalities as discrete tokens and training a single decoder-only backbone on unified token sequences. Introduced together with the Discrete Native Autoregressive framework, or DiNA, and the Discrete Native Any-resolution Visual Transformer, or dNaViT, it is presented as an industrial-strength model that can “see, paint, and talk” within one architecture, using minimal modality-specific design while open-sourcing both the backbone and tokenizers (Team et al., 29 Mar 2026).

1. Historical positioning and system identity

Before becoming the name of a released model, “LongCat-Next” appeared across the LongCat literature as a forward-looking designation for a successor system. In those earlier reports, the label was associated with several different but compatible trajectories: an ultra-low-bitrate speech codec layer for end-to-end speech LLMs, an agentic 560B MoE trained with DORA and Heavy Thinking, efficient long-context attention via LoZA, and an embedding-scaled 68.5B MoE backbone with roughly 3B activated parameters (Zhao et al., 17 Oct 2025, Team et al., 23 Jan 2026, Zhang et al., 30 Dec 2025, Liu et al., 29 Jan 2026).

The 2026 multimodal paper turns that prospective label into a concrete system. Its backbone is LongCat-Flash-Lite A3B, a decoder-only MoE Transformer with 68.5B total parameters and approximately 3B active parameters per token. The model uses Zero-Expert and Shortcut MoE designs from LongCat-Flash, but otherwise introduces no modality-aware architectural modifications: the same backbone is used for pure text and multimodal training. The only modality-specific elements are the tokenizer-detokenizer pairs, the visual and audio discrete-token embedding tables, and modality-specific heads, including DepthTransformer heads for multi-level visual and audio decoding (Team et al., 29 Mar 2026).

This positioning is important because LongCat-Next is not merely a multimodal adaptor grafted onto a text LLM. It is defined as a “native multimodal” system in which non-linguistic inputs are represented as first-class sequence elements rather than as continuous side channels. In the LongCat family, that makes it the canonical realization of the project’s native multimodality program.

2. Discrete Native Autoregression

DiNA reframes multimodal modeling as a single discrete autoregressive problem. Let $x=(x_1,\dots,x_T)$ be a unified sequence whose elements may be text tokens, visual code indices, or audio code indices. LongCat-Next parameterizes the joint distribution as

$$p_\theta(x_1,\dots,x_T)=\prod_{t=1}^T p_\theta(x_t\mid x_{<t}),$$

with one next-token objective across modalities rather than separate discriminative and generative heads for different media (Team et al., 29 Mar 2026).

This design opposes the common “language + attachments” pattern in multimodal systems. Under that older pattern, images and audio are encoded into continuous features and injected into the LLM through cross-attention, projection layers, or adapters, while the underlying autoregressive objective remains text-centric. LongCat-Next instead treats all modalities as native to the same discrete space. The tokenizers and detokenizers live at the system boundary; the backbone itself only sees discrete IDs.

The training loss is the standard cross-entropy objective,

$$\mathcal{L}_\text{NTP} = -\mathbb{E}_{x_{1:T}} \sum_{t=1}^T \log p_\theta(x_t \mid x_{<t}),$$

but the representational semantics are richer than a literal shared vocabulary would suggest. Visual and audio RVQ levels are present as discrete indices, yet for understanding tasks the model additively combines multi-level codes into a single embedding per position, while for generation a DepthTransformer reconstructs multi-level codes in parallel from one LLM step. This preserves the computational profile of text-style autoregression while exposing an exponentially large representational space for image and audio synthesis (Team et al., 29 Mar 2026).

A central conceptual claim follows. Under DiNA, understanding and generation differ in conditioning pattern rather than in architectural kind: VQA, captioning, T2I, ASR, TTS, audio QA, and audio-chat are all realized as token prediction over differently arranged multimodal sequences. This is the sense in which LongCat-Next treats multimodality as native rather than attached.

3. Vision and audio tokenization

The viability of DiNA depends on whether non-text modalities can be discretized without destroying semantics. LongCat-Next answers this with two tokenizer families: dNaViT for vision and an 8-layer RVQ audio tokenizer for speech and audio (Team et al., 29 Mar 2026).

For images, dNaViT is built around a Semantic-and-Aligned Encoder, reusing Qwen2.5-ViT as a frozen SAE. The requirement is “semantic completeness”: if an image $I$ is mapped to a discrete representation $z=Q(I)$, then for image-centric tasks $\mathcal{Q}$ with answers $A$, one should have $$P(A\mid z,\mathcal{Q}) \approx P(A\mid I,\mathcal{Q})$$. In practice, the SAE output is projected to $\mathbf{r}_0=f_{\text{proj}}(\mathbf{z})$, then quantized through residual vector quantization,

$$\hat{\mathbf{q}}_l = \operatorname{VQ}(\mathbf{r}_{l-1}),\qquad \mathbf{r}_l = \mathbf{r}_{l-1} - \hat{\mathbf{q}}_l,$$

with aggregated latent $$p_\theta(x_1,\dots,x_T)=\prod_{t=1}^T p_\theta(x_t\mid x_{<t}),$$0. Codebooks use EMA updates, and the tokenizer is trained at native resolution with images up to $$p_\theta(x_1,\dots,x_T)=\prod_{t=1}^T p_\theta(x_t\mid x_{<t}),$$1 and sequence length up to 8192. The released configuration uses 8 RVQ levels with 16,384 entries each for the LLM embeddings, together with 28× spatial compression, a 400M-parameter pixel decoder, and a flow-matching refiner initialized from OmniGen2 (Team et al., 29 Mar 2026).

Two subtleties are central. First, dNaViT is any-resolution rather than fixed-resolution; it operates on SAE features at native resolution and relies on variable-length sequence modeling. Second, the codebook embeddings used by the LLM are not treated as fixed semantics. The assignments are fixed by quantization, but the embeddings themselves are randomly initialized and trained end-to-end with the backbone. The model therefore learns what the paper calls its own native visual representation language on top of discrete IDs (Team et al., 29 Mar 2026).

For audio, LongCat-Next uses a Whisper-large-v3 encoder with 8-layer RVQ at 12.5 Hz, with codebook sizes $$p_\theta(x_1,\dots,x_T)=\prod_{t=1}^T p_\theta(x_t\mid x_{<t}),$$2, followed by a decoder, a flow-matching refiner, and a vocoder. Audio training uses

$$p_\theta(x_1,\dots,x_T)=\prod_{t=1}^T p_\theta(x_t\mid x_{<t}),$$3

where $$p_\theta(x_1,\dots,x_T)=\prod_{t=1}^T p_\theta(x_t\mid x_{<t}),$$4 is an LLM-based understanding loss computed through a small frozen LLM during tokenizer training. The system supports ASR, TTS, audio QA, and audio-chat, with modality markers AS, AE, and TE delimiting audio and text segments. Text-guided audio is trained in both serial and parallel regimes, unified through random delay sampling between the first text token and the first audio token (Team et al., 29 Mar 2026).

Within the broader family, this audio design contrasts with the earlier LongCat-Audio-Codec, which was explicitly proposed as the audio front-end most likely to underlie a future “LongCat-Next” style system and emphasized semantic-acoustic separation, ultra-low bitrate, and streaming synthesis at 16.67 Hz and 0.43–0.87 kbps. The released LongCat-Next instead adopts a Whisper-based 12.5 Hz RVQ tokenizer, indicating that the family’s multimodal path remained open at the tokenizer level rather than fixed to a single codec lineage (Zhao et al., 17 Oct 2025).

4. Backbone and training regime

LongCat-Next trains the tokenizers separately, then performs native multimodal training on the LongCat-Flash-Lite A3B backbone. The full multimodal model uses approximately 2T pretraining tokens and a four-stage curriculum that begins with alignment and ends with 64K-token instruction tuning (Team et al., 29 Mar 2026).

Stage	Trainable parameters	Sequence / batch
Pre-align	Modality embeddings and DepthTransformers	8k / 8192
Pre-training	All parameters	8k / 8192
Mid-training	All parameters	32k / 1024
SFT	All parameters	64k / 128

The data mixture expands with each stage. Pre-align focuses on image-caption, pure audio, ASR, TTS, and interleaved multimodal data while freezing the backbone. Pre-training unfreezes all parameters and adds interleaved image-text, OCR, grounding, fixed-resolution generation, and pure text. Mid-training introduces long CoT, generic QA, video, GUI tasks, arbitrary-resolution generation, high-quality audio, and multi-turn dialogue. SFT then concentrates on high-quality multimodal instructions, reasoning-heavy data, arbitrary-resolution generation, audio instruction tuning, and long pure-text data (Team et al., 29 Mar 2026).

The infrastructure is equally specific. The paper describes profile-guided VHalf-based pipeline parallelism in which the embedding layer and modality-specific heads share one pipeline stage, the LLM head is placed on a separate stage, and the remaining Transformer layers are evenly distributed. This arrangement reduces cross-stage communication overhead and mitigates pipeline bubbles during multimodal training (Team et al., 29 Mar 2026).

Because the backbone is LongCat-Flash-Lite A3B, the multimodal system inherits a model family already designed around embedding scaling and sparse activation. LongCat-Flash-Lite allocates over 30B parameters to embeddings and reports exceptional competitiveness in agentic and coding domains while using approximately 3B activated parameters per token. That architecture becomes the substrate on which LongCat-Next layers DiNA-style native multimodality (Liu et al., 29 Jan 2026).

5. Empirical profile across modalities

LongCat-Next is evaluated as a unified model rather than as separate specialist subsystems, and the reported numbers are distributed across vision understanding, image generation, audio, and pure text (Team et al., 29 Mar 2026).

In visual understanding, the model reports MMMU 70.6, MMMU-Pro 60.3, MathVista 83.1, MathVision 64.7, VisuLogic 29.4, and BabyVision 14.4. On OCR and document tasks it reports OmniDocBench-en 0.152, OmniDocBench-zh 0.226, and OCRBench 86.5, alongside strong DocVQA, InfoVQA, CharXiv, and ChartQA results. In general VQA and GUI grounding it reports MMStar 69.3, RealWorldQA 72.0, CountBench 82.1, OSWorld-G 58.3, and ScreenSpot-V2 88.3 (Team et al., 29 Mar 2026).

In visual generation, LongCat-Next reports GenEval 84.44, DPG 84.66, WISE 57.0, LongText-EN 93.15, LongText-ZH 89.08, TIFF 82.85 / 84.38, and CVTG 76.36. The paper emphasizes that these results are competitive not only against unified multimodal models such as Janus-Pro, Show-o2, OneCAT, BAGEL, NEO-unify, Ovis-U1, Lumina, OmniGen2, UniWorld-V1, X-Omni, InternVL-U, and BLIP3-o, but also against specialized T2I systems such as Emu-3.5, Qwen-Image-2507, Gemini 2.5 Flash Image, FLUX.1-dev, and SeedDream 3.0 (Team et al., 29 Mar 2026).

In audio, the reported summary is similarly broad. The model is described as competitive or near state-of-the-art in WER across LibriSpeech, AISHELL-1/2, FLEURS, and Wenetspeech; it reports MMAU 76.4; and it is evaluated against Gemini-3.1/2.5 Flash-Lite, Qwen3-Omni-A3B, MiMo-Audio, Kimi-Audio, and Step-Audio-2-mini on ASR, TTS, audio understanding, and audio-to-text chat. The paper further states that on SeedTTS zh/en WER it is better than most, with Qwen3-Omni sometimes stronger (Team et al., 29 Mar 2026).

Pure text performance remains strong despite the multimodal objective. Reported scores include MMLU 83.95, MMLU-Pro 77.02, CEval 86.8, and CMMLU 82.13. In agentic and coding evaluations it reports Tau2-Telecom 62.06 versus 4.39 for Qwen3-Omni in the cited comparison, SWE-Bench 43.0, and TerminalBench 18.75. The paper interprets this as evidence that multimodality under DiNA does not impose a “multimodal tax” on language intelligence and may encourage better generalization through richer cross-modal training (Team et al., 29 Mar 2026).

6. Discreteness, understanding, and generation

LongCat-Next directly addresses two persistent objections to discrete multimodal modeling: that discrete vision has a hard understanding ceiling, and that a single representation cannot serve both understanding and generation effectively (Team et al., 29 Mar 2026).

On the first point, the paper treats “breaking the discrete understanding ceiling” as a central claim. Historically, VQ-VAE-style discrete image models were thought to underperform continuous visual encoders on tasks requiring OCR, diagram reasoning, or other high-level semantics. LongCat-Next reports strong STEM, OCR, and document-understanding scores despite 28× compression, and ablations at the Qwen-7B scale reportedly shrink the discrete-versus-continuous gap to roughly 1% once training data and a small Pre-Buffer re-encoding FFN are sufficient (Team et al., 29 Mar 2026).

On the second point, the model is explicitly designed to reconcile understanding and generation. The paper describes controlled experiments in which pure understanding training, pure generation training, and a unified 50/50 mix are compared under equal effective token budgets. The unified model reportedly matches understanding loss and improves generation loss over pure-generation training; its loss is described as only approximately 0.006 higher than pure understanding while approximately 0.02 lower than pure generation. The argument is not that the conflict vanishes in principle, but that under SAE + RVQ tokenization, additive multi-level embeddings for understanding, and DepthTransformer-based parallel decoding for generation, the conflict is substantially mitigated (Team et al., 29 Mar 2026).

This matters beyond LongCat-Next itself. Related LongCat reports define other ingredients of a broader roadmap: DORA-based asynchronous RL and Heavy Thinking for agentic reasoning, 1M-token long-context scaling via LoZA, and LongCat-Video’s continuation-based path toward world models (Team et al., 23 Jan 2026, Zhang et al., 30 Dec 2025, Team et al., 25 Oct 2025). This suggests that the released LongCat-Next is best read as the native multimodal branch of a larger LongCat program rather than as an isolated multimodal endpoint.

7. Limitations and future directions

The LongCat-Next paper is explicit that several parts of the system remain unfinished. The vision tokenizer is “not fully optimized”: dNaViT prioritizes semantic reconstruction and any-resolution support rather than maximal pixel fidelity, and the SAE is reused rather than retrained from scratch under a DiNA-specific objective. More complex any-to-any multimodal reasoning is only partially explored; current experiments focus on image-to-text, text-to-image, ASR/TTS, and audio QA rather than arbitrary multimodal transformations across many turns (Team et al., 29 Mar 2026).

The paper also identifies unresolved issues around data and representation scaling. It states that it remains unclear how much multimodal data can add beyond text for general intelligence, and that the interaction between continuous pretraining and discrete modeling is underexplored. On the RL side, multimodal RL is described as difficult because image understanding suffers from entropy explosion and training-inference mismatch, while multimodal reward modeling is vulnerable to reward hacking (Team et al., 29 Mar 2026).

The future directions listed are correspondingly concrete: improved tokenizers; better SAE encoders co-designed for discretization; more powerful RVQ with adaptive quantization and better codebook utilization; reconstruction-focused variants preserving high-frequency detail; fully general any-to-any multimodal generation; long-context multimodal dialogue with interleaved visual and audio contexts; carefully curated multimodal corpora; co-design of continuous and discrete pretraining; and GRPO-based RL for more multimodal tasks such as image editing, video, and grounded multi-step problems (Team et al., 29 Mar 2026).

Broader LongCat work points toward adjacent expansions. LongCat-Video explicitly names better physical knowledge modeling, multi-modal memory integration in video generation, and incorporation of knowledge from LLMs and MLLMs as future work, while LoZA and LongCat-Flash-Thinking-2601 establish paths toward 1M-token contexts, environment scaling, and Heavy Thinking for agentic reasoning (Team et al., 25 Oct 2025, Team et al., 23 Jan 2026, Zhang et al., 30 Dec 2025). A plausible implication is that later LongCat systems may attempt to fuse DiNA-style native tokenization with stronger agentic RL, longer contexts, and richer world modeling, but the released LongCat-Next is already a distinct result: a single autoregressive model in which text, vision, and audio are all rendered as discrete sequence elements and trained under one objective.