Discrete Native Autoregressive (DiNA)

• DiNA is a multimodal paradigm that converts text, vision, and audio into discrete token streams for unified autoregressive modeling.
• It employs modality-specific tokenizer–detokenizer pairs alongside a decoder-only, modality-agnostic backbone to support both understanding and generation.
• DiNA simplifies architecture design by internalizing non-text modalities as native discrete sequences, enhancing scalability and computational efficiency.

Searching arXiv for the target paper and closely related tokenizer context. Discrete Native Autoregressive (DiNA) is a multimodal modeling paradigm introduced in “LongCat-Next: Lexicalizing Modalities as Discrete Tokens” (Team et al., 29 Mar 2026). In this formulation, text, vision, and audio are represented within a shared discrete token space and modeled under a single next-token prediction objective. The defining premise is that multimodal signals can be treated “natively” as discrete sequences rather than as auxiliary continuous side channels attached to a LLM. In LongCat-Next, this premise is realized through modality-specific tokenizer–detokenizer pairs, a decoder-only modality-agnostic backbone, and a training setup intended to make seeing, painting, and talking instances of one autoregressive process (Team et al., 29 Mar 2026).

1. Definition and conceptual scope

DiNA stands for Discrete Native Autoregressive. Its central operation is to lexicalize non-linguistic modalities into discrete symbols and then apply the same autoregressive machinery used in LLMs to mixed multimodal token streams (Team et al., 29 Mar 2026). The paper characterizes this as extending next-token prediction (NTP) from text to multimodality, with the backbone kept decoder-only and modality-agnostic and with paired tokenizers/detokenizers for each modality.

At the level of sequence modeling, the organizing principle is the standard autoregressive factorization

$p(\mathbf{x}) = \prod_t p(x_t \mid x_{<t}),$

where $\mathbf{x}$ may consist of text tokens, visual tokens, audio tokens, or interleavings of them. In DiNA, the important claim is not merely that multimodal generation is possible, but that multimodal understanding and multimodal generation can both be cast as prediction over one discrete token interface (Team et al., 29 Mar 2026).

This suggests a narrower and more technical meaning of “native multimodality” than is common in language-centric multimodal systems. Here, “native” does not mean only joint training or broad modality coverage; it refers specifically to the internalization of vision and audio as intrinsic elements of the model’s own token language.

2. Motivation relative to language-centric multimodal systems

The motivating contrast in DiNA is between language, which is already discrete and readily tokenized, and modalities such as vision and audio, which are continuous and high-dimensional. The paper argues that the success of NTP in language should extend to other modalities if sufficiently faithful tokenizers can convert them into discrete sequences (Team et al., 29 Mar 2026). On this view, the principal obstacle is not the autoregressive objective itself, but the design of modality-specific tokenization and de-tokenization.

The paper explicitly criticizes the prevailing pattern in which vision or audio is handled by separate branches, projected into a language embedding space, and attached to a language backbone as auxiliary inputs. DiNA instead treats all modalities as internal discrete sequences. The claimed consequences are architectural simplicity, unified understanding and generation, a shared representational space, and infrastructure friendliness, since the system can reuse the scaling and optimization stack of decoder-only LLMs (Team et al., 29 Mar 2026).

A key statement in this framework is that the multimodal modeling problem “reduces to designing modality-specific tokenizer–detokenizer pairs.” That formulation is consequential because it relocates the main research burden away from cross-modal fusion heuristics and toward the fidelity, semantics, and reversibility of discrete token interfaces. A plausible implication is that DiNA is as strong as its tokenizers: improvements in multimodal capability depend directly on how well those tokenizers preserve semantics for understanding while retaining enough structure for reconstruction and generation.

3. Token interfaces and the shared autoregressive backbone

LongCat-Next instantiates DiNA through a modular architecture composed of standard text tokenization, a visual tokenizer called dNaViT, an RVQ-based audio tokenizer, and a decoder-only MoE backbone that processes all resulting token streams (Team et al., 29 Mar 2026). The backbone is described as modality-agnostic, while tokenizers and detokenizers perform the modality-specific work.

The paper emphasizes that multimodal embeddings are randomly initialized and jointly trained, rather than injected as fixed continuous projections into an existing LLM. This design is central to the “discrete native” claim: visual and audio symbols are not external descriptors consumed by a text model, but part of the model’s own learned discrete vocabulary.

The architecture can therefore be summarized as a sequence pipeline: raw inputs in text, image, or audio form are converted into discrete token streams; these streams are modeled by a shared decoder-only MoE; and modality-specific decoders or detokenizers produce text, images, or speech outputs as required (Team et al., 29 Mar 2026). The conceptual significance is that the same backbone is used regardless of whether the task is image understanding, text-to-image generation, ASR, TTS, or audio-conditioned dialogue.

The paper also notes that modality-agnostic MoE layers nevertheless develop some functional specialization during multimodal training. This is important because it indicates that DiNA does not require explicit modality routing to induce differentiated computation, even though specialized behavior can still emerge.

4. dNaViT and discrete visual tokenization

The visual tokenizer in LongCat-Next is dNaViT, expanded as Discrete Native Any-resolution Visual Transformer and also described later as Discrete Native-Resolution Vision Transformer (Team et al., 29 Mar 2026). Its purpose is to convert images into hierarchical discrete tokens that are adequate for both visual understanding and image generation, while operating at arbitrary input resolutions.

The paper frames the core requirement as semantic completeness. Formally, if $I$ denotes the input image and $z = \{idx_1, idx_2, \dots, idx_n\}$ the discrete visual indices, then the target condition is

$$\mathcal{P}(A \mid z, \mathcal{Q}) \approx \mathcal{P}(A \mid I, \mathcal{Q}),$$

where $\mathcal{Q}$ is an image-centric query and $A$ is the response (Team et al., 29 Mar 2026). In other words, downstream behavior conditioned on the discrete tokens should approximate downstream behavior conditioned on the original image.

To produce the pre-quantization representation, the paper uses a Semantic-and-Aligned Encoder (SAE). The SAE is intended to generate continuous latents $z_p$ that are semantically rich and aligned with language, preserving information useful for captioning, OCR, QA, and visual reasoning. The tokenizer then applies Residual Vector Quantization (RVQ) in multiple stages, so that early codebooks capture coarser semantic structure and later codebooks retain finer detail (Team et al., 29 Mar 2026). The paper states that this hierarchy is crucial for balancing understanding and generation.

The quantization objective includes both a commitment loss and a semantic reconstruction loss based on cosine similarity, and the codebook is updated with EMA rather than direct gradients (Team et al., 29 Mar 2026). This makes the tokenizer a form of semantically regularized compression rather than a purely reconstructive codec. The intended distinction from conventional tokenizers is therefore fourfold: semantic completeness for understanding, generative sufficiency for reconstruction, hierarchical discretization to reduce information loss, and native-resolution processing for arbitrary-size inputs.

The paper contrasts dNaViT with low-level reconstructive models like VAE/VQ-VAE, semantic encoders like DINOv2/SigLIP, and raw-pixel tokenization methods (Team et al., 29 Mar 2026). That contrast is not presented as a benchmark taxonomy with formal equivalence classes, but as a statement of design intent: dNaViT seeks a compromise between semantic abstraction and recoverable image structure.

5. Any-resolution processing and de-tokenization

A major design claim is that dNaViT works at native resolution or any resolution and therefore avoids distortions associated with forced resizing or cropping (Team et al., 29 Mar 2026). The paper states that images are flattened into variable-length sequences and processed with variable-length FlashAttention, allowing arbitrary input sizes. This design positions visual tokenization as sequence formation without a fixed-resolution bottleneck.

After tokenization, the SAE and codebook are frozen, and image reconstruction proceeds through a pixel decoder followed by a lightweight refiner trained with flow matching (Team et al., 29 Mar 2026). The pixel decoder is described as ViT-based and optimized to reconstruct spatial layout, object structure, and semantic content, while the refiner improves high-frequency details and sharpness. The decoder loss is given as a weighted sum of pixel, perceptual, and alignment terms.

The paper further argues that reconstruction remains possible not only because the encoder is semantically aligned, but also because residual encoder architectures preserve low-level signals through residual pathways (Team et al., 29 Mar 2026). This is presented through a residual-stack expression for $\mathbf{z}_p$, with the intended interpretation that pixel-level information can remain recoverable even when the encoder is principally optimized for semantic objectives.

This suggests that dNaViT’s de-tokenization strategy depends on two coupled assumptions. First, semantic alignment should make the token sequence useful for downstream reasoning. Second, the structure of the encoder and the residual quantization hierarchy should preserve enough recoverable detail to support image synthesis. The paper’s claim is that both are required if a single discrete interface is to support understanding and generation.

6. Audio tokenization and multimodal unification

DiNA extends the same discrete-native logic to audio. In LongCat-Next, audio is processed with a Whisper encoder for feature extraction, followed by downsampling and an 8-layer RVQ tokenizer (Team et al., 29 Mar 2026). The resulting audio tokenizer compresses waveforms into discrete tokens at 12.5 Hz.

The audio detokenizer consists of a decoder that reconstructs coarse Mel spectrograms, a flow-matching refiner, and a vocoder (Team et al., 29 Mar 2026). The audio training objective is described as a weighted combination of reconstruction, commitment, and LLM losses. The setup is intended to support both speech understanding and text-guided speech generation.

The paper also introduces text-guided audio using aligned text/audio segments and stochastic delays, enabling both parallel generation and serial generation (Team et al., 29 Mar 2026). This is important within DiNA because it shows that the framework is not restricted to static modality conversion; it also accommodates temporal generation strategies within the same discrete sequence paradigm.

In the broader architecture, vision and audio are therefore not handled as exceptional subsystems. They are both brought into the same tokenized autoregressive substrate as text. A plausible implication is that DiNA treats modality boundaries as differences in token formation and de-tokenization, rather than as differences in the core predictive mechanism.

7. Training procedure, empirical behavior, and limitations

The training pipeline in LongCat-Next is organized into three phases: tokenizer training, a pre-align stage in which embeddings and DepthTransformer decoders are warmed up while the language backbone is frozen, and full multimodal training with the full model unfrozen (Team et al., 29 Mar 2026). For vision, the paper highlights RVQ tokenization, multi-level token embeddings summed together, and a DepthTransformer for decoding multi-level outputs. For audio, it uses 8 RVQ layers, a decoder warm-up stage, joint semantic-acoustic training, and decoder fine-tuning.

A central empirical claim concerns the long-standing tension between visual understanding and visual generation in discrete modeling. The paper states that these are reformulated as “two instances of the same predictive process, differing only in their conditional priors” (Team et al., 29 Mar 2026). In the reported experiment, a unified model trained on mixed data has loss only marginally worse than a pure understanding model and better than a pure generation model, which the authors interpret as evidence that generation does not degrade understanding and that understanding can help generation.

The reported benchmark summary spans text, vision, and audio. For visual understanding, LongCat-Next is reported at MathVista 83.1, MathVision 64.7, MMMU 70.6, MMMU-Pro 60.3, VisuLogic 29.4, OCRBench 86.5, DocVQA 94.2, and ChartQA 88.0 (Team et al., 29 Mar 2026). For visual generation, the paper reports GenEval 84.44, DPG 84.66, LongText-EN 93.15, LongText-ZH 89.08, and CVTG-2K 76.36. It further claims strong performance across ASR, TTS, audio understanding, and audio-to-text chat, while also preserving strong text capabilities such as good agentic tool use, solid coding, and strong general knowledge benchmarks.

Several ablations refine the interpretation. The paper compares discrete dNaViT with a continuous version and finds that discrete training initially has higher loss, that a Pre-Buffer module helps, and that larger data and longer training close the gap, so that discrete performance approaches continuous performance (Team et al., 29 Mar 2026). It also reports that parallel generation is faster while serial generation tends to be semantically cleaner in audio, with both made workable by the unified delay-based training. In addition, t-SNE analyses are said to indicate a more interwoven modality embedding space than in non-native or partially aligned baselines.

The paper is explicit that DiNA, as implemented in LongCat-Next, is an initial step rather than a complete solution (Team et al., 29 Mar 2026). The stated limitations include incomplete tokenizer optimization, a detokenizer that emphasizes semantic consistency more than pixel fidelity, a present focus on image-to-text and text-to-image rather than general any-to-any multimodal interaction, and unresolved questions about data scaling, long-context multimodal interaction, multi-turn grounded dialogue, compositional generation, and the co-design of data, objectives, and discretization. These limitations are integral to the definition of DiNA’s current research status: the framework is presented as a demonstration that discrete-native multimodality is viable to a significant extent, not as a closed formulation of multimodal intelligence.