dNaViT: Discrete Native Any-resolution Transformer

• The paper introduces dNaViT as a visual tokenizer that discretizes native-resolution images using a language-aligned encoder and hierarchical RVQ for seamless integration with autoregressive frameworks.
• It employs native-resolution processing by flattening arbitrary image grids into token sequences, effectively preserving both high-level semantics and low-level details.
• The design unifies understanding and generation tasks within a single model, achieving competitive performance on vision, VQA, and text-to-image benchmarks.

Searching arXiv for the specified papers and closely related work on dNaViT/NaViT/ViQ. Discrete Native Any-resolution Visual Transformer (dNaViT) denotes a class of visual tokenization and de-tokenization mechanisms that convert images at arbitrary native resolutions into discrete token sequences suitable for autoregressive multimodal modeling. In "LongCat-Next: Lexicalizing Modalities as Discrete Tokens," dNaViT is introduced as a key innovation inside the Discrete Native Autoregressive (DiNA) framework, where vision, audio, and text are all represented as sequences of discrete tokens under one next-token-prediction objective (Team et al., 29 Mar 2026). In that formulation, dNaViT combines a Semantic-and-Aligned Encoder (SAE), hierarchical Residual Vector Quantization (RVQ), any-resolution flattening, and a paired de-tokenizer. Related work places the idea in the broader lineage of Native Resolution ViTs, especially NaViT’s "Patch n’ Pack" design for arbitrary aspect ratios and resolutions (Dehghani et al., 2023), while ViQ presents another dNaViT formulation centered on text-aligned visual quantized representations at any resolution (Yu et al., 25 Jun 2026).

1. Conceptual lineage and motivation

The immediate motivation for dNaViT in LongCat-Next is the DiNA objective: to treat vision, audio, and text all as sequences of discrete tokens under one next-token-prediction objective (Team et al., 29 Mar 2026). The design target is explicitly analogous to subword tokenizers in language modeling: subword tokenizers achieve near-lossless encoding with strong semantics and faithful reconstruction, and dNaViT is designed to play the same role for images. Within this framing, two core hurdles in discrete vision modeling are identified: representation capacity, termed "semantic completeness," and information loss in quantization.

LongCat-Next addresses those two hurdles in distinct ways. It addresses representation capacity by building on a Semantic-and-Aligned Encoder, defined as a vision transformer pretrained with large-scale vision-language objectives so that its latent features already carry rich, language-aligned semantics. It addresses information loss by applying multi-stage Residual Vector Quantization to hierarchically quantize SAE features with far less fidelity loss than single-stage VQ (Team et al., 29 Mar 2026). The result is a discrete interface intended to support both any-resolution understanding and any-resolution generation.

The broader lineage comes from NaViT. NaViT shows that resizing images to a fixed resolution before processing is not necessary for ViT-based models, and instead uses sequence packing, masked self-attention, masked pooling, and factorized fractional positional embeddings to process arbitrary resolutions and aspect ratios (Dehghani et al., 2023). That work does not itself define the LongCat-Next tokenization stack, but it establishes the native-resolution premise from which dNaViT derives its "any-resolution" character.

ViQ extends the same general direction but with a different discretization strategy. Its dNaViT is built around a standard ViT backbone—specifically the SigLIP2-g model—with dynamic position handling, 2D Rotary Position Embedding in the quantization stage, and Finite Scalar Quantization (Yu et al., 25 Jun 2026). This suggests that "dNaViT" is not a single fixed architecture but a design family: native-resolution visual transformers that produce discrete visual codes while attempting to preserve both semantics and low-level detail.

2. Core architecture in LongCat-Next

In LongCat-Next, dNaViT consists of three modules: an SAE encoder, a hierarchical RVQ tokenizer, and a paired de-tokenizer comprising a Vision-Transformer pixel-decoder plus a lightweight flow-matching refiner (Team et al., 29 Mar 2026). The input image is denoted $I \in \mathbb{R}^{H \times W \times 3}$. A pretrained Semantic-and-Aligned Encoder maps the image to continuous features

$$z_p = E_{\mathrm{sae}}(I) \in \mathbb{R}^{N \times D},$$

where $N = (H \cdot W)/(p^2)$ is the number of $p \times p$ patches at native resolution. The summary states that, because of residual connections, $z_p$ retains both low-level detail and high-level semantics.

These encoder features are then projected into the RVQ input space,

$$r_0 = f_{\mathrm{proj}}(z_p) \in \mathbb{R}^{N \times D},$$

and quantized recursively over $L$ residual levels: $$\hat q_l = VQ_l(r_{l-1}), \qquad r_l = r_{l-1} - \hat q_l.$$ The final quantized representation is the sum of all codebook outputs,

$\hat z = \sum_{l=1}^L \hat q_l.$

Each stage $VQ_l$ uses its own codebook of $$z_p = E_{\mathrm{sae}}(I) \in \mathbb{R}^{N \times D},$$0 embeddings, updated by exponential moving average, and yields a token index $$z_p = E_{\mathrm{sae}}(I) \in \mathbb{R}^{N \times D},$$1 for each spatial position $$z_p = E_{\mathrm{sae}}(I) \in \mathbb{R}^{N \times D},$$2 and level $$z_p = E_{\mathrm{sae}}(I) \in \mathbb{R}^{N \times D},$$3 (Team et al., 29 Mar 2026).

Any-resolution support is implemented by operating at the encoder’s native spatial resolution, without forced resizing, and flattening arbitrary $$z_p = E_{\mathrm{sae}}(I) \in \mathbb{R}^{N \times D},$$4 patch grids into token sequences. The summary describes this as "pack-and-pack": given any $$z_p = E_{\mathrm{sae}}(I) \in \mathbb{R}^{N \times D},$$5, the $$z_p = E_{\mathrm{sae}}(I) \in \mathbb{R}^{N \times D},$$6 patches and their $$z_p = E_{\mathrm{sae}}(I) \in \mathbb{R}^{N \times D},$$7-level quantization indices are flattened into a length-$$z_p = E_{\mathrm{sae}}(I) \in \mathbb{R}^{N \times D},$$8 discrete token sequence, which can be processed with variable-length FlashAttention. De-tokenization reverses this pipeline. A pixel decoder, described as a 400 M-parameter Vision Transformer, reverses the pack-and-unmerge operation to recover an $$z_p = E_{\mathrm{sae}}(I) \in \mathbb{R}^{N \times D},$$9 feature grid and then projects to RGB via a linear head. A flow-matching refiner then adds back fine-grained textures, conditioned on both the decoder output and the same discrete embeddings.

The architectural logic is therefore bidirectional. The encoder-plus-RVQ stack produces discrete visual tokens suitable for shared autoregression with text and audio, while the decoder-plus-refiner stack maps those tokens back to pixels. A plausible implication is that dNaViT is intended not merely as a compression module, but as the visual analogue of a tokenizer/de-tokenizer pair in language modeling.

3. Mathematical specification and token hierarchy

The LongCat-Next formulation gives an explicit mathematical specification of arbitrary-resolution tokenization and de-tokenization (Team et al., 29 Mar 2026). For the quantization stage, the nearest-neighbor assignment is written as

$N = (H \cdot W)/(p^2)$0

with

$N = (H \cdot W)/(p^2)$1

EMA codebook updates for entry $N = (H \cdot W)/(p^2)$2 are given by

$N = (H \cdot W)/(p^2)$3

$N = (H \cdot W)/(p^2)$4

$N = (H \cdot W)/(p^2)$5

The quantizer loss is

$N = (H \cdot W)/(p^2)$6

where

$N = (H \cdot W)/(p^2)$7

and

$N = (H \cdot W)/(p^2)$8

The de-tokenizer loss is

$N = (H \cdot W)/(p^2)$9

and the flow refiner uses

$p \times p$0

The hierarchical token organization is central to the model’s dual use. The $p \times p$1 residual levels form a hierarchy from coarse $p \times p$2 to fine $p \times p$3, so each spatial patch $p \times p$4 is represented by an $p \times p$5-tuple of indices $p \times p$6. For understanding, per-patch embeddings $p \times p$7 are summed across levels to form a single vector and then linearly projected into the LLM’s embedding space. The summary states that this additive fusion ensures all semantic granularity is preserved. For generation, the shared LLM output hidden state is sent to a DepthTransformer head, which in one step emits all $p \times p$8 levels of indices for the next patch in parallel. This is described as an "exponential" vocabulary that yields very large capacity while keeping inference as fast as a single step per patch (Team et al., 29 Mar 2026).

This hierarchy distinguishes dNaViT from single-stage discrete tokenizers. The representation is not a single code per patch, but a structured coarse-to-fine code stack. That design is directly tied to the stated goal of reconciling understanding and generation within the same discrete interface.

4. Native-resolution processing and relation to NaViT

The "any-resolution" property of dNaViT is best understood against the NaViT baseline. NaViT processes arbitrary resolutions and aspect ratios by using a ViT backbone with two minimal changes: masked self-attention or masked pooling, and factorized fractional positional embeddings (Dehghani et al., 2023). Images are tiled into non-overlapping patches, and multiple variable-size images are greedily packed into a single sequence of fixed maximum length $p \times p$9. Attention leakage across images is prevented by an attention mask $z_p$0, defined as $z_p$1 when two tokens belong to the same image and $z_p$2 otherwise. Positional information is encoded via fractional coordinates

$z_p$3

with

$z_p$4

Because $z_p$5 operates on $z_p$6, NaViT can be evaluated at arbitrarily large $z_p$7 and $z_p$8 without modifying weights (Dehghani et al., 2023).

LongCat-Next’s dNaViT reuses the native-resolution premise, but its implementation focus is different. Instead of continuous positional interpolation being the main novelty, the crucial step is flattening native-resolution SAE features and their hierarchical quantization indices into a discrete token sequence via "pack-and-pack" (Team et al., 29 Mar 2026). In effect, NaViT supplies the conceptual substrate for arbitrary-resolution sequence modeling, whereas dNaViT repurposes that substrate for discrete multimodal autoregression.

NaViT’s own summary also sketches what a discrete dNaViT could look like: a model restricted to a finite set of $z_p$9 discrete resolutions or aspect-ratio pairs, using pure lookup tables rather than continuous $$r_0 = f_{\mathrm{proj}}(z_p) \in \mathbb{R}^{N \times D},$$0. That sketch includes discrete resolution bins, discrete 1D positional tables, unchanged packing and mask logic, and identifies three open challenges: discretization error, table size, and generalization (Dehghani et al., 2023). LongCat-Next’s realized dNaViT differs from that sketch by relying on native-resolution SAE features and hierarchical RVQ rather than a fixed-table discretization of NaViT’s positional mechanism.

A plausible implication is that "dNaViT" has two related but distinct meanings in the literature: first, a generic idea of discretizing native-resolution ViT processing; second, a specific visual tokenizer/de-tokenizer architecture inside DiNA.

5. Training objectives and multimodal integration

LongCat-Next trains dNaViT in two stages (Team et al., 29 Mar 2026). In Stage A, tokenizer training, the SAE and RVQ are trained jointly with $$r_0 = f_{\mathrm{proj}}(z_p) \in \mathbb{R}^{N \times D},$$1, while the pixel decoder and refiner are trained with $$r_0 = f_{\mathrm{proj}}(z_p) \in \mathbb{R}^{N \times D},$$2. The SAE is frozen at quantizer training time, according to the architectural summary. In Stage B, native multimodal training, the procedure first performs a pre-align phase: the LLM is frozen, and the codebook embeddings and DepthTransformer are warmed up on vision token reconstruction. End-to-end training then freezes the quantizer and decoder, and trains the LLM together with vision and audio embeddings and the DepthTransformer head using the standard autoregressive loss

$$r_0 = f_{\mathrm{proj}}(z_p) \in \mathbb{R}^{N \times D},$$3

Discrete vision tokens $$r_0 = f_{\mathrm{proj}}(z_p) \in \mathbb{R}^{N \times D},$$4 are concatenated with text and audio tokens into one sequence, and the shared Transformer backbone predicts each next token under this cross-entropy objective.

The empirical pre-align findings are specific. Under the same captioning proxy task, vanilla discrete, with no Pre-Buffer, started with a $$r_0 = f_{\mathrm{proj}}(z_p) \in \mathbb{R}^{N \times D},$$5 higher loss than continuous features. Introducing a one-layer Pre-Buffer FFN after codebook lookup closed this gap by 50% in a few epochs. Scaling up data to $$r_0 = f_{\mathrm{proj}}(z_p) \in \mathbb{R}^{N \times D},$$6 vision tokens drove discrete loss within 1% of continuous (Team et al., 29 Mar 2026). These results are relevant because they quantify the discrete-continuous alignment problem in the transition from continuous vision encoders to shared discrete autoregression.

The same source reports an intrinsic information recovery result: even random ViT-Base encoders reconstruct images with PSNR 30.5, far better than CLIP-style pretrained ViTs, which is used to justify freezing SAE during quantizer training. For RVQ depth, a single-stage VQ had $$r_0 = f_{\mathrm{proj}}(z_p) \in \mathbb{R}^{N \times D},$$7 the feature recon-loss of 8-stage RVQ, and eight levels proved a practical sweet spot. On the understanding-versus-generation question, pure understanding, pure generation, and 1:1 joint training curves show that generation loss is lower when mixed with understanding, whereas understanding loss is unchanged by adding generation data. The paper treats this as corroboration for DiNA’s unification of both tasks (Team et al., 29 Mar 2026).

These training results define dNaViT not as an isolated tokenizer, but as a component coupled to a shared autoregressive backbone. Its technical role is to make vision natively compatible with next-token prediction rather than to remain an external encoder attached to a language-centric system.

6. Variants and comparative formulations

A concise comparison clarifies how the term is used across related work.

System	Core mechanism	Native-resolution handling
NaViT	ViT with sequence packing, masked self-attention, masked pooling, factorized fractional positional embeddings	Arbitrary resolutions and aspect ratios via Patch n’ Pack (Dehghani et al., 2023)
LongCat-Next dNaViT	SAE encoder, hierarchical RVQ tokenizer, ViT pixel-decoder, flow-matching refiner	Operates at encoder’s native spatial resolution; flattening into $$r_0 = f_{\mathrm{proj}}(z_p) \in \mathbb{R}^{N \times D},$$8 discrete tokens (Team et al., 29 Mar 2026)
ViQ dNaViT	SigLIP2-g ViT, text-aligned pre-training, proximal bottleneck, FSQ, 2D RoPE, head-wise quantization	No resizing or center-cropping; dynamic position handling and on-the-fly positional computation (Yu et al., 25 Jun 2026)

ViQ presents a technically distinct dNaViT. Its visual encoder is patch-based, with non-overlapping $$r_0 = f_{\mathrm{proj}}(z_p) \in \mathbb{R}^{N \times D},$$9 patches and feature dimension $L$0. Unlike a vanilla ViT with fixed-size positional embeddings, it uses a dynamically resizable embedding layer in the any-resolution ViT head and, in the quantization stage, a true 2D Rotary Position Embedding layer (Yu et al., 25 Jun 2026). Stage 1 is text-aligned pre-training with a frozen LLM, optimizing a text cross-entropy loss and a self-distillation loss against a fixed-resolution teacher. Stage 2 performs feature discretization through a proximal bottleneck and then inserts a Finite Scalar Quantization module.

The ViQ quantizer compresses to $L$1 dimensions and uses FSQ levels $L$2, giving total codebook size $L$3. To capture local detail, each patch feature is expanded through a small multi-head self-attention block that splits each patch token into $L$4 sub-tokens, producing $L$5 sub-patch tokens, and each of the $L$6 channels is quantized independently. Before quantization, a true 2D Rotary Position Embedding is added: $L$7 After quantization, a linear projection recombines the $L$8 sub-tokens back to $L$9 tokens (Yu et al., 25 Jun 2026).

The empirical profile is also different. ViQ reports an average score of 57.2 over nine benchmarks with Qwen2.5-1.5B, versus 57.0 for a continuous InternViT-6B baseline; under Qwen2.5-7B, it reports 63.9 versus 63.8. On ImageNet-1K $$\hat q_l = VQ_l(r_{l-1}), \qquad r_l = r_{l-1} - \hat q_l.$$0, it reports PSNR $$\hat q_l = VQ_l(r_{l-1}), \qquad r_l = r_{l-1} - \hat q_l.$$1, SSIM $$\hat q_l = VQ_l(r_{l-1}), \qquad r_l = r_{l-1} - \hat q_l.$$2, and rFID $$\hat q_l = VQ_l(r_{l-1}), \qquad r_l = r_{l-1} - \hat q_l.$$3. By precomputing ViQ codes offline, VLM SFT forward time is reduced by 20–70% across model sizes from 0.5B to 7B (Yu et al., 25 Jun 2026).

The two dNaViT instantiations therefore differ in quantization philosophy. LongCat-Next emphasizes hierarchical RVQ over SAE features and direct integration into a shared autoregressive DiNA backbone. ViQ emphasizes text-aligned pre-training, proximal representation learning, position-aware head-wise quantization, and FSQ-based compact codes. This suggests that the common denominator is the combination of native-resolution visual processing with discrete, semantically usable codes, while the specific discretization stack remains an active design variable.

7. Benchmarks, trade-offs, and open questions

LongCat-Next reports that on standard VQA tasks such as ChartQA and DocVQA, and on STEM reasoning with MathVista, LongCat-Next with dNaViT matches or surpasses specialist continuous models such as Qwen3-VL-A3B. On text-to-image generation benchmarks including GenEval, LongText, and TIFF, it remains competitive with dedicated T2I systems despite the unified backbone (Team et al., 29 Mar 2026). These are the principal end-task validations for the claim that dNaViT can support both understanding and generation within a single framework.

The current limitations are stated explicitly. De-tokenizer fidelity is tuned for semantic sufficiency rather than perfect pixel-level replication, and high-frequency textures can still blur. The current scheme only handles image-to-text and text-to-image as inverse sequence tasks; "truly native multimodality" would support any-to-any interleaved mappings such as image plus audio to video. Although 8-stage RVQ balances capacity versus speed, dynamic level allocation is proposed as a possible efficiency improvement. Reinforcement learning on discrete vision tokens via GRPO is described as promising for T2I fine-tuning, with sequence-level filtering likely to be crucial for longer rollouts. SAE itself is treated as fixed, and co-training SAE and quantizer in end-to-end multitask regimes is proposed as a possible route to more compact, robust discrete manifolds (Team et al., 29 Mar 2026).

NaViT’s discrete sketch contributes a different set of trade-offs. A discrete dNaViT based on fixed resolution bins would incur discretization error when $$\hat q_l = VQ_l(r_{l-1}), \qquad r_l = r_{l-1} - \hat q_l.$$4, would require large positional tables for large bins, and would lose the smooth extrapolation property of fractional positional embeddings (Dehghani et al., 2023). ViQ reports that removing the $$\hat q_l = VQ_l(r_{l-1}), \qquad r_l = r_{l-1} - \hat q_l.$$5 proximal stage, dropping 2D RoPE, or omitting the VAE latent loss each costs 2–5 points on core understanding benchmarks, indicating that its dNaViT formulation is sensitive to all three components (Yu et al., 25 Jun 2026).

A recurring misconception is that discrete visual tokenization is necessarily incompatible with high-level understanding because quantization destroys semantics. The LongCat-Next and ViQ summaries both argue against that view, but in different ways: LongCat-Next relies on a language-aligned SAE plus hierarchical RVQ, while ViQ relies on text-aligned pre-training, proximal feature shaping, and position-aware quantization. Another misconception is that any-resolution processing is purely an efficiency device. In these works it is also a representational commitment: avoiding forced resizing and preserving native spatial structure are integral to the claim that visual tokens can become first-class citizens in multimodal autoregression.

In that sense, dNaViT names a convergence point between native-resolution vision transformers and discrete multimodal modeling. In LongCat-Next, it fills the role of a "visual tokenizer + detokenizer" analogous to subword tokenizers in LLMs, combining a language-aligned encoder, hierarchical RVQ, any-resolution flattening, and a paired pixel decoder to provide a semantically complete and generatively sufficient discrete interface for vision (Team et al., 29 Mar 2026).