NaViT: Native Resolution Vision Transformers

• Native Resolution Vision Transformers (NaViT) are architectures that process images at their original scale, preserving fine spatial details and natural aspect ratios.
• They incorporate adaptive multi-scale patch embeddings, dynamic token packing, and factorized positional encodings to maintain contextual fidelity.
• NaViT models improve tasks like classification, detection, and OCR by retaining native resolution information, leading to enhanced robustness and performance.

Native Resolution Vision Transformers (NaViT) denote a class of Vision Transformer architectures and training protocols that operate on input images at their original, unresized spatial resolution and aspect ratio. Unlike traditional pipelines that enforce a fixed, square input for efficiency and architectural convenience, NaViT models accommodate real-world visual data diversity—enabling spatially faithful processing and tokenization, and supporting arbitrary image (and sometimes video) sizes. This approach addresses both the architectural and data-processing limitations of fixed-resolution models, enhancing spatial-contextual fidelity, performance on fine-grained or detail-dependent tasks, and robustness to a wide variety of natural and artificial conditions (Dehghani et al., 2023, Qiao et al., 2 Apr 2025, Niu et al., 15 Jun 2025). The following sections provide a comprehensive overview of NaViT: from core principles and algorithmic innovations, through applications and benchmarks, to unresolved challenges and future directions.

1. Motivation and Conceptual Foundations

The core motivation for Native Resolution Vision Transformers arises from the limitations observed when standard computer vision models—especially CNNs and early ViT implementations—enforce fixed, canonical input sizes. Resizing or cropping destroys fine-scale details, distorts spatial relationships, and often wastes computational resources for non-square or variable-size images (Khan et al., 2021, Qiao et al., 2 Apr 2025). Natural data—ranging from everyday photographs to scientific diagrams or documents—exhibits a broad distribution of spatial sizes and aspect ratios.

Transformers’ inherent sequence-based processing allows, in principle, for variable-length inputs, presenting an opportunity to process image content at its native resolution. By leveraging patchification and sequence-packing, the NaViT paradigm reframes vision modeling as dynamic, flexible, and contextually adaptive, enabling preservation of aspect ratio, detailed structures, and high-frequency information, all within a scalable attention-based architecture (Dehghani et al., 2023, Liu et al., 11 Dec 2024, Niu et al., 15 Jun 2025).

2. Key Architectural and Algorithmic Innovations

NaViT models implement several innovations to accommodate native resolution inputs:

• Patch n’ Pack Sequence Packing: Instead of resizing each image to a square size, images are decomposed into variable-length sequences of patches and concatenated within a batch. Attention masks ensure patch tokens do not cross image boundaries, preserving independence and spatial integrity (Dehghani et al., 2023, Liu et al., 11 Dec 2024, Niu et al., 15 Jun 2025).
• Adaptive and Multi-Scale Patch Embedding: Methods such as Multi-Scale Patch Embedding (MSPE) substitute the standard convolutional embedding layer with learnable, resolution-adaptive kernels. Kernel weights are dynamically resized to match each image’s dimensions, removing the need for global resizing while preserving local detail (Liu et al., 28 May 2024). The kernel size is computed as $$(h_k, w_k) = (\lfloor h/N \rfloor, \lfloor w/N \rfloor)$$, adapting to the input $h \times w$.
• Flexible and Factorized Positional Embeddings: To generalize to arbitrary sizes, positional encodings are computed in a factorized (x, y) format or with 2D Rotary Position Embeddings (2D RoPE), allowing the model to represent spatial position accurately for different patch layouts (Dehghani et al., 2023, Qiao et al., 2 Apr 2025, Niu et al., 15 Jun 2025).
• Resolution Curriculum Learning: Progressive training schemes begin with fixed, low-resolution pretraining for stability and resource efficiency, gradually shifting to native, variable resolutions for fine-tuning. This decouples feature learning from token length and preserves training tractability for very high resolution (Qiao et al., 2 Apr 2025).
• Dynamic Token/Sample Dropping and Merging: To control computational costs associated with longer sequences, continuous token dropping or patch merging (e.g., $4\times$ average pooling) is performed either probabilistically during training or explicitly for efficiency at inference (Dehghani et al., 2023, Niu et al., 15 Jun 2025).
• Unified Image/Video Handling: Some frameworks adopt a unified 3D convolutional patchify operation and an inter-batch image–video switching strategy, maintaining homogeneous spatial-temporal context handling in both modalities (Qiao et al., 2 Apr 2025).
• Hybrid Training with Feature Distillation and Contrastive Learning: Hybrid objectives combine sigmoid-based contrastive vision–language alignment with auxiliary feature distillation losses (e.g., from DINO teacher models). The total loss is

$$\mathcal{L}_{overall} = \mathcal{L}_{contrastive}(f_v^{\mathrm{UniViTAR}}, f_t^{\mathrm{LLaMA}}) + \lambda \cdot \mathcal{L}_{distillation}(f_v^{\mathrm{UniViTAR}}, f_v^{\mathrm{Teacher}})$$

with $\lambda$ decayed over training (Qiao et al., 2 Apr 2025).

3. Performance Characteristics and Applications

NaViT models consistently demonstrate advantages over fixed-resolution approaches across a spectrum of vision tasks:

• Image Classification and Recognition: Flexible resolution training enables models to match or exceed fixed-resolution ViTs on benchmarks such as ImageNet, with more efficient compute usage and improved generalization to atypical image geometries (Dehghani et al., 2023, Liu et al., 28 May 2024).
• Object Detection and Segmentation: Native aspect ratio and fine-scale spatial context are preserved, leading to better dense prediction and accurate boundary localization in segmentation and detection pipelines (Dehghani et al., 2023, Liu et al., 28 May 2024, Qiao et al., 2 Apr 2025).
• Robustness and Out-of-Distribution Generalization: Experiments across datasets (e.g., ImageNet-A, ImageNet-C, ObjectNet) indicate improved robustness to corruptions, rare resolutions, and OOD scenarios due to the model’s exposure to real-world spatial variation during training (Dehghani et al., 2023, Qiao et al., 2 Apr 2025).
• Vision-Language and Multimodal Models: NaViT-style encoders have been incorporated into vision-LLMs for OCR, diagram analysis, and complex scene understanding, with preserved spatial fidelity directly benefiting text recognition and geometry-intensive tasks (Liu et al., 11 Dec 2024, Niu et al., 15 Jun 2025).
• Benchmarks for Resolution Robustness: RC-Bench introduces systematic evaluation of VLMs under diverse area and aspect ratio regimes, using metrics like Exact Match (EM), Average Normalized Levenshtein Similarity (ANLS), and coefficient of variation along area and ratio axes (ACV and RCV), directly measuring the benefits of native resolution encoding (Niu et al., 15 Jun 2025).

4. Frequency Properties, Multi-Scale Context, and Inductive Bias

NaViT models can be further enhanced by integrating multi-scale processing and frequency-aware augmentation:

• High-Frequency Component Capture: Standard ViT architectures tend to attenuate high-frequency details due to their patch-based tokenization and sequential low-pass attention. High-frequency adversarial training (HAT) and architectural hybrids (e.g., convolutional token mixers or overlapping patches) restore sensitivity to fine details and improve transfer to detection/segmentation (Bai et al., 2022).
• Explicit Multi-Scale Designs: RetinaViT and related architectures concatenate patches from multiple downscaled images, allowing the attention mechanism to access both global (low-frequency) structure and high-frequency detail. Positional embeddings are scaled and averaged over each patch’s receptive field, broadening the context from 2D to a conceptual 3D space of (x, y, scale) (Shu et al., 20 Mar 2024).
• Adaptive Mixed-Resolution Tokenization: Quadformer-style tokenizers employ algorithms (e.g., Quadtree) and saliency scoring to allocate more tokens to critical or high-detail regions and fewer to backgrounds, optimizing both computational efficiency and local sensitivity while preserving global spatial fidelity (Ronen et al., 2023).

5. Multi-Modality, Unified Representation, and Real-World Integration

Native resolution modeling extends naturally to multimodal and real-world scenarios:

• Unified Foundation Models: Systems such as UniViTAR are designed with a homogeneous architectural backbone—integrating both image and video patchification, spatial-temporal 2D RoPE, and normalization strategies that scale to both modalities while harnessing native aspect ratios and spatial diversity (Qiao et al., 2 Apr 2025).
• Vision-LLM Integration: NaViT-style vision encoders serve as drop-in replacements for previous fixed-resolution CLIP-style encoders in large-scale VLMs (e.g., POINTS1.5, NativeRes-LLaVA), enabling batch processing of variable-length patch tokens and leveraging batching techniques inspired by LLMing (e.g., FlashAttention-2) (Liu et al., 11 Dec 2024, Niu et al., 15 Jun 2025).
• Applications in OCR, Document Analysis, and Scene Understanding: The ability to preserve and process high-resolution, arbitrarily sized images improves performance on OCR, diagram interpretation, mathematical problem extraction, and detailed geometric or semantic reasoning, as validated on RC-Bench, DocVQA, TextVQA, and dedicated OCRBench datasets (Niu et al., 15 Jun 2025, Liu et al., 11 Dec 2024).

6. Open Challenges, Limitations, and Future Directions

Several challenges and avenues for research are highlighted:

• Computational and Memory Efficiency: Native resolution yields more tokens, and quadratic self-attention cost remains a significant bottleneck for large images. Approaches such as windowed or sparse attention, patch/sequence merging, continuous token dropping, and efficient FlashAttention variants are under exploration (Khan et al., 2021, Dehghani et al., 2023, Niu et al., 15 Jun 2025).
• Data Diversity and Training Protocols: Effective NaViT training at scale requires datasets evenly distributed across resolutions and aspect ratios. Existing datasets tend to cluster at small or canonical sizes, necessitating new collection and curation efforts (Niu et al., 15 Jun 2025). Curriculum learning strategies and careful pretraining/fine-tuning phasing are critical for convergence and transfer.
• Positional Encoding and Generalization: As positional encodings become more complex to accommodate arbitrary spatial layouts and scales, research continues on more robust and generalizable embedding schemes (e.g., 2D RoPE, factorized embeddings, learned scale-aware position encodings) (Dehghani et al., 2023, Qiao et al., 2 Apr 2025).
• Unified Multimodal Token Processing: Extending native processing from static images to video (temporal sequences) and non-image modalities (documents with text and visuals) is facilitated by unified patchification and sequential batching, but computational and representation trade-offs remain open for investigation (Qiao et al., 2 Apr 2025).
• Benchmarks and Evaluation Metrics: The systematic design of benchmarks like RC-Bench is essential for rigorous evaluation under varied native conditions, as traditional leaderboards and datasets do not sufficiently probe area/aspect-ratio robustness (Niu et al., 15 Jun 2025).
• Implementation Practicalities: Cross-image interference in packed sequences, batch management, and hardware utilization strategies (e.g., overlapping tokens, boundary anchoring) remain active areas for engineering optimization and hardware-aware design (Dehghani et al., 2023, Qiao et al., 2 Apr 2025).

7. Summary Table: NaViT Model Characteristics and Innovations

Model/Method	Native Resolution Input	Key Mechanism(s)	Reference
NaViT (Patch n’ Pack)	Yes	Sequence packing, factorized position, variable-length batch	(Dehghani et al., 2023)
MSPE	Yes	Multi-scale patch embedding, kernel adaptation	(Liu et al., 28 May 2024)
NativeRes-LLaVA	Yes	2D RoPE, flexible patch packing, RC-Bench eval	(Niu et al., 15 Jun 2025)
UniViTAR	Yes (image/video)	Unified patchify, 2D RoPE, curriculum learning	(Qiao et al., 2 Apr 2025)
POINTS1.5	Yes	NaViT-style encoder, packed attention for VLM	(Liu et al., 11 Dec 2024)
RetinaViT	Yes (multi-scale)	Multi-scale input pyramid, scaled embedding	(Shu et al., 20 Mar 2024)
Quadformer	Mixed-resolution	Saliency-driven adaptive tokenization	(Ronen et al., 2023)

• "Transformers in Vision: A Survey" (Khan et al., 2021)
• "Improving Vision Transformers by Revisiting High-frequency Components" (Bai et al., 2022)
• "Vision Transformer: Vit and its Derivatives" (Fu, 2022)
• "Vision Transformers with Mixed-Resolution Tokenization" (Ronen et al., 2023)
• "Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution" (Dehghani et al., 2023)
• "Retina Vision Transformer (RetinaViT): Introducing Scaled Patches into Vision Transformers" (Shu et al., 20 Mar 2024)
• "MSPE: Multi-Scale Patch Embedding Prompts Vision Transformers to Any Resolution" (Liu et al., 28 May 2024)
• "POINTS1.5: Building a Vision-LLM towards Real World Applications" (Liu et al., 11 Dec 2024)
• "UniViTAR: Unified Vision Transformer with Native Resolution" (Qiao et al., 2 Apr 2025)
• "Native Visual Understanding: Resolving Resolution Dilemmas in Vision-LLMs" (Niu et al., 15 Jun 2025)