ViT-UHD Encoder: Efficient High-Res Encoding

• ViT-UHD encoder is an advanced visual module that efficiently compresses high-resolution images for MLLMs using refined patch embedding and windowed token compression.
• It introduces a progressive visual compression paradigm and a hierarchical semantic pyramid with window-based attention to reduce token count and preserve fine details.
• ViT-UHD achieves notable reductions in processing time and computational cost while maintaining or improving accuracy on various vision-language benchmarks.

The ViT-UHD encoder is an advanced visual encoding module designed to enable efficient and high-fidelity native-resolution visual representation for multi-modal LLMs (MLLMs). It builds upon the standard Vision Transformer (ViT) architecture but introduces targeted modifications that allow for dramatic reduction in token count and computational complexity, while further preserving or enhancing fine-grained visual detail necessary for vision-language reasoning. Two principal forms of ViT-UHD encoders have been developed: (1) a progressive visual compression (PVC) paradigm, and (2) a hierarchical semantic pyramid with window-based attention. These approaches have notably advanced the state of the art in efficient visual encoding for MLLMs, as substantiated in "LLaVA-UHD v3: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs" (Sun et al., 26 Nov 2025) and "LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformer" (Zhang et al., 2024).

1. Architectural Principles and Encoding Workflow

ViT-UHD encoders reconfigure pretrained plain ViT models to mitigate the quadratic complexity in token count associated with high-resolution inputs. This is achieved by incorporating modules that condense and compress dense token sequences created from native-resolution images, aligning the encoding process with downstream LLMs without sacrificing visual granularity.

Progressive Visual Compression (PVC) (Sun et al., 26 Nov 2025)

• Refined Patch Embedding (RPE): Replaces the default fixed-size patch tokenizer in ViT with a flexible, often smaller patch size $\hat P < P$, by analytically converting pretrained embedding weights $W \in \mathbb{R}^{D \times (C P^2)}$ to $\hat W \in \mathbb{R}^{D \times (C \hat P^2)}$ via the Moore–Penrose pseudoinverse. This enables tokenization at finer resolution while maintaining pretrained weight initialization.
• Windowed Token Compression (WTC): Hierarchically inserted lightweight modules that aggregate local $2\times2$ token windows at selected transformer block depths. Compression is realized through average pooling or adaptive pooling via a learnable MLP, yielding a progressive reduction in total token count.

Hierarchical Semantic Pyramid with Window Attention (Zhang et al., 2024)

• Backbone: Utilizes a frozen CLIP-ViT to extract an initial single-scale feature map.
• Inverse Semantic Pyramid (ISP): Progressive upsampling constructs feature maps at increasing spatial resolutions ($1\times$, $2\times$, $4\times$), using joint bilateral upsampling (JBU) conditioned on the image. Each upsampling stage injects low-level detail to higher-level semantic features.
• Hierarchical Window Attention (Hiwin): Applies cross-scale window-based attention using learnable queries mapped to spatial anchors across pyramid levels. Regions are RoI-aligned and their features aggregated via learned attention to produce a compressed set of visual tokens that retain high-resolution detail.

2. Technical Mechanisms: RPE and WTC

Refined Patch Embedding (RPE) (Sun et al., 26 Nov 2025)

RPE modifies the initial patch extraction from the image such that the original large patches ($P \times P$) are decomposed into smaller ones ($\hat P \times \hat P$). Given a fine-grained patch vector $\hat t \in \mathbb{R}^{C \hat P^2}$, the coarse patch $t$ can be written as $t = B \hat t$ for a fixed matrix $B$. Embedding weights for the finer grid are computed by the pseudoinverse relation:

$\hat W = (B^T)^+ W$

This design preserves compatibility with pretrained ViT weights, allowing fine-resolution token creation without retraining the encoder from scratch.

Windowed Token Compression (WTC) (Sun et al., 26 Nov 2025)

WTC modules, strategically interleaved after designated transformer blocks, aggregate information from local spatial neighborhoods. For each non-overlapping $2 \times 2$ window of token vectors $\{x_i\}$:

• Average Pooling: $x_{\mathrm{avg}} = \frac{1}{4}\sum_{i=1}^4 x_i$
• Content-Adaptive Pooling: Each $x_i$ is concatenated with $x_{\mathrm{avg}}$, passed through a 2-layer MLP $f_\theta$, and softmax-normalized to obtain content-adaptive weights $w_i$. The compressed token is $x_{\mathrm{out}} = \sum_{i=1}^4 w_i x_i$. These operations reduce the spatial token grid by a factor of $4$ per WTC stage, allowing for deep compression hierarchies.

3. Hierarchical Semantic Pyramid and Window Attention

The ViT-UHD encoder in (Zhang et al., 2024) introduces an alternative approach with explicit multi-scale processing and attention over spatially organized regions.

• Pyramid Construction: Starting from $F^0 \in \mathbb{R}^{H/p \times W/p \times C}$, two upsamplings are performed by the JBU module, producing $F^1$, $F^2$ that double and quadruple the original patch resolution respectively. The upsampling operation:

$$F^{l+1}[x, y] = \frac{1}{|U|} \sum_{(x', y') \in U} \text{Up}(F^l)[x', y'] \cdot D_{\text{dist}} \cdot D_{\text{sim}}$$

with spatial and content similarity kernels $D_{\text{dist}}$, $D_{\text{sim}}$.

• Hierarchical Window Attention: Defines $N^2$ learnable queries per image slice, each corresponding to an anchor region. Feature vectors from each pyramid level are RoI-aligned and concatenated as keys and values for attention with the queries. The attended outputs are spatially reassembled and, together with tokens marking slice and spatial boundaries, are supplied to the LLM.

4. Training Procedures and Optimization

Training ViT-UHD models is performed in two stages (Sun et al., 26 Nov 2025):

1. Pre-alignment: With the LLM frozen, only RPE, WTC modules, and the output projector are trained on image-text and OCR pairs to align the new compression stages with the original feature space.
2. Full Fine-Tuning: All parameters (ViT-UHD, projector, and LLM) are jointly trained on a mixture of contrastive, prefix-LM, and instruction-tuning objectives, using standard AdamW and cosine learning rate scheduling. Zero-initialization of adaptive poolers stabilizes early-stage optimization.

For Hiwin-augmented encoders, auxiliary losses supervise each pyramid level's reconstructed features to preserve semantic alignment (Zhang et al., 2024).

5. Computational Complexity and Empirical Evaluation

ViT-UHD achieves substantial reductions in computational cost while maintaining or slightly improving accuracy on vision-language benchmarks.

Model Variant	Patch Size / Stages	Tokens (Input/Final)	TTFT (ms)	Accuracy (MMBench, AI2D, etc.)
Baseline ViT	16 / none	4096	233	62.1%
ViT + 2x WTC (avg pool)	16 / 2 (layers 4,18)	4096 / 256	82	↓ (drops on fine-grained)
ViT + 2x WTC (adaptive)	16 / 2	4096 / 256	83	60.7%
ViT-UHD (PVC full)	8 / 3 (4,18,27)	16384 / 256	160	63.0%

ViT-UHD with progressive visual compression enables:

• 2.4× reduction in time-to-first-token (TTFT) compared to MoonViT (121 ms vs 296 ms at $1024 \times 1024$).
• 1.9× lower TTFT versus slice-based Qwen2-VL (153.8 ms vs ~290 ms at $1344^2$).
• Near parity or better performance compared to state-of-the-art models on 15+ benchmarks (Sun et al., 26 Nov 2025).

Hiwin-based ViT-UHD provides:

• +3.7% mean improvement over baseline on 14 MLLM benchmarks, +9.3% on DocVQA, at substantially reduced computational cost (17.5T vs 44.4T FLOPs for LLaVA-Next at $1008 \times 672$ vs $672 \times 672$).
• Effective token-budgeted compression that avoids quadratic scaling with input resolution, crucial for high-res documents and dense image tasks (Zhang et al., 2024).

6. Implementation Specifications

• Patch sizes: Pretrained ViT models typically use $P=14$ or $16$; ViT-UHD applies $\hat P=8$ as the default.
• Compression stages: Three WTC layers are commonly used, e.g., after transformer blocks 4, 18, and 27 for a 36-layer backbone.
• Window sizes: $2\times2$ for PVC/WTC; window grid parameters for Hiwin are set via learned anchors.
• Optimization: AdamW with $\beta=(0.9,0.95)$, cosine LR schedule, 3% warm-up. Multi-stage data curriculum, ranging from pre-alignment with 4.3M pairs, to joint pre-training (5M pairs), to fine-tuning with 13.3M examples.
• Scaling: Training of the full pipeline (ViT-UHD + LLM + projector) is typically conducted on 32 $\times$ 80GB A100 GPUs, requiring approximately 300 hours in (Sun et al., 26 Nov 2025).

7. Relationship to Prior Approaches and Significance

ViT-UHD encoders depart from global pooling or single-scale token emission characteristic of standard ViT architectures. By exploiting patch-wise replacement via RPE, multistage spatial compression via WTC or Hiwin modules, and explicit multi-scale information injection, ViT-UHD models consistently address the token explosion and attention inefficiency issues that plague high-resolution visual inputs in MLLMs. Empirical evidence from recent benchmarks supports the efficacy of these schemes in balancing memory efficiency, computational speed, and fine-grained vision-language task performance. The results confirm that ViT-UHD methodologies achieve efficient native-resolution encoding, outperforming or matching previous models on both general and fine-grained benchmarks while substantially reducing inference and training costs (Sun et al., 26 Nov 2025, Zhang et al., 2024).