---
title: 'ConvLLaVA: Efficient High-Res LMM Architecture'
url: https://www.emergentmind.com/topics/convllava
type: topic
---

# ConvLLaVA: Efficient High-Res LMM Architecture

ConvLLaVA is a large multimodal model (LMM) architecture that leverages a hierarchical convolutional backbone to efficiently encode high-resolution visual inputs for downstream language models. By integrating a five-stage ConvNeXt as its visual encoder and introducing optimization strategies tailored for high-resolution processing, ConvLLaVA addresses both token redundancy and quadratic compute scaling that limit Vision Transformer (ViT)-based LMMs. It achieves highly competitive results on general and fine-grained vision-language benchmarks while maintaining flexibility for arbitrary image resolutions and aspect ratios [2405.15738].

## 1. Motivation and Background

High-resolution LMMs must balance computational efficiency with information retention. ViT-based encoders split images of size $H \times W$ into $P \times P$ patches, yielding
$$N_{\rm ViT} = \frac{H\,W}{P^2}$$
visual tokens and introducing self-attention complexity of
$$\mathcal{O}(N_{\rm ViT}^2) = \mathcal{O}\left(\frac{H^2W^2}{P^4}\right).$$
At $1536 \times 1536$ resolution with $P=16$, this results in $9216$ tokens, creating significant overhead in memory and FLOPs due to quadratic scaling.

Hierarchical backbones such as ConvNeXt sequentially downsample feature maps, resulting in a final spatial shape of $H/32 \times W/32$ after four stages. The token count for a four-stage ConvNeXt is
$$N_{\rm 4-stage} = \frac{H\,W}{32^2},$$
approximately one-ninth of ViT at equivalent resolution, while the depth-wise convolutions scale linearly as $\mathcal{O}(k^2 C N)$. Figure 2 in the source demonstrates an approximate $8\times$ reduction in compute for ConvNeXt versus ViT-L when processing high-resolution images [2405.15738].

## 2. Model Architecture and Visual Token Compression

ConvLLaVA replaces the ViT visual encoder with a ConvNeXt-L backbone, originally comprising four stages and a $4 \times 4$ convolutional patch embed for up to $32\times$ downsampling. To address persistent redundancy at higher resolutions, ConvLLaVA extends the backbone with a fifth ConvNeXt stage, reaching a $64\times$ total downsampling:

$$N_{\rm 5-stage} = \frac{H\,W}{64^2}.$$

For $1536 \times 1536$ images, this amounts to just $576$ visual tokens, a drastic reduction compared to conventional approaches.

Two primary optimizations enable this architecture to perform on par or better than ViT-based models at high resolutions:

1. **Visual Encoder Re-Tuning for High Resolution:** The pretrained CLIP-ConvNeXt-L, optimized for images up to $320$ px, exhibits degraded performance when evaluated on larger inputs. ConvLLaVA addresses this through a three-stage training pipeline:
   - Projector initialization using $558$ K caption pairs.
   - Vision–Language pretraining with $\sim2$ M ShareGPT4V-PT caption pairs.
   - Instruction tuning with $665$ K LLaVA examples.
   During the latter two stages, the last $18$ ConvNeXt blocks, the projector, and the LLM adapter are unfrozen and trained to bridge the gap between low- and high-resolution regimes.

2. **Successive Downsampling (Fifth Stage):** An additional ConvNeXt stage (six blocks) further downscales feature maps, maintaining the token count at $576$ for $1536$ px images without loss of fine-grained detail. Table 4 in the source shows that this enables improvements on tasks such as TextVQA and DocVQA as resolution increases [2405.15738].

## 3. Flexible Handling of Resolutions and Aspect Ratios

ConvLLaVA’s convolutional visual backbone is translation-equivariant and agnostic to image aspect ratio. After training at a canonical “square” resolution (e.g., $1536 \times 1536$), the system can be deployed on images of arbitrary aspect ratio: only the short side must be resized to $1536$ px, while the long side is preserved. Empirical results (Table 7) indicate minimal performance degradation, and sometimes performance gains on OCR-oriented tasks like DocVQA. Inference at even higher resolutions (e.g., short side $1664$ px) offers further improvements for select benchmarks [2405.15738].

## 4. Experimental Evaluation and Quantitative Comparison

The ConvLLaVA model suite is evaluated on a spectrum of mainstream benchmarks, including:

- **General capability:** MME, MMBench, SEEDBench, RealWorldQA, MMMU, MMVet, POPE
- **Fine-grained OCR:** TextVQA, DocVQA
- **Referring comprehension:** RefCOCO, RefCOCO+, RefCOCOg

All closed-form vision–language responses are scored using VLMEVALKIT (exact match/accuracy), and grounding is measured via standard IOU $\geq0.5$.

### Compute and Token Count Efficiency

A major contribution is reduction in both token count and computation:
- ViT at $336$ px: $576$ tokens (quadratic compute)
- MiniGemini-HD at $1536$ px: $2880$ tokens (cropping+Enum-aspect)
- ConvLLaVA-1536: $576$ tokens (linear compute)

At $768$ px, ConvNeXt offers an $8\times$ compute saving over ViT; with the fifth stage, an additional $6\times$ reduction at $1536$ px is observed (see Figure 2).

### Comparative Performance

A summary from Table 5 is provided below for representative 7B models:

| Method           | Res   | #Tokens | MMBench | SEEDBench | TextVQA | DocVQA | POPE  |
|------------------|-------|---------|---------|-----------|---------|--------|-------|
| LLaVA-1.5 (7B)   | 336   | 576     | 64.3    | 66.2      | 45.5    | 21.6   | 85.9  |
| ShareGPT4V (7B)  | 336   | 576     | 68.8    | 69.7      | 51.1    | 26.6   | 86.0  |
| MiniGemini-HD    | 1536  | 2880    | 65.8    | —         | —       | —      | —     |
| ConvLLaVA (7B)   | 1536  | 576     | 68.7    | 70.2      | 65.8    | 59.0   | 87.3  |

ConvLLaVA at $1536$ px matches or exceeds models of similar or greater scale on MMBench and SEEDBench with substantially fewer tokens and demonstrates dramatic improvements on OCR tasks (e.g., DocVQA: $59.0$ vs $26.6$ for $336$ px models). On referring comprehension tasks, ConvLLaVA outperforms LLaVA-1.5 (Table 6), achieving an average $82.3\%$ versus $79.3\%$ [2405.15738].

## 5. Implications and Technical Significance

ConvLLaVA demonstrates that hierarchical, convolutional visual encoders—when appropriately extended and retrained—can obviate the quadratic bottlenecks of ViT at high resolution. By maintaining only a few hundred tokens even for megapixel-scale images, ConvLLaVA achieves $\sim6$–$8\times$ compute savings compared to leading high-resolution LMMs, while retaining or improving performance on both general and fine-grained tasks.

Key architectural properties include linear spatial complexity ($\mathcal{O}(N)$), token compression via staged downsampling, and equivariance permitting arbitrary aspect ratio inference without retraining.

## 6. Limitations and Prospects for Advancement

The ConvNeXt backbone’s configuration—kernel sizes, stage depths, and pretraining regimen—remains optimized for low-resolution settings. Adapting backbones specifically for high-resolution input, for example by enlarging kernel sizes or rebalancing stage depths, potentially yields better representations. The trade-off between compressing information and preserving small-object details persists: excessive downsampling risks information loss; retaining excessive tokens burdens subsequent language model processing. Adaptive or content-aware compression schemes represent a promising direction.

The architecture’s principles are broadly applicable to multi-image, video, or interleaved vision/text settings, although explosive token growth in these domains will require further innovations in visual token adaptation and complexity control [2405.15738].

Source: https://www.emergentmind.com/topics/convllava