InternLM-XComposer2-4KHD: High-Res LVLM Innovation

• InternLM-XComposer2-4KHD is a high-resolution LVLM that extends input support from 336×336 pixels to 4K HD using dynamic image partitioning.
• The model leverages a frozen ViT-L/14 backbone with dual global and local views, incorporating newline tokens to preserve aspect ratios and spatial structures.
• It achieves competitive benchmark results in OCR, chart analysis, and visual reasoning while addressing computational scalability challenges.

InternLM-XComposer2-4KHD is a state-of-the-art Large Vision-LLM (LVLM) that addresses high-resolution visual understanding by extending input support from the standard 336×336 pixels up to 4K HD (3840×1600) and beyond, operating across a continuous space of resolutions. Designed to overcome prior limitations in fine-grained content comprehension, InternLM-XComposer2-4KHD employs a dynamic image partitioning algorithm, maintains aspect ratios, and efficiently scales patch layouts based on a frozen pre-trained Vision Transformer (ViT) backbone. The approach achieves and often surpasses the performance of proprietary models such as GPT-4V and Gemini Pro on multiple high-resolution benchmarks (Dong et al., 9 Apr 2024).

1. Model Architecture and Core Innovations

At its foundation, InternLM-XComposer2-4KHD incorporates the OpenAI CLIP ViT-L/14 as its vision encoder, with all patch embedding and self-attention layers held frozen from the pre-trained 336×336 checkpoint. Rather than retrain ViTs capable of handling extreme resolutions, the model dynamically partitions any input image into multiple 336×336 local patches, each processed by the frozen backbone. For each input image $x$ of size $(h, w)$, two streams are constructed:

• Global View: $x_g \in \mathbb{R}^{336 \times 336}$, representing a coarse summary for global context.
• Local View: $x_\ell$, generated by dividing the image into $p_h \times p_w$ patches according to a formal patch-division paradigm. After zero-padding $x$ to $$[\hat{H}, \hat{W}] = [p_h \cdot 336, p_w \cdot 336]$$, the partition ensures $\hat{H} / \hat{W} \approx h/w$, preserving the native aspect ratio.

To enhance feature fusion and maintain structural integrity:

• A learnable newline indicator token (“\n”) is inserted after each patch row. This structural token is empirically shown to improve layout-aware reasoning, especially for large, irregular grids.
• The global and local views’ tokens are separated by a designated <sep> token.

The formal function $P(H, W; \mathcal{H})$ specifies permissible grid dimensions $(p_h, p_w)$ for any input dimension $(H, W)$ up to a maximum patch quota $\mathcal{H}$:

$$p_w = \operatorname{argmin}_{1 \leq n \leq \mathcal{H}} \{ n : n \cdot 336 \geq W \wedge n \cdot \lceil n H/W \rceil \leq \mathcal{H}\}, \quad p_h = \lceil p_w H/W \rceil$$

This generalizes patchification for arbitrary resolutions up to and beyond 4K, while maintaining computational tractability (Dong et al., 9 Apr 2024).

2. Dynamic Resolution Training and Patch Management

InternLM-XComposer2-4KHD employs a dynamic training regime designed to maximize both data variety and model generalizability:

• Automatic Patch Configuration: For each image, a maximum patch count $\mathcal{H}$ is sampled from a predefined set (e.g., $\mathcal{H} \in \{9, 16, 25, 55\}$). The image is dynamically partitioned, and both global and local features are extracted at every training iteration.
• Hierarchical Training Phases:
  • Pre-training (HD-25): Operates with $\mathcal{H}=25$; images sourced from web-scale and OCR-centric datasets. Large batch size (4096) and vision encoder layer-wise decay ($=0.9$) are used.
  • Supervised Fine-tuning (HD-55 and Mixed): For high-resolution OCR tasks (DocVQA, ChartQA, InfoVQA, TextVQA, OCRBench), $\mathcal{H}=55$ is used to fully cover 4K inputs. Other tasks sample $\mathcal{H}$ uniformly from $[9, 25]$.

The dynamic curriculum results in a patch budget scaling protocol that maintains both computational feasibility and label fidelity across arbitrarily high resolutions.

3. Benchmark Performance Across Resolution Regimes

Evaluation of InternLM-XComposer2-4KHD was conducted on 16 public benchmarks, including a diverse spectrum covering OCR, chart and table QA, general vision-language reasoning, and more. Table 3 from (Dong et al., 9 Apr 2024) summarizes leading results:

Benchmark	GPT-4V	Gemini Pro	IXC2-4KHD
DocVQA	88.4	88.1	90.0
ChartQA	78.5	74.1	81.0
TextVQA	78.0	74.6	77.2
MathVista	47.8	45.8	57.8
OCRBench	51.6	68.0	67.5
MMStar	57.1	42.6	54.1

Performance scales monotonically with patch budget and resolution on high-resolution OCR tasks (e.g., InfoVQA: 50.5→58.6→63.6→69.3 from HD-9 to 4KHD). No saturation is observed at 4K, indicating continued improvement potential with larger patch quotas or higher native input resolution.

Ablation studies highlight:

• Removing the global view causes up to a –4.4% drop in layout-aware benchmarks.
• Excluding the newline token reduces accuracy by up to 1.9% on 4KHD settings.
• Simplified concatenation for token merging matches the performance of advanced C-Abstractor schemes at 4× compression.

4. Analysis of Scaling, Limitations, and Efficiency Trade-offs

The gains afforded by ultra-high-resolution input are most pronounced for tasks involving fine-grained spatial structures (OCR, charts, tables). At resolutions beyond prior LVLMs ($\sim$1500×1500), previously unreadable text, icons, or fine contours become accessible to the encoder, enhancing output fidelity and answer accuracy.

However, computational and memory costs scale linearly with the patch count $p_h \cdot p_w$. For 4KHD inputs, token count ($\sim$8737) nearly doubles that of HD-25 ($\sim$4057), increasing both training and inference resource demands. Inference latency rises due to the need for patch extraction, feature aggregation, and token merging, making efficient parallelization and hardware acceleration increasingly relevant for production deployment.

A persistent open challenge is the limited availability of high-res instruction-tuning data, with dynamic resizing partially addressing—but not fully solving—this data scarcity.

5. Practical Applications and Broader Implications

InternLM-XComposer2-4KHD extends the application scope of LVLMs to:

• Document understanding: Rapid and robust processing of full-page scans for OCR and form filling, obviating the need for crop-based preprocessing.
• Chart and infographic analysis: Interpretation of web-native diagrams, blueprints, and structured graphics at their original resolution.
• Flexible aspect ratio chatbots: Supports UIs, signage, and screenshots of arbitrary size and orientation, facilitating universal vision-language interfaces.

The integration of global and local context, structural tokens, and dynamic partitioning generalizes the LVLM paradigm beyond fixed-resolution limits, indicating a broader trend toward resolution-adaptive architectures.

6. Future Research Directions

Key avenues for further investigation include:

• Efficient encoding mechanisms: Sparse attention and windowed transformer variants may alleviate token overhead inherent to ultra-high resolution partitioning.
• Adaptive patch sizing: Dynamically shrinking patch size in high-density regions (e.g., dense text or symbols) could further enhance content sensitivity without excessive token inflation.
• Overcoming 4K limitations: Strategies involving patch quotas $\mathcal{H}>55$ and distributed inference on accelerator hardware (e.g., tensor-slicing, model parallelism) remain to be explored.
• Resolution curricula: Progressive training from low to high patch counts may improve convergence and robustness.

A plausible implication is that, given the absence of observable performance saturation at 4K, further scaling of both hardware and patch partitioning schemes will continue to yield improved results on challenging high-resolution visual reasoning tasks (Dong et al., 9 Apr 2024).

7. Comparative Perspective and Position in the LVLM Landscape

InternLM-XComposer2-4KHD distinguishes itself by its flexible, resolution-agnostic processing pipeline and empirically validated improvements across a suite of high-resolution, structure-dependent tasks. Its approach of reusing a frozen ViT-L/14 backbone via dynamic partitioning incurs minimal redevelopment cost, fostering practical scalability. Evaluation against both open and closed-source models demonstrates competitive or superior results in 10 out of 16 benchmarks, consolidating its role as a pioneering model in the evolution of high-resolution LVLMs (Dong et al., 9 Apr 2024).