Task-driven Multi-granularity Resolution Adjustment

• TMRA is a framework that dynamically adjusts visual resolution based on task demands, balancing computational efficiency with the need for fine detail.
• It optimizes processing by selecting the minimal sufficient resolution for various tasks, from global scene understanding to detailed OCR and pixel-level parsing.
• Empirical results show significant reductions in token usage and faster inference speeds, demonstrating its practical value in multimodal reasoning and remote sensing.

Task-driven Multi-granularity Resolution Adjustment (TMRA) denotes a class of methods that match resolution or granularity to task demand instead of imposing a single fixed scale on all inputs. In visual LLMs and related multimodal systems, the central premise is that coarse global understanding, region-centric reasoning, OCR-like perception, pixel-level parsing, navigation, and manipulation do not require the same perceptual granularity. TMRA therefore treats resolution as a task-sensitive modeling choice: it selects, mixes, or constructs the coarsest representation that is sufficient for the current image, instruction, prompt, or query, while preserving finer detail where downstream utility requires it (Lan et al., 2024, Luo et al., 10 Oct 2025, Ma et al., 8 Jul 2025).

1. Problem setting and conceptual scope

TMRA arises from a recurring mismatch between downstream task requirements and fixed-resolution model design. In high-resolution LMMs such as LLaVA-NeXT, one common strategy is to divide an image into multiple local images and one global image; this preserves detail but produces a large number of visual tokens and high quadratic attention cost. The underlying inefficiency is that many tasks do not need the finest detail: color questions or scene-level questions can often be answered from coarse evidence, whereas OCR, chart reading, or tiny text recognition may require finer granularity (Lan et al., 2024).

A closely related formulation appears in task-aware resolution optimization for VLLMs. There, the core argument is that real-world vision-language tasks do not all need the same image resolution, yet most VLLMs are built and used with a single fixed input resolution. The paper explicitly states that very low resolution can remove the detail needed for hard visual reasoning or OCR-like tasks, while very high resolution does not automatically help because it increases the number of image tokens, can misalign with the model’s learned visual representation, and may introduce irrelevant or redundant visual tokens that hurt performance unless the model is adapted. Across eight tasks, optimal resolutions are scattered across $336^2$, $448^2$, and $560^2$, rather than collapsing to a single best setting (Luo et al., 10 Oct 2025).

The same logic is sharpened in remote sensing. GeoMag frames TMRA as a response to a mismatch between task granularity and input resolution: image-level tasks mostly need global semantics, region-level tasks need fine local detail, and pixel-level tasks require precise spatial structure where the target lies. High-resolution remote sensing imagery also contains large amounts of task-irrelevant background, so full-resolution processing wastes tokens, memory, and compute (Ma et al., 8 Jul 2025).

A plausible synthesis is that TMRA is best understood not as a single architecture, but as a principle of conditioned granularity control. The relevant conditioning signal may be the image-instruction pair, a benchmark-level task descriptor, a prompt, a query, or a query token itself.

2. Sample-level TMRA via adaptive granularity routing

AVG-LLaVA is a concrete implementation of TMRA for multimodal reasoning. Its guiding principle is explicit: choose the minimal sufficient resolution for the task, preserving or improving performance while reducing computation. The model adds two modules on top of LLaVA-NeXT: a parameter-free visual granularity scaler and a visual granularity router. With CLIP-ViT-L/336, each image initially yields a $24 \times 24$ token grid; the scaler sequentially applies $1\times 2$ and $2 \times 1$ average pooling, preserving spatial layout while progressively reducing token count and producing levels such as $24 \times 12$, $12 \times 12$, $12 \times 6$, and $6 \times 6$. In the broader experimental setting, the router considers a granularity set like $448^2$0 tokens (Lan et al., 2024).

The router then selects which resolution to use for the current input. It concatenates tokens from all granularity levels,

$448^2$1

filters instruction tokens by cosine similarity with the original-granularity visual tokens, retains the top-$448^2$2 instruction tokens as $448^2$3, and processes the combined sequence with three learned components: a single Transformer layer, an MLP layer, and a voter layer. The MLP predicts token-wise granularity logits

$448^2$4

and a learnable voter matrix

$448^2$5

aggregates these into final granularity logits

$448^2$6

A softmax over $448^2$7 yields a probability distribution over granularities, and the granularity with the highest probability is selected (Lan et al., 2024).

A central technical point is that the router is not trained by ordinary visual instruction tuning. AVG-LLaVA introduces RGLF, “Ranking Granularity to Align LMM Feedback,” which uses the LMM itself as a preference signal. For a given image and instruction, the system evaluates the answer log-likelihood under each granularity, sorts granularities in descending order of LMM answer probability, and trains the router with a pairwise ranking loss,

$448^2$8

plus a cross-entropy term toward the single best granularity,

$448^2$9

This supervision requires no manually annotated granularity labels (Lan et al., 2024).

Empirically, this routing policy yields both efficiency and accuracy gains. The reported results include up to an 85.3% reduction in visual tokens and up to a 2.53$560^2$0 inference speedup on AI2D. Representative benchmark values include 67.1 on TextVQA, 74.6 on DocVQA, 67.3 on AI2D, 1557.4 on MME, 69.9 on MMB, and 37.4 on MMMU. The reported visualizations also show the expected TMRA behavior: text-centric benchmarks tend to select fine tokens, while object-level reasoning tasks often prefer coarser grids (Lan et al., 2024).

3. Task-level TMRA through task-aware resolution optimization

A different TMRA formulation appears in task-aware resolution optimization for VLLMs. Instead of choosing a granularity separately for each sample, this method estimates a task-optimal discrete resolution for an entire benchmark or downstream task. The study covers ScienceQA-IMG, VizWiz, VQAv2, GQA, TextVQA, OKVQA, MMBench, and MMBench-CN, and reports that ScienceQA-IMG and VizWiz prefer $560^2$1, VQAv2/TextVQA/GQA prefer $560^2$2, and OKVQA/MMBench/MMBench-CN prefer $560^2$3 (Luo et al., 10 Oct 2025).

The selection rule is built from two task-level heuristics. The first is image complexity $560^2$4, defined as a task-level average complexity score based on a minimum-description-length clustering method for perceptual image complexity. The second is uncertainty variance $560^2$5, defined from the relative change in model uncertainty when a pretrained VLLM is run at two resolutions, where uncertainty is measured from token-level entropy under random augmentations: $560^2$6 The authors report that the product $560^2$7 is more consistent than either factor individually (Luo et al., 10 Oct 2025).

The resulting empirical formula is

$560^2$8

where $560^2$9 is the baseline resolution, $24 \times 24$0 is average normalized image complexity, $24 \times 24$1 is average uncertainty variance, and $24 \times 24$2 is a user-specified non-negative hyperparameter. In the reported implementation, $24 \times 24$3, the search space is $24 \times 24$4, $24 \times 24$5, and $24 \times 24$6, and $24 \times 24$7 is selected using SciQA-IMG, VQAv2, and OKVQA as reference tasks. Intermediate outputs are mapped to the largest supported resolution not exceeding the predicted value (Luo et al., 10 Oct 2025).

TMRA in this setting does not stop at selecting a resolution. The method adapts LLaVA-1.5-7B to the selected higher resolution with a parameter-efficient fine-tuning strategy. The procedure first interpolates the visual positional embeddings to the new patch grid, then fine-tunes only three components: position embeddings in the visual encoder, projector parameters, and LoRA adapter parameters in the LLM backbone. All other parameters are frozen. The training setup follows LLaVA stage-2 fine-tuning with image-text pairs drawn from COCO, GQA, OCR-VQA, TextVQA, and Visual Genome (Luo et al., 10 Oct 2025).

The reported task-wise adaptive results include VQAv2: 80.19, GQA: 63.87, TextVQA: 60.25, OKVQA: 50.86, MMBench: 67.70, and MMBench-CN: 61.08. The paper also reports that naive training-free interpolation is worse than the adapted model, and that tuning position embeddings + projector + LoRA adapters is the best ablation setting (Luo et al., 10 Oct 2025).

4. Prompt- and query-conditioned TMRA

GeoMag implements TMRA as a prompt- and task-aware preprocessing policy for remote sensing image parsing across image-level, region-level, and pixel-level tasks. The system first classifies the query into one of three task types using BERT-base to produce a 768-dimensional representation, followed by FC layers with 256 neurons, 128 neurons, and an output layer with 3 neurons. The resulting task granularity $24 \times 24$8 determines the image transformation: $24 \times 24$9 with

$1\times 2$0

Image-level queries therefore use a globally downsampled image; region-level queries keep one prompt-relevant region intact and compress the rest; pixel-level queries recursively crop more precisely and compress broader context more aggressively (Ma et al., 8 Jul 2025).

The task-relevant region is identified by Prompt-guided Semantic-aware Cropping (PSC), which constructs a gradient-weighted attention heatmap from last-layer attention: $1\times 2$1 GeoMag then divides the image into grid cells, proposes candidate boxes, compares total attention inside each candidate with neighboring regions, and selects the box with the largest attention contrast. In the reported ablation, removing MagCrop leads to large performance drops: on RRSIS-D referring segmentation, [email protected] 65.45, OIoU 74.20, MIoU 57.47 becomes 81.30, 82.67, 65.71 with the full model; on RefSegRS, 44.93, 62.74, 48.51 becomes 77.50, 80.27, 61.39; and on referring object classification, SS 75.04, SIoU 64.29 becomes 97.84, 96.47 (Ma et al., 8 Jul 2025).

FOUND-IT transfers the same principle to hierarchical 3D scene graphs. Its defining move is to make granularity a query-time decision rather than a mapping-time commitment. The system stores a lightweight geometric backbone continuously, then resolves objects, places, and regions at the right level of detail on demand. For objects, it uses a two-stage memory consisting of visual memory and cached memory: keyframes are stored with semantic embeddings, text queries are matched by cosine similarity, top-matching keyframes are segmented with SAM3, and resolved instances are cached as 3D point clouds plus oriented bounding boxes. This enables coarse queries such as “stove” for navigation and fine queries such as “knobs” on the stove for manipulation. Regions are similarly query-driven: place nodes are scored against a text query, smoothed on the places graph, and thresholded or clustered into connected components rather than being fixed in advance as room partitions (Maggio et al., 25 May 2026).

FOUND-IT reports 79% higher accuracy on the ASHiTA SG3D task grounding benchmark, real-time operation on a ground robot using a Jetson Thor, and runtime figures including about 6 fps with VGGT, about 7 fps with Depth Anything DA3-LARGE-1.1, and 4 Hz on a Jetson Thor mounted on Spot. A plausible implication is that TMRA can be deployed not only as an efficiency mechanism for offline inference, but also as an online policy for dynamically evolving task lists in embodied systems (Maggio et al., 25 May 2026).

5. Related multi-granularity formulations

Several adjacent methods instantiate closely related ideas even when they do not use the exact phrase TMRA. “Mixture-of-Resolution Adaptation” (MRA) for MLLMs is explicitly described as a concrete realization of task-driven multi-granularity resolution adaptation. It processes the same image through a low-resolution pathway for global structure and a high-resolution pathway for local detail, then injects high-resolution information into the low-resolution stream through MR-Adapters: $1\times 2$2 The adapter uses a dynamic gate,

$1\times 2$3

and the best reported design is convolution block for low-resolution mapping, MLP block for high-resolution mapping, and tanh as the gating function. The reported model, LLaVA-HR, handles images as large as 1536 \times 1536, outperforms existing MLLMs on 8 of 11 tasks, and achieves about 3$1\times 2$4 inference speed compared with LLaVA-1.5 at the same resolution (Luo et al., 2024).

Adaptive Multi-Resolution Attention (AdaMRA) moves the same principle into sequence modeling. Standard self-attention is described as both quadratic in sequence length and restricted to same-resolution processing. AdaMRA assigns each head $1\times 2$5 a compression rate $1\times 2$6, forms compressed memories $1\times 2$7, and lets each query route to a resolution-specific head through

$1\times 2$8

Each query is routed to the head with the highest router probability $1\times 2$9. The resulting architecture combines coarse-to-fine memory, query-driven routing, and kernel attention with linear complexity. On Long Range Arena, the reported average accuracy is 63.09, compared with 58.77 for vanilla Transformer and 59.43 for BigBird, with about 10.4$2 \times 1$0 speedup and about 20% of the memory of vanilla Transformer at sequence length 4K (Zhang et al., 2021).

In time series, MUG provides an implicit, latent-space TMRA framework. It combines a fine-grained timestamp-level path, a coarse-grained segment-level path, a cross-granularity transformer, and a retrieval-based unsupervised objective. Fine-grained features are summarized as

$2 \times 1$1

coarse features are projected into the fine-grained latent space by

$2 \times 1$2

and the final representation is trained with a retrieval ranking formulation converted into a BCE-style loss. On 10 UEA multivariate datasets, MUG (TST-ShapeNet) achieves best performance on 8 of 10 datasets, with average accuracy 0.768 and average rank 1.4 (Ye et al., 2023).

MG-MotionLLM transfers the same granularity logic to motion-language modeling. It distinguishes coarse sequence-level captions from fine-grained motion scripts, and trains a unified T5-based model with 28 tasks total: 12 classical coarse-grained tasks and 16 newly proposed fine-grained tasks. The fine-grained tasks include temporal boundary localization, detailed snippet captioning, and constrained motion generation with time and script conditions. The paper reports that direct instruction tuning on both coarse and detailed descriptions led to lower Top-3 retrieval accuracy (75.0%) than using coarse descriptions alone (77.3%), which motivates a granularity-synergy training scheme rather than a uniformly fine description regime (Wu et al., 3 Apr 2025).

6. Empirical regularities, misconceptions, and open problems

One recurrent misconception is that the highest available resolution is necessarily optimal. The available evidence contradicts that view. Task-aware resolution optimization reports that the best resolution is not always the highest one; some tasks perform best at $2 \times 1$3, others at $2 \times 1$4, and others at $2 \times 1$5 (Luo et al., 10 Oct 2025). AVG-LLaVA reports that text-heavy tasks shift toward finer granularity, while tasks where coarse granularity suffices show sharp reductions in token usage and substantial speed improvement (Lan et al., 2024). GeoMag similarly treats image-level, region-level, and pixel-level queries differently rather than uniformly maximizing spatial detail (Ma et al., 8 Jul 2025).

A second misconception is that TMRA is merely token compression. In AVG-LLaVA, the router uses both image content and instruction semantics, and RGLF aligns router predictions with downstream task utility as measured by answer log-likelihood (Lan et al., 2024). In GeoMag, the retained region is determined by prompt-conditioned PSC rather than by generic saliency (Ma et al., 8 Jul 2025). In FOUND-IT, object and region granularity are deferred until query time, so the representation changes as the robot’s task list evolves (Maggio et al., 25 May 2026). These formulations all indicate that TMRA is conditioned resolution selection rather than blind reduction.

The open problems are also explicit. Task-aware resolution optimization is currently task-level, not sample-level; its stated future direction is dynamic, sample-specific resolution adjustment (Luo et al., 10 Oct 2025). GeoMag reports spatial relationship errors and states that future work should improve understanding of spatial relationships (Ma et al., 8 Jul 2025). FOUND-IT addresses evolving task lists, but its significance also implies a broader systems challenge: granularity control must remain computationally practical within a real-time loop (Maggio et al., 25 May 2026).

Taken together, the current literature suggests a common operational definition. TMRA is the deliberate adjustment of representational granularity to the semantic and computational demands of the current task, with the decision conditioned by task semantics, image or sequence content, uncertainty, prompt relevance, or query structure. The concrete instantiations differ—sample-level routing, task-level resolution selection, prompt-aware cropping, query-time scene-graph construction, multi-resolution attention, and latent-space fusion—but they converge on the same principle: choose the minimal sufficient granularity rather than fixing resolution as an architectural constant (Lan et al., 2024, Luo et al., 10 Oct 2025, Ma et al., 8 Jul 2025).