---
title: Task-driven Multi-granularity Resolution Adjustment
url: https://www.emergentmind.com/topics/task-driven-multi-granularity-resolution-adjustment-tmra
type: topic
---

# Task-driven Multi-granularity Resolution Adjustment

Task-driven Multi-granularity Resolution Adjustment (TMRA) denotes a class of methods that match resolution or granularity to task demand instead of imposing a single fixed scale on all inputs. In visual large language models and related multimodal systems, the central premise is that coarse global understanding, region-centric reasoning, OCR-like perception, pixel-level parsing, navigation, and manipulation do not require the same perceptual granularity. TMRA therefore treats resolution as a task-sensitive modeling choice: it selects, mixes, or constructs the coarsest representation that is sufficient for the current image, instruction, prompt, or query, while preserving finer detail where downstream utility requires it [2410.02745][2510.09822][2507.05887].

## 1. Problem setting and conceptual scope

TMRA arises from a recurring mismatch between downstream task requirements and fixed-resolution model design. In high-resolution LMMs such as LLaVA-NeXT, one common strategy is to divide an image into multiple local images and one global image; this preserves detail but produces a large number of visual tokens and high quadratic attention cost. The underlying inefficiency is that many tasks do not need the finest detail: color questions or scene-level questions can often be answered from coarse evidence, whereas OCR, chart reading, or tiny text recognition may require finer granularity [2410.02745].

A closely related formulation appears in task-aware resolution optimization for VLLMs. There, the core argument is that real-world vision-language tasks do not all need the same image resolution, yet most VLLMs are built and used with a single fixed input resolution. The paper explicitly states that very low resolution can remove the detail needed for hard visual reasoning or OCR-like tasks, while very high resolution does not automatically help because it increases the number of image tokens, can misalign with the model’s learned visual representation, and may introduce irrelevant or redundant visual tokens that hurt performance unless the model is adapted. Across eight tasks, optimal resolutions are scattered across \(336^2\), \(448^2\), and \(560^2\), rather than collapsing to a single best setting [2510.09822].

The same logic is sharpened in remote sensing. GeoMag frames TMRA as a response to a mismatch between task granularity and input resolution: image-level tasks mostly need global semantics, region-level tasks need fine local detail, and pixel-level tasks require precise spatial structure where the target lies. High-resolution remote sensing imagery also contains large amounts of task-irrelevant background, so full-resolution processing wastes tokens, memory, and compute [2507.05887].

A plausible synthesis is that TMRA is best understood not as a single architecture, but as a principle of conditioned granularity control. The relevant conditioning signal may be the image-instruction pair, a benchmark-level task descriptor, a prompt, a query, or a query token itself.

## 2. Sample-level TMRA via adaptive granularity routing

AVG-LLaVA is a concrete implementation of TMRA for multimodal reasoning. Its guiding principle is explicit: choose the minimal sufficient resolution for the task, preserving or improving performance while reducing computation. The model adds two modules on top of LLaVA-NeXT: a parameter-free visual granularity scaler and a visual granularity router. With CLIP-ViT-L/336, each image initially yields a \(24 \times 24\) token grid; the scaler sequentially applies \(1\times 2\) and \(2 \times 1\) average pooling, preserving spatial layout while progressively reducing token count and producing levels such as \(24 \times 12\), \(12 \times 12\), \(12 \times 6\), and \(6 \times 6\). In the broader experimental setting, the router considers a granularity set like \(\{36,72,144,288,576\}\) tokens [2410.02745].

The router then selects which resolution to use for the current input. It concatenates tokens from all granularity levels,
\[
\overline{\mathbf{X}_v} = [\mathbf{X}_v^1;\mathbf{X}_v^2;\cdots;\mathbf{X}_v^N],
\]
filters instruction tokens by cosine similarity with the original-granularity visual tokens, retains the top-\(k\) instruction tokens as \(\overline{\mathbf{X}_{instruct}}\), and processes the combined sequence with three learned components: a single Transformer layer, an MLP layer, and a voter layer. The MLP predicts token-wise granularity logits
\[
\mathbf{Z}_{out} \in \mathbb{R}^{L \times N},
\]
and a learnable voter matrix
\[
\mathbf{W} \in \mathbb{R}^{1 \times L}
\]
aggregates these into final granularity logits
\[
\mathbf{Z}_{final} \in \mathbb{R}^{1 \times N}.
\]
A softmax over \(\mathbf{Z}_{final}\) yields a probability distribution over granularities, and the granularity with the highest probability is selected [2410.02745].

A central technical point is that the router is not trained by ordinary visual instruction tuning. AVG-LLaVA introduces RGLF, “Ranking Granularity to Align LMM Feedback,” which uses the LMM itself as a preference signal. For a given image and instruction, the system evaluates the answer log-likelihood under each granularity, sorts granularities in descending order of LMM answer probability, and trains the router with a pairwise ranking loss,
\[
\mathcal{L}_{rank} = \sum_{i=1} \sum_{j>i} \max(0, s_j - s_i + \lambda_{ij}),
\]
plus a cross-entropy term toward the single best granularity,
\[
\mathcal{L}_4 = \mathcal{L}_{rank} + \alpha \mathcal{L}_{ce}.
\]
This supervision requires no manually annotated granularity labels [2410.02745].

Empirically, this routing policy yields both efficiency and accuracy gains. The reported results include up to an **85.3% reduction in visual tokens** and up to a **2.53\(\times\) inference speedup** on AI2D. Representative benchmark values include **67.1 on TextVQA**, **74.6 on DocVQA**, **67.3 on AI2D**, **1557.4 on MME**, **69.9 on MMB**, and **37.4 on MMMU**. The reported visualizations also show the expected TMRA behavior: text-centric benchmarks tend to select fine tokens, while object-level reasoning tasks often prefer coarser grids [2410.02745].

## 3. Task-level TMRA through task-aware resolution optimization

A different TMRA formulation appears in task-aware resolution optimization for VLLMs. Instead of choosing a granularity separately for each sample, this method estimates a task-optimal discrete resolution for an entire benchmark or downstream task. The study covers ScienceQA-IMG, VizWiz, VQAv2, GQA, TextVQA, OKVQA, MMBench, and MMBench-CN, and reports that ScienceQA-IMG and VizWiz prefer \(336\times336\), VQAv2/TextVQA/GQA prefer \(448\times448\), and OKVQA/MMBench/MMBench-CN prefer \(560\times560\) [2510.09822].

The selection rule is built from two task-level heuristics. The first is image complexity \(C(T)\), defined as a task-level average complexity score based on a minimum-description-length clustering method for perceptual image complexity. The second is uncertainty variance \(V(T)\), defined from the relative change in model uncertainty when a pretrained VLLM is run at two resolutions, where uncertainty is measured from token-level entropy under random augmentations:
\[
H(p) = -\sum_{i=1}^{n} p_i \log p_i,\qquad
V(T) = \frac{U_2(T)-U_1(T)}{U_1(T)}.
\]
The authors report that the product \(C(T)\times V(T)\) is more consistent than either factor individually [2510.09822].

The resulting empirical formula is
\[
Reso(T) = Reso_0 \cdot (1 + k \cdot C(T) \cdot V(T)),
\]
where \(Reso_0\) is the baseline resolution, \(C(T)\) is average normalized image complexity, \(V(T)\) is average uncertainty variance, and \(k\) is a user-specified non-negative hyperparameter. In the reported implementation, \(Reso_0=336\), the search space is \(336^2\), \(448^2\), and \(560^2\), and **\(k=34\)** is selected using **SciQA-IMG**, **VQAv2**, and **OKVQA** as reference tasks. Intermediate outputs are mapped to the largest supported resolution not exceeding the predicted value [2510.09822].

TMRA in this setting does not stop at selecting a resolution. The method adapts **LLaVA-1.5-7B** to the selected higher resolution with a parameter-efficient fine-tuning strategy. The procedure first interpolates the visual positional embeddings to the new patch grid, then fine-tunes only three components: **position embeddings** in the visual encoder, **projector parameters**, and **LoRA adapter parameters** in the LLM backbone. All other parameters are frozen. The training setup follows LLaVA stage-2 fine-tuning with image-text pairs drawn from **COCO, GQA, OCR-VQA, TextVQA, and Visual Genome** [2510.09822].

The reported task-wise adaptive results include **VQAv2: 80.19**, **GQA: 63.87**, **TextVQA: 60.25**, **OKVQA: 50.86**, **MMBench: 67.70**, and **MMBench-CN: 61.08**. The paper also reports that naive training-free interpolation is worse than the adapted model, and that tuning **position embeddings + projector + LoRA adapters** is the best ablation setting [2510.09822].

## 4. Prompt- and query-conditioned TMRA

GeoMag implements TMRA as a prompt- and task-aware preprocessing policy for remote sensing image parsing across image-level, region-level, and pixel-level tasks. The system first classifies the query into one of three task types using **BERT-base** to produce a 768-dimensional representation, followed by FC layers with 256 neurons, 128 neurons, and an output layer with 3 neurons. The resulting task granularity \(G\) determines the image transformation:
\[
I^{\prime}= \begin{cases}
I_{100 \times 100}, & G=\text{Image} \\
C_1(I, P) \oplus D(I), & G=\text{Region} \\
C_2\left(C_1(I, P), P\right) \oplus D\left(C_1(I, P)\right) \oplus D(D(I)), & G=\text{Pixel}
\end{cases}
\]
with
\[
D(\cdot)=\operatorname{Resize}\left(\operatorname{Resize}\left(\cdot, \frac{H}{4} \times \frac{W}{4}\right), H \times W\right).
\]
Image-level queries therefore use a globally downsampled image; region-level queries keep one prompt-relevant region intact and compress the rest; pixel-level queries recursively crop more precisely and compress broader context more aggressively [2507.05887].

The task-relevant region is identified by Prompt-guided Semantic-aware Cropping (PSC), which constructs a gradient-weighted attention heatmap from last-layer attention:
\[
H=\frac{1}{N} \sum_{i=1}^N\left(\operatorname{ReLU}\left(\frac{\partial L}{\partial A_i}\right) \odot A_i\right).
\]
GeoMag then divides the image into grid cells, proposes candidate boxes, compares total attention inside each candidate with neighboring regions, and selects the box with the largest attention contrast. In the reported ablation, removing MagCrop leads to large performance drops: on RRSIS-D referring segmentation, **P@0.5 65.45, OIoU 74.20, MIoU 57.47** becomes **81.30, 82.67, 65.71** with the full model; on RefSegRS, **44.93, 62.74, 48.51** becomes **77.50, 80.27, 61.39**; and on referring object classification, **SS 75.04, SIoU 64.29** becomes **97.84, 96.47** [2507.05887].

FOUND-IT transfers the same principle to hierarchical 3D scene graphs. Its defining move is to make granularity a query-time decision rather than a mapping-time commitment. The system stores a lightweight geometric backbone continuously, then resolves objects, places, and regions at the right level of detail on demand. For objects, it uses a two-stage memory consisting of visual memory and cached memory: keyframes are stored with semantic embeddings, text queries are matched by cosine similarity, top-matching keyframes are segmented with SAM3, and resolved instances are cached as 3D point clouds plus oriented bounding boxes. This enables coarse queries such as “stove” for navigation and fine queries such as “knobs” on the stove for manipulation. Regions are similarly query-driven: place nodes are scored against a text query, smoothed on the places graph, and thresholded or clustered into connected components rather than being fixed in advance as room partitions [2605.25371].

FOUND-IT reports **79% higher accuracy on the ASHiTA SG3D task grounding benchmark**, real-time operation on a ground robot using a Jetson Thor, and runtime figures including about **6 fps** with VGGT, about **7 fps** with Depth Anything DA3-LARGE-1.1, and **4 Hz** on a Jetson Thor mounted on Spot. A plausible implication is that TMRA can be deployed not only as an efficiency mechanism for offline inference, but also as an online policy for dynamically evolving task lists in embodied systems [2605.25371].

## 5. Related multi-granularity formulations

Several adjacent methods instantiate closely related ideas even when they do not use the exact phrase TMRA. “Mixture-of-Resolution Adaptation” (MRA) for MLLMs is explicitly described as a concrete realization of task-driven multi-granularity resolution adaptation. It processes the same image through a low-resolution pathway for global structure and a high-resolution pathway for local detail, then injects high-resolution information into the low-resolution stream through MR-Adapters:
\[
F_y = F_{IL}(I_l, F_A(F_{uh})) + F_{uh}, \qquad F_{uh} = F_{IN}(I_h).
\]
The adapter uses a dynamic gate,
\[
g = \delta(W_2 \, \sigma(W_1 f_y)),
\]
and the best reported design is **convolution block** for low-resolution mapping, **MLP block** for high-resolution mapping, and **tanh** as the gating function. The reported model, LLaVA-HR, handles images as large as **1536 \times 1536**, outperforms existing MLLMs on **8 of 11** tasks, and achieves about **3\(\times\) inference speed** compared with LLaVA-1.5 at the same resolution [2403.03003].

Adaptive Multi-Resolution Attention (AdaMRA) moves the same principle into sequence modeling. Standard self-attention is described as both quadratic in sequence length and restricted to same-resolution processing. AdaMRA assigns each head \(h\) a compression rate \(c_h\), forms compressed memories \(\widetilde{K}^h,\widetilde{V}^h\), and lets each query route to a resolution-specific head through
\[
P = Softmax(F(Q)), \qquad F(Q)=QW.
\]
Each query is routed to the head with the highest router probability \(P\). The resulting architecture combines coarse-to-fine memory, query-driven routing, and kernel attention with linear complexity. On Long Range Arena, the reported average accuracy is **63.09**, compared with **58.77** for vanilla Transformer and **59.43** for BigBird, with about **10.4\(\times\)** speedup and about **20%** of the memory of vanilla Transformer at sequence length 4K [2108.04962].

In time series, MUG provides an implicit, latent-space TMRA framework. It combines a fine-grained timestamp-level path, a coarse-grained segment-level path, a cross-granularity transformer, and a retrieval-based unsupervised objective. Fine-grained features are summarized as
\[
v_{xi} = \text{Softmax}\left(\frac{v_{i,\text{Maxpooling}} \cdot v_i}{\sqrt{d}}\right)\cdot v_i,
\]
coarse features are projected into the fine-grained latent space by
\[
Attention(Q_x, K_S, V_S) = \text{softmax}\left(\frac{Q_xK_S^T}{\sqrt{d_k}}\right)V_S,
\]
and the final representation is trained with a retrieval ranking formulation converted into a BCE-style loss. On 10 UEA multivariate datasets, **MUG (TST-ShapeNet)** achieves best performance on **8 of 10** datasets, with average accuracy **0.768** and average rank **1.4** [2312.07248].

MG-MotionLLM transfers the same granularity logic to motion-language modeling. It distinguishes coarse sequence-level captions from fine-grained motion scripts, and trains a unified T5-based model with **28 tasks total**: **12 classical coarse-grained tasks** and **16 newly proposed fine-grained tasks**. The fine-grained tasks include temporal boundary localization, detailed snippet captioning, and constrained motion generation with time and script conditions. The paper reports that direct instruction tuning on both coarse and detailed descriptions led to **lower Top-3 retrieval accuracy (75.0%) than using coarse descriptions alone (77.3%)**, which motivates a granularity-synergy training scheme rather than a uniformly fine description regime [2504.02478].

## 6. Empirical regularities, misconceptions, and open problems

One recurrent misconception is that the highest available resolution is necessarily optimal. The available evidence contradicts that view. Task-aware resolution optimization reports that the best resolution is not always the highest one; some tasks perform best at \(336^2\), others at \(448^2\), and others at \(560^2\) [2510.09822]. AVG-LLaVA reports that text-heavy tasks shift toward finer granularity, while tasks where coarse granularity suffices show sharp reductions in token usage and substantial speed improvement [2410.02745]. GeoMag similarly treats image-level, region-level, and pixel-level queries differently rather than uniformly maximizing spatial detail [2507.05887].

A second misconception is that TMRA is merely token compression. In AVG-LLaVA, the router uses both image content and instruction semantics, and RGLF aligns router predictions with downstream task utility as measured by answer log-likelihood [2410.02745]. In GeoMag, the retained region is determined by prompt-conditioned PSC rather than by generic saliency [2507.05887]. In FOUND-IT, object and region granularity are deferred until query time, so the representation changes as the robot’s task list evolves [2605.25371]. These formulations all indicate that TMRA is conditioned resolution selection rather than blind reduction.

The open problems are also explicit. Task-aware resolution optimization is currently **task-level, not sample-level**; its stated future direction is **dynamic, sample-specific resolution adjustment** [2510.09822]. GeoMag reports spatial relationship errors and states that future work should improve understanding of spatial relationships [2507.05887]. FOUND-IT addresses evolving task lists, but its significance also implies a broader systems challenge: granularity control must remain computationally practical within a real-time loop [2605.25371].

Taken together, the current literature suggests a common operational definition. TMRA is the deliberate adjustment of representational granularity to the semantic and computational demands of the current task, with the decision conditioned by task semantics, image or sequence content, uncertainty, prompt relevance, or query structure. The concrete instantiations differ—sample-level routing, task-level resolution selection, prompt-aware cropping, query-time scene-graph construction, multi-resolution attention, and latent-space fusion—but they converge on the same principle: choose the minimal sufficient granularity rather than fixing resolution as an architectural constant [2410.02745][2510.09822][2507.05887].

Source: https://www.emergentmind.com/topics/task-driven-multi-granularity-resolution-adjustment-tmra