---
title: Decomposed Grounding with Selection
url: https://www.emergentmind.com/topics/decomposed-grounding-with-selection
type: topic
---

# Decomposed Grounding with Selection

Decomposed Grounding with Selection denotes a recurrent design in which a grounding problem is first factored into simpler units and then resolved by choosing, weighting, or verifying the units that best match the current input. In medical abnormality grounding, the decomposition transforms a named abnormality such as “lung opacity” into attribute- and pattern-level textual cues that can align with image features, and selection is realized implicitly through multimodal attention during autoregressive box-token decoding [2503.03278]. In the broader literature, closely related formulations decompose temporal boundary prediction from top-1 proposal choice in weakly supervised video grounding, tile high-resolution GUI screenshots before candidate adjudication, separate disjunctive STL branches before heuristic branch selection, split multi-anchor 3D descriptions into single-anchor statements before multi-view fusion, or choose among context-dependent Toffoli decompositions only after per-instance verification [2503.23181][2507.22025][2605.01222][2507.11261][2606.31791]. This suggests that the phrase names not a single fixed architecture but a family of instance-adaptive grounding procedures.

## 1. Definition and conceptual scope

In the cited works, “decomposition” refers to replacing a monolithic grounding target with a structured intermediate representation, while “selection” refers to the mechanism that identifies which decomposed unit, candidate, or verified rewrite should govern the final prediction. The decomposition may be semantic, spatial, temporal, logical, or algebraic; the selection may be implicit in attention weights, explicit in argmax scoring, or enforced through correctness verification.

| Domain | Decomposition | Selection |
|---|---|---|
| Medical abnormality grounding | Abnormality $\rightarrow$ attributes and common visual patterns | Implicit via cross-attention [2503.03278] |
| Weakly supervised video grounding | Boundary prediction $\rightarrow$ proposal selection | IoU, Loss, IoU+LossSum, IoU+LossMax [2503.23181] |
| GUI grounding | Screenshot $\rightarrow$ overlapping tiles or retained tokens | QA scoring, top-$k$, or correction-model choice [2507.22025][2601.03928][2605.06664] |
| STL planning | STL formula $\rightarrow$ disjunction-free branches and sub-tasks | Heuristic branch score $S(\phi_k)$ [2605.01222] |
| 3D visual grounding | Query $\rightarrow$ single-anchor statements or attribute/spatial branches | View/token/object score aggregation [2507.11261][2406.08907] |
| Toffoli compilation | CCX gate $\rightarrow$ exact, RP, approximate, or control-drop variants | Budget-aware choice gated by verification [2606.31791] |

A common point across these instantiations is that selection is not merely a post-processing convenience. In each case it is the mechanism that turns decomposition into a usable grounding procedure: without selection, decomposition would only proliferate alternatives; without decomposition, selection would operate on undifferentiated inputs and lose the fine-grained structure the method is designed to expose.

## 2. Medical abnormality localization: the canonical formulation

In medical abnormality grounding, the task is: given a chest X-ray and a textual query describing an abnormality, the model must detect and localize the abnormal region or regions by outputting bounding boxes as discrete tokens. The paper frames this as autoregressive sequence generation rather than standard box regression, or “detection as language.” A medical concept is defined as a named abnormality with a definition from clinical sources, and decomposition maps that concept into fundamental attributes—shape, location, density, and color—and into common visual patterns such as “white or grayish patch,” “linear band,” “rounded nodule,” or “pleural line without lung markings” [2503.03278].

The construction pipeline begins from VinDr-CXR definitions. GPT-4 is prompted with a fixed template: “Here is the medical definition of [abnormality name]: '[definition]'. Based on this definition and focusing on the following visual attributes (e.g., shape, location, density, color), provide a brief description of the abnormality.” For lung opacity, the definition “Any abnormal focal or generalized opacity…” is rewritten as “An area of increased density in the lung fields, typically appearing as a white or grayish patch.” The training prompt then contains the abnormality label plus the decomposed description, so the visual-language model receives attribute-specific cues and pattern phrases rather than only an abstract abnormality name [2503.03278].

The model is Florence-2-base with 0.23B parameters, a DaViT visual encoder, and a multimodal encoder-decoder. From image $I$, the visual encoder produces $N$ visual token embeddings $V \in \mathbb{R}^{N \times D}$. Coordinates are quantized to $[0,1000]$, with
$$
l_x = (x_{\text{pixel}} / W)\cdot 1000,\qquad l_y = (y_{\text{pixel}} / H)\cdot 1000.
$$
Each predicted box $L_i = \{l_{x_0}^i, l_{y_0}^i, l_{x_1}^i, l_{y_1}^i\}$ is thus a sequence of discrete tokens. The paper formalizes decomposition by letting a concept $c$ be split into attributes $A = \{a_i\}$ and patterns $P = \{p_j\}$, with textual embeddings $f_t(a_i)$ and $f_t(p_j)$, and region features $f_v(x,r)$ conceptualized as an aggregation over visual tokens inside region $r$. Attribute- and pattern-region similarities are scored as
$$
s_i(r) = \langle f_t(a_i), f_v(x,r)\rangle,\qquad s'_j(r) = \langle f_t(p_j), f_v(x,r)\rangle,
$$
with softmax selection weights over attributes and patterns, and an aggregated grounding score
$$
S_c(r) = \sum_i w_i(r)s_i(r) + \sum_j u_j(r)s'_j(r).
$$
Training minimizes cross-entropy over autoregressively decoded localization tokens,
$$
L = -\sum_{t=1}^{T} \log p(y_t \mid V, T).
$$
No explicit contrastive or IoU regression loss is added; localization quality emerges from sequence modeling and attention-based alignment [2503.03278].

Selection in this formulation is implicit. There is no separate top-$k$ or gating module. During decoding, the multimodal transformer’s attention weights over text attribute/pattern tokens and visual tokens act as relevance scores, so knowledge-enhanced prompts sharpen the attention-based selection compared with a baseline that uses only abnormality labels. The result is a bridge from abstract medical terminology to pixel-level appearance, especially for abnormalities whose clinical names do not themselves encode obvious visual cues [2503.03278].

## 3. Decomposition strategies across domains

The literature exhibits several distinct decomposition regimes. In weakly supervised video grounding, the decomposition is explicitly staged: first derive candidate temporal boundaries $(s,e)$ from Gaussian mixture proposals, then select the final top-1 prediction among multiple proposals. The paper explores five boundary prediction strategies—Long Tail, Short Tail, Shortest Tail, Average, and Attention—and four selection strategies—IoU voting, Loss, IoU+LossSum, and IoU+LossMax—without requiring additional training [2503.23181].

In GUI grounding, decomposition is often spatial. UI-AGILE partitions a high-resolution screenshot into overlapping tiles, runs the grounding model independently on each tile, crops an element image around each tile-level point prediction, and uses a selection VLM to answer “Does this image match the instruction?” with the “Yes” token’s logit as the relevance score. The default uses $T=4$ tiles, each of width $0.6 \cdot W$ and height $0.6 \cdot H$, with overlap $0.1 \cdot W$ and $0.1 \cdot H$ [2507.22025]. FocusUI instead decomposes the visual stream into retained and dropped patches: it constructs fused patch-level saliency supervision from an instruction-conditioned overlap score and a rule-based UI-graph score, selects a top-$k$ subset of visual tokens, and then preserves positional continuity with PosPad, which compresses each contiguous dropped run into a single special marker placed at the run’s last index [2601.03928]. BAMI decomposes inference into coarse-to-fine focus and candidate selection, using masked prediction to produce mutually exclusive candidate boxes and a correction model to reorder them [2605.06664].

In logic-guided planning, the decomposition is symbolic and temporal. The zero-shot STL planner converts an STL formula into a DNF-style strengthening $\tilde{\phi} = \bigvee_k \phi_k$, where each branch has no explicit disjunctions, and then decomposes the selected branch into sub-tasks with explicit time intervals. Temporal grounding is delegated to a time predictor trained with Transitive Reinforcement Learning, while branch selection is performed by a score that balances temporal-operator complexity, map reachability, and temporal slack [2605.01222].

In 3D visual grounding, decomposition appears in both language and multimodal structure. ViewSRD restructures complex multi-anchor queries into single-anchor statements through Simple Relation Decoupling, encodes the original query and the decomposed statements, and then injects viewpoint semantics through shared Cross-modal Consistent View Tokens across multi-view textual-scene interaction [2507.11261]. DASANet separates language into attribute text and spatial-relation text, decomposes each 3D object into an attribute point set and a spatial bounding-box representation, and processes them with dual branches whose scores are later combined [2406.08907].

A different but structurally analogous decomposition appears in the Toffoli-network compiler. There the object being decomposed is not a query or perceptual input but a CCX gate instance. The admissible candidates include exact CCX, relative-phase Toffoli, conditionally-clean ancilla Toffoli, approximate Toffoli, and exact-on-reachable control drop. Selection is only allowed after context analysis computes a reachable basis set and a verification gate certifies observational equivalence on the reachable input domain [2606.31791]. This suggests that Decomposed Grounding with Selection can extend beyond perception tasks to any setting where a complex object is replaced by a structured candidate set whose correctness is instance-dependent.

## 4. Selection mechanisms

Selection mechanisms in this literature range from fully implicit weighting to explicit decision rules. In the medical formulation, selection is emergent from transformer cross-attention: higher attention to specific attribute or pattern tokens biases autoregressive box-token generation toward corresponding visual regions, and the paper explicitly states that no explicit gating, ranking, or top-$k$ module is added [2503.03278]. This is a soft, token-level, query-conditioned selection.

Weakly supervised video grounding makes selection explicit and discrete. For each proposal, IoU voting computes agreement with the other $N-1$ proposals; Loss selects the proposal with the lowest cross-entropy reconstruction loss; IoU+LossSum and IoU+LossMax weight the IoU vote by normalized proposal quality, with IoU+LossMax designed for stronger suppression of poor proposals. The final top-1 prediction is then $\arg\max_n S^{(n)}$ or $\arg\min_n L^{(n)}_{ce}$, depending on the strategy [2503.23181].

GUI grounding supplies several explicit selection paradigms. UI-AGILE uses binary QA adjudication over cropped element images, with $S_i = z_{\text{yes}}(E_i,q)$ and $i^* = \arg\max_i S_i$ [2507.22025]. FocusUI performs top-$k$ visual token selection using the predicted saliency scores $s_{\text{pred}}(i)$ and a retention ratio $r$, with $K = \lfloor r\cdot M \rfloor$ and a threshold at the $K$-th largest score [2601.03928]. BAMI uses a correction model that reorders masked candidates using prompt principles such as “Functional Preference,” “Memory Comparison,” and “Interactive Components,” and then crops around the preferred box for the next iteration [2605.06664].

Selection can also be heuristic or proof-theoretic. In STL planning, one branch is chosen by minimizing
$$
S(\phi_k) = \alpha C_{\text{op}}(\phi_k) + \beta C_{\text{map}}(\phi_k) - \gamma C_{\text{slack}}(\phi_k),
$$
where $C_{\text{op}}$, $C_{\text{map}}$, and $C_{\text{slack}}$ summarize operator complexity, map reachability, and temporal slack, respectively [2605.01222]. In DETACH, selection is neither argmax over candidates nor attention over text tokens, but adaptive reweighting of negatives in a weighted InfoNCE loss: hard negatives are upweighted through $W_{ij}^{\text{spatial}}$, probable false negatives are downweighted through $W_{ij}^{\text{temporal}}$, and the final weight is $W_{ij} = W_{ij}^{\text{spatial}} W_{ij}^{\text{temporal}}$ [2512.20409]. In the Toffoli compiler, the selected decomposition is the one minimizing the two-qubit infidelity budget among admissible candidates, but only after a per-instance verification gate discharges the correctness obligation; if verification fails, the pass falls back to the cheapest certified alternative [2606.31791].

Taken together, these mechanisms show that “selection” in this family of methods is a broad technical category. It can mean attention-based scoring, vote aggregation, prompt-conditioned reranking, heuristic branch choice, adaptive negative weighting, or theorem-backed admissibility filtering.

## 5. Empirical results and efficiency

Reported results indicate that decomposition and selection are often responsible for measurable gains in accuracy, generalization, or efficiency rather than only interpretability. The medical system trained on VinDr-CXR achieves competitive abnormality grounding with a compact 0.23B Florence-2-base model trained on 16,087 image–abnormality pairs, despite comparisons against 7B and 13B medical VLMs trained on much larger datasets [2503.03278]. In weakly supervised video grounding, inference-only changes improve both Charades-STA and ActivityNet Captions without retraining [2503.23181]. In GUI grounding, decomposition is especially effective on high-resolution ScreenSpot-Pro, while already-cropped ScreenSpot-v2 benefits less or does not use decomposition [2507.22025][2605.06664].

| System | Representative result | Setting |
|---|---|---|
| Medical abnormality grounding [2503.03278] | VinDr-CXR: mAP50 25.5, mAP75 7.45, mAP50:95 10.81, RoDeO $R_{\text{total}}$ 54.38 | 0.23B Florence-2-base; first on 14/21 disease classes by RoDeO total |
| WSVG inference strategies [2503.23181] | Charades-STA: mIoU 45.95, IoU@0.5 51.39, IoU@0.7 25.90 | Shortest Tail + IoU+LossMax |
| UI-AGILE [2507.22025] | ScreenSpot-Pro: using training and inference enhancement methods brings 23% grounding accuracy improvement over the best baseline | Decomposed Grounding with Selection at inference |
| FocusUI [2601.03928] | ScreenSpot-Pro: FocusUI-7B improves 3.7% over GUI-Actor-7B; at 30% retention, up to 1.44x faster inference and 17% lower peak GPU memory | Position-preserving visual token selection |
| BAMI [2605.06664] | TianXi-Action-7B: 51.9% $\rightarrow$ 57.8% on ScreenSpot-Pro | Training-free coarse-to-fine focus + candidate selection |
| ViewSRD [2507.11261] | Nr3D overall accuracy 69.9% | Gains larger under view-dependent settings |
| DASANet [2406.08907] | Nr3D overall 65.1%, 1.3% higher than the best competitor | Dual attribute-spatial alignment |
| DETACH [2512.20409] | Opportunity++: 0.73 F1 / 0.83 mAP | Exocentric video and ambient sensors |
| Toffoli selection pass [2606.31791] | Up to 39.5% fewer two-qubit gates and 36.7% lower infidelity over exact-only | Compute/uncompute-heavy suite |

The medical ablations make the role of decomposition particularly explicit. On VinDr-CXR, knowledge-enhanced prompts improve mAP50 from 13.26 to 25.50, mAP75 from 2.76 to 7.45, and RoDeO total from 45.22 to 54.38; on PadChest-unknown, they improve mAP50 from 1.48 to 3.05, mAP75 from 0.03 to 0.29, and $R_{\text{total}}$ from 32.05 to 33.72 [2503.03278]. In the video-grounding study, Shortest Tail generally gives the most precise localization, and IoU+LossMax consistently outperforms IoU-only and Loss-only across boundary strategies; on Charades-STA, Long Tail + IoU+LossMax improves IoU@0.5 from 35.81% to 51.23% [2503.23181].

Efficiency claims are likewise bound to decomposition. UI-AGILE reports that with a VLM whose self-attention complexity is $O(n^2)$ on image token length $n$, decomposing into $T$ tiles reduces the dominant prefilling cost to approximately $n^2/T$; with $T=4$, this is approximately $n^2/4$ [2507.22025]. FocusUI shows that retaining only 30% of visual tokens can preserve near-baseline accuracy because PosPad maintains positional continuity, a property that general visual token pruning methods often break [2601.03928]. In the Toffoli setting, the gains are hardware-relevant rather than perceptual: the selector minimizes a two-qubit-infidelity budget and certifies every committed substitution beyond the exhaustive-verification limit with zero certified errors [2606.31791].

## 6. Limitations, failure modes, and broader interpretations

The literature also places clear bounds on when decomposition and selection help. In medical abnormality grounding, the method depends on high-quality definitions and LLM rewriting; the compact attribute set of shape, location, density, and color may not capture modality-specific nuances such as distribution patterns, temporal change, or comparative signs, and attention-only selection may be suboptimal in ambiguous cases [2503.03278]. In weakly supervised video grounding, the framework relies on Gaussian mixture parameterizations being well learned during training, and the best selection rule is dataset-dependent: IoU+LossMax performs best on Charades-STA, whereas IoU+LossSum performs best on ActivityNet Captions [2503.23181].

GUI decompositions can fail through overcropping or context loss. UI-AGILE notes that very large elements or split visuals may still yield partial crops with lower “Yes” scores, and that ambiguous instructions or multiple similar elements can make selection indecisive [2507.22025]. FocusUI reports that aggressive retention may remove necessary background or relational context, especially for instructions such as “the button under the second chart” [2601.03928]. BAMI similarly warns that if the crop ratio $\lambda$ is too small, cropping may remove needed context, and if the number of focus iterations is too large, cumulative cropping can degrade performance [2605.06664].

Logical and algebraic decompositions introduce a different kind of limitation: soundness. The STL planner’s DNF conversion is explicitly a strengthening, not an equivalence-preserving rewrite; $\tilde{\phi} \Rightarrow \phi$, but not conversely, so some behaviors that satisfy the original formula may be excluded [2605.01222]. In the Toffoli compiler, the central safety claim is that pattern-matched relative-phase substitution is silently incorrect: the verifier flagged 66 library rewrites as non-equivalent without a context check, and count-greedy substitution silently corrupted 6 of 12 benchmark circuits [2606.31791]. These cases show that decomposition can enlarge the admissible candidate set only when a sufficiently strong selection criterion rejects invalid candidates.

A broader theoretical interpretation appears in the philosophical framework “On measuring grounding and generalizing grounding problems,” which decomposes grounding itself into authenticity, preservation, faithfulness, robustness, and compositionality, all indexed by an evaluation tuple $(C, M, T, R)$. In that framework, “selection-for-success” is formalized etiologically through an average causal effect criterion,
$$
\text{ACE}_{E}(M)=\mathbb{E}\!\left[\Pr(\text{succ}\mid \text{do}(M=\text{on}))-\Pr(\text{succ}\mid \text{do}(M=\text{off}))\right]\ge \eta^{k,t}.
$$
This suggests that algorithmic Decomposed Grounding with Selection can be understood not only as an engineering pattern for localization and decision-making, but also as part of a larger program that asks whether internal mechanisms are causally responsible for successful grounding claims [2512.06205].

Across these strands, the most stable conclusion is technical rather than terminological. Decomposition is useful when the original grounding target is too abstract, too high-dimensional, too ambiguous, or too context-dependent to be handled reliably in one step. Selection is useful when the decomposed representation introduces multiple plausible candidates whose relevance is instance-specific. The cited works differ sharply in modality and formalism, but they converge on that joint logic.

Source: https://www.emergentmind.com/topics/decomposed-grounding-with-selection