Decomposed Grounding with Selection

Updated 7 July 2026

Decomposed Grounding with Selection is defined as a process that factors a complex grounding target into structured sub-units (such as attributes and patterns) before applying a selection mechanism to pick the most relevant candidate.
It leverages diverse strategies—including implicit attention and explicit heuristic rules—to align decomposed components with multimodal inputs across domains like medical imaging, video analysis, and GUI interpretation.
Empirical results demonstrate that integrating decomposition with selection enhances performance metrics and computational efficiency, yielding notable improvements in localization accuracy and reduced inference cost.

Decomposed Grounding with Selection denotes a recurrent design in which a grounding problem is first factored into simpler units and then resolved by choosing, weighting, or verifying the units that best match the current input. In medical abnormality grounding, the decomposition transforms a named abnormality such as “lung opacity” into attribute- and pattern-level textual cues that can align with image features, and selection is realized implicitly through multimodal attention during autoregressive box-token decoding (Li et al., 5 Mar 2025). In the broader literature, closely related formulations decompose temporal boundary prediction from top-1 proposal choice in weakly supervised video grounding, tile high-resolution GUI screenshots before candidate adjudication, separate disjunctive STL branches before heuristic branch selection, split multi-anchor 3D descriptions into single-anchor statements before multi-view fusion, or choose among context-dependent Toffoli decompositions only after per-instance verification (Kim et al., 29 Mar 2025, Lian et al., 29 Jul 2025, Ye et al., 2 May 2026, Huang et al., 15 Jul 2025, Bartkiewicz et al., 30 Jun 2026). This suggests that the phrase names not a single fixed architecture but a family of instance-adaptive grounding procedures.

1. Definition and conceptual scope

In the cited works, “decomposition” refers to replacing a monolithic grounding target with a structured intermediate representation, while “selection” refers to the mechanism that identifies which decomposed unit, candidate, or verified rewrite should govern the final prediction. The decomposition may be semantic, spatial, temporal, logical, or algebraic; the selection may be implicit in attention weights, explicit in argmax scoring, or enforced through correctness verification.

Domain	Decomposition	Selection
Medical abnormality grounding	Abnormality $\rightarrow$ attributes and common visual patterns	Implicit via cross-attention (Li et al., 5 Mar 2025)
Weakly supervised video grounding	Boundary prediction $\rightarrow$ proposal selection	IoU, Loss, IoU+LossSum, IoU+LossMax (Kim et al., 29 Mar 2025)
GUI grounding	Screenshot $\rightarrow$ overlapping tiles or retained tokens	QA scoring, top- $k$ , or correction-model choice (Lian et al., 29 Jul 2025, Ouyang et al., 7 Jan 2026, Zhang et al., 7 May 2026)
STL planning	STL formula $\rightarrow$ disjunction-free branches and sub-tasks	Heuristic branch score $S(\phi_k)$ (Ye et al., 2 May 2026)
3D visual grounding	Query $\rightarrow$ single-anchor statements or attribute/spatial branches	View/token/object score aggregation (Huang et al., 15 Jul 2025, Xu et al., 2024)
Toffoli compilation	CCX gate $\rightarrow$ exact, RP, approximate, or control-drop variants	Budget-aware choice gated by verification (Bartkiewicz et al., 30 Jun 2026)

A common point across these instantiations is that selection is not merely a post-processing convenience. In each case it is the mechanism that turns decomposition into a usable grounding procedure: without selection, decomposition would only proliferate alternatives; without decomposition, selection would operate on undifferentiated inputs and lose the fine-grained structure the method is designed to expose.

2. Medical abnormality localization: the canonical formulation

In medical abnormality grounding, the task is: given a chest X-ray and a textual query describing an abnormality, the model must detect and localize the abnormal region or regions by outputting bounding boxes as discrete tokens. The paper frames this as autoregressive sequence generation rather than standard box regression, or “detection as language.” A medical concept is defined as a named abnormality with a definition from clinical sources, and decomposition maps that concept into fundamental attributes—shape, location, density, and color—and into common visual patterns such as “white or grayish patch,” “linear band,” “rounded nodule,” or “pleural line without lung markings” (Li et al., 5 Mar 2025).

The construction pipeline begins from VinDr-CXR definitions. GPT-4 is prompted with a fixed template: “Here is the medical definition of [abnormality name]: '[definition]'. Based on this definition and focusing on the following visual attributes (e.g., shape, location, density, color), provide a brief description of the abnormality.” For lung opacity, the definition “Any abnormal focal or generalized opacity…” is rewritten as “An area of increased density in the lung fields, typically appearing as a white or grayish patch.” The training prompt then contains the abnormality label plus the decomposed description, so the visual-LLM receives attribute-specific cues and pattern phrases rather than only an abstract abnormality name (Li et al., 5 Mar 2025).

The model is Florence-2-base with 0.23B parameters, a DaViT visual encoder, and a multimodal encoder-decoder. From image $I$ , the visual encoder produces $N$ visual token embeddings $\rightarrow$ 0. Coordinates are quantized to $\rightarrow$ 1, with

$\rightarrow$ 2

Each predicted box $\rightarrow$ 3 is thus a sequence of discrete tokens. The paper formalizes decomposition by letting a concept $\rightarrow$ 4 be split into attributes $\rightarrow$ 5 and patterns $\rightarrow$ 6, with textual embeddings $\rightarrow$ 7 and $\rightarrow$ 8, and region features $\rightarrow$ 9 conceptualized as an aggregation over visual tokens inside region $\rightarrow$ 0. Attribute- and pattern-region similarities are scored as

$\rightarrow$ 1

with softmax selection weights over attributes and patterns, and an aggregated grounding score

$\rightarrow$ 2

Training minimizes cross-entropy over autoregressively decoded localization tokens,

$\rightarrow$ 3

No explicit contrastive or IoU regression loss is added; localization quality emerges from sequence modeling and attention-based alignment (Li et al., 5 Mar 2025).

Selection in this formulation is implicit. There is no separate top- $\rightarrow$ 4 or gating module. During decoding, the multimodal transformer’s attention weights over text attribute/pattern tokens and visual tokens act as relevance scores, so knowledge-enhanced prompts sharpen the attention-based selection compared with a baseline that uses only abnormality labels. The result is a bridge from abstract medical terminology to pixel-level appearance, especially for abnormalities whose clinical names do not themselves encode obvious visual cues (Li et al., 5 Mar 2025).

3. Decomposition strategies across domains

The literature exhibits several distinct decomposition regimes. In weakly supervised video grounding, the decomposition is explicitly staged: first derive candidate temporal boundaries $\rightarrow$ 5 from Gaussian mixture proposals, then select the final top-1 prediction among multiple proposals. The paper explores five boundary prediction strategies—Long Tail, Short Tail, Shortest Tail, Average, and Attention—and four selection strategies—IoU voting, Loss, IoU+LossSum, and IoU+LossMax—without requiring additional training (Kim et al., 29 Mar 2025).

In GUI grounding, decomposition is often spatial. UI-AGILE partitions a high-resolution screenshot into overlapping tiles, runs the grounding model independently on each tile, crops an element image around each tile-level point prediction, and uses a selection VLM to answer “Does this image match the instruction?” with the “Yes” token’s logit as the relevance score. The default uses $\rightarrow$ 6 tiles, each of width $\rightarrow$ 7 and height $\rightarrow$ 8, with overlap $\rightarrow$ 9 and $k$ 0 (Lian et al., 29 Jul 2025). FocusUI instead decomposes the visual stream into retained and dropped patches: it constructs fused patch-level saliency supervision from an instruction-conditioned overlap score and a rule-based UI-graph score, selects a top- $k$ 1 subset of visual tokens, and then preserves positional continuity with PosPad, which compresses each contiguous dropped run into a single special marker placed at the run’s last index (Ouyang et al., 7 Jan 2026). BAMI decomposes inference into coarse-to-fine focus and candidate selection, using masked prediction to produce mutually exclusive candidate boxes and a correction model to reorder them (Zhang et al., 7 May 2026).

In logic-guided planning, the decomposition is symbolic and temporal. The zero-shot STL planner converts an STL formula into a DNF-style strengthening $k$ 2, where each branch has no explicit disjunctions, and then decomposes the selected branch into sub-tasks with explicit time intervals. Temporal grounding is delegated to a time predictor trained with Transitive Reinforcement Learning, while branch selection is performed by a score that balances temporal-operator complexity, map reachability, and temporal slack (Ye et al., 2 May 2026).

In 3D visual grounding, decomposition appears in both language and multimodal structure. ViewSRD restructures complex multi-anchor queries into single-anchor statements through Simple Relation Decoupling, encodes the original query and the decomposed statements, and then injects viewpoint semantics through shared Cross-modal Consistent View Tokens across multi-view textual-scene interaction (Huang et al., 15 Jul 2025). DASANet separates language into attribute text and spatial-relation text, decomposes each 3D object into an attribute point set and a spatial bounding-box representation, and processes them with dual branches whose scores are later combined (Xu et al., 2024).

A different but structurally analogous decomposition appears in the Toffoli-network compiler. There the object being decomposed is not a query or perceptual input but a CCX gate instance. The admissible candidates include exact CCX, relative-phase Toffoli, conditionally-clean ancilla Toffoli, approximate Toffoli, and exact-on-reachable control drop. Selection is only allowed after context analysis computes a reachable basis set and a verification gate certifies observational equivalence on the reachable input domain (Bartkiewicz et al., 30 Jun 2026). This suggests that Decomposed Grounding with Selection can extend beyond perception tasks to any setting where a complex object is replaced by a structured candidate set whose correctness is instance-dependent.

4. Selection mechanisms

Selection mechanisms in this literature range from fully implicit weighting to explicit decision rules. In the medical formulation, selection is emergent from transformer cross-attention: higher attention to specific attribute or pattern tokens biases autoregressive box-token generation toward corresponding visual regions, and the paper explicitly states that no explicit gating, ranking, or top- $k$ 3 module is added (Li et al., 5 Mar 2025). This is a soft, token-level, query-conditioned selection.

Weakly supervised video grounding makes selection explicit and discrete. For each proposal, IoU voting computes agreement with the other $k$ 4 proposals; Loss selects the proposal with the lowest cross-entropy reconstruction loss; IoU+LossSum and IoU+LossMax weight the IoU vote by normalized proposal quality, with IoU+LossMax designed for stronger suppression of poor proposals. The final top-1 prediction is then $k$ 5 or $k$ 6, depending on the strategy (Kim et al., 29 Mar 2025).

GUI grounding supplies several explicit selection paradigms. UI-AGILE uses binary QA adjudication over cropped element images, with $k$ 7 and $k$ 8 (Lian et al., 29 Jul 2025). FocusUI performs top- $k$ 9 visual token selection using the predicted saliency scores $\rightarrow$ 0 and a retention ratio $\rightarrow$ 1, with $\rightarrow$ 2 and a threshold at the $\rightarrow$ 3-th largest score (Ouyang et al., 7 Jan 2026). BAMI uses a correction model that reorders masked candidates using prompt principles such as “Functional Preference,” “Memory Comparison,” and “Interactive Components,” and then crops around the preferred box for the next iteration (Zhang et al., 7 May 2026).

Selection can also be heuristic or proof-theoretic. In STL planning, one branch is chosen by minimizing

$\rightarrow$ 4

where $\rightarrow$ 5, $\rightarrow$ 6, and $\rightarrow$ 7 summarize operator complexity, map reachability, and temporal slack, respectively (Ye et al., 2 May 2026). In DETACH, selection is neither argmax over candidates nor attention over text tokens, but adaptive reweighting of negatives in a weighted InfoNCE loss: hard negatives are upweighted through $\rightarrow$ 8, probable false negatives are downweighted through $\rightarrow$ 9, and the final weight is $S(\phi_k)$ 0 (Yoon et al., 23 Dec 2025). In the Toffoli compiler, the selected decomposition is the one minimizing the two-qubit infidelity budget among admissible candidates, but only after a per-instance verification gate discharges the correctness obligation; if verification fails, the pass falls back to the cheapest certified alternative (Bartkiewicz et al., 30 Jun 2026).

Taken together, these mechanisms show that “selection” in this family of methods is a broad technical category. It can mean attention-based scoring, vote aggregation, prompt-conditioned reranking, heuristic branch choice, adaptive negative weighting, or theorem-backed admissibility filtering.

5. Empirical results and efficiency

Reported results indicate that decomposition and selection are often responsible for measurable gains in accuracy, generalization, or efficiency rather than only interpretability. The medical system trained on VinDr-CXR achieves competitive abnormality grounding with a compact 0.23B Florence-2-base model trained on 16,087 image–abnormality pairs, despite comparisons against 7B and 13B medical VLMs trained on much larger datasets (Li et al., 5 Mar 2025). In weakly supervised video grounding, inference-only changes improve both Charades-STA and ActivityNet Captions without retraining (Kim et al., 29 Mar 2025). In GUI grounding, decomposition is especially effective on high-resolution ScreenSpot-Pro, while already-cropped ScreenSpot-v2 benefits less or does not use decomposition (Lian et al., 29 Jul 2025, Zhang et al., 7 May 2026).

System	Representative result	Setting
Medical abnormality grounding (Li et al., 5 Mar 2025)	VinDr-CXR: mAP50 25.5, mAP75 7.45, mAP50:95 10.81, RoDeO $S(\phi_k)$ 1 54.38	0.23B Florence-2-base; first on 14/21 disease classes by RoDeO total
WSVG inference strategies (Kim et al., 29 Mar 2025)	Charades-STA: mIoU 45.95, [email protected] 51.39, [email protected] 25.90	Shortest Tail + IoU+LossMax
UI-AGILE (Lian et al., 29 Jul 2025)	ScreenSpot-Pro: using training and inference enhancement methods brings 23% grounding accuracy improvement over the best baseline	Decomposed Grounding with Selection at inference
FocusUI (Ouyang et al., 7 Jan 2026)	ScreenSpot-Pro: FocusUI-7B improves 3.7% over GUI-Actor-7B; at 30% retention, up to 1.44x faster inference and 17% lower peak GPU memory	Position-preserving visual token selection
BAMI (Zhang et al., 7 May 2026)	TianXi-Action-7B: 51.9% $S(\phi_k)$ 2 57.8% on ScreenSpot-Pro	Training-free coarse-to-fine focus + candidate selection
ViewSRD (Huang et al., 15 Jul 2025)	Nr3D overall accuracy 69.9%	Gains larger under view-dependent settings
DASANet (Xu et al., 2024)	Nr3D overall 65.1%, 1.3% higher than the best competitor	Dual attribute-spatial alignment
DETACH (Yoon et al., 23 Dec 2025)	Opportunity++: 0.73 F1 / 0.83 mAP	Exocentric video and ambient sensors
Toffoli selection pass (Bartkiewicz et al., 30 Jun 2026)	Up to 39.5% fewer two-qubit gates and 36.7% lower infidelity over exact-only	Compute/uncompute-heavy suite

The medical ablations make the role of decomposition particularly explicit. On VinDr-CXR, knowledge-enhanced prompts improve mAP50 from 13.26 to 25.50, mAP75 from 2.76 to 7.45, and RoDeO total from 45.22 to 54.38; on PadChest-unknown, they improve mAP50 from 1.48 to 3.05, mAP75 from 0.03 to 0.29, and $S(\phi_k)$ 3 from 32.05 to 33.72 (Li et al., 5 Mar 2025). In the video-grounding study, Shortest Tail generally gives the most precise localization, and IoU+LossMax consistently outperforms IoU-only and Loss-only across boundary strategies; on Charades-STA, Long Tail + IoU+LossMax improves [email protected] from 35.81% to 51.23% (Kim et al., 29 Mar 2025).

Efficiency claims are likewise bound to decomposition. UI-AGILE reports that with a VLM whose self-attention complexity is $S(\phi_k)$ 4 on image token length $S(\phi_k)$ 5, decomposing into $S(\phi_k)$ 6 tiles reduces the dominant prefilling cost to approximately $S(\phi_k)$ 7; with $S(\phi_k)$ 8, this is approximately $S(\phi_k)$ 9 (Lian et al., 29 Jul 2025). FocusUI shows that retaining only 30% of visual tokens can preserve near-baseline accuracy because PosPad maintains positional continuity, a property that general visual token pruning methods often break (Ouyang et al., 7 Jan 2026). In the Toffoli setting, the gains are hardware-relevant rather than perceptual: the selector minimizes a two-qubit-infidelity budget and certifies every committed substitution beyond the exhaustive-verification limit with zero certified errors (Bartkiewicz et al., 30 Jun 2026).

6. Limitations, failure modes, and broader interpretations

The literature also places clear bounds on when decomposition and selection help. In medical abnormality grounding, the method depends on high-quality definitions and LLM rewriting; the compact attribute set of shape, location, density, and color may not capture modality-specific nuances such as distribution patterns, temporal change, or comparative signs, and attention-only selection may be suboptimal in ambiguous cases (Li et al., 5 Mar 2025). In weakly supervised video grounding, the framework relies on Gaussian mixture parameterizations being well learned during training, and the best selection rule is dataset-dependent: IoU+LossMax performs best on Charades-STA, whereas IoU+LossSum performs best on ActivityNet Captions (Kim et al., 29 Mar 2025).

GUI decompositions can fail through overcropping or context loss. UI-AGILE notes that very large elements or split visuals may still yield partial crops with lower “Yes” scores, and that ambiguous instructions or multiple similar elements can make selection indecisive (Lian et al., 29 Jul 2025). FocusUI reports that aggressive retention may remove necessary background or relational context, especially for instructions such as “the button under the second chart” (Ouyang et al., 7 Jan 2026). BAMI similarly warns that if the crop ratio $\rightarrow$ 0 is too small, cropping may remove needed context, and if the number of focus iterations is too large, cumulative cropping can degrade performance (Zhang et al., 7 May 2026).

Logical and algebraic decompositions introduce a different kind of limitation: soundness. The STL planner’s DNF conversion is explicitly a strengthening, not an equivalence-preserving rewrite; $\rightarrow$ 1, but not conversely, so some behaviors that satisfy the original formula may be excluded (Ye et al., 2 May 2026). In the Toffoli compiler, the central safety claim is that pattern-matched relative-phase substitution is silently incorrect: the verifier flagged 66 library rewrites as non-equivalent without a context check, and count-greedy substitution silently corrupted 6 of 12 benchmark circuits (Bartkiewicz et al., 30 Jun 2026). These cases show that decomposition can enlarge the admissible candidate set only when a sufficiently strong selection criterion rejects invalid candidates.

A broader theoretical interpretation appears in the philosophical framework “On measuring grounding and generalizing grounding problems,” which decomposes grounding itself into authenticity, preservation, faithfulness, robustness, and compositionality, all indexed by an evaluation tuple $\rightarrow$ 2. In that framework, “selection-for-success” is formalized etiologically through an average causal effect criterion,

$\rightarrow$ 3

This suggests that algorithmic Decomposed Grounding with Selection can be understood not only as an engineering pattern for localization and decision-making, but also as part of a larger program that asks whether internal mechanisms are causally responsible for successful grounding claims (Quigley et al., 5 Dec 2025).

Across these strands, the most stable conclusion is technical rather than terminological. Decomposition is useful when the original grounding target is too abstract, too high-dimensional, too ambiguous, or too context-dependent to be handled reliably in one step. Selection is useful when the decomposed representation introduces multiple plausible candidates whose relevance is instance-specific. The cited works differ sharply in modality and formalism, but they converge on that joint logic.