---
title: 'PosPad: Visual Token Compression for Efficient UI Grounding'
url: https://www.emergentmind.com/topics/pospad
type: topic
---

# PosPad: Visual Token Compression for Efficient UI Grounding

Searching arXiv for the cited papers to ground the article.
PosPad is a **position-preserving visual token compression strategy** introduced in the UI-grounding framework FocusUI. It is designed for the setting in which a high-resolution screenshot is patchified into a long visual-token sequence and then aggressively reduced before multimodal decoding. The central problem is that direct visual token pruning shortens the sequence but creates **positional jumps in the \((h,w)\) dimensions of M-RoPE**, which is especially harmful for fine-grained UI grounding. PosPad addresses this by compressing each contiguous sequence of dropped visual tokens into a single learnable marker, written as `<pos>`, placed at the sequence’s **last index** so that the reduced sequence preserves positional continuity while still achieving substantial length reduction [2601.03928].

## 1. Problem formulation and motivation

FocusUI studies **efficient UI grounding**: given a high-resolution screenshot \(I\) and a natural-language instruction \(q\), the model must identify the target UI region while using fewer visual tokens than a dense VLM baseline. This setting is unusually stringent because screenshots are tokenized into thousands of visual tokens; for 2K screenshots the paper notes roughly **4700 visual tokens**, and across its benchmark statistics visual tokens account for at least **84.3%–98.1%** of the entire input sequence, depending on model and benchmark [2601.03928].

The resulting bottlenecks are twofold. First, long visual sequences create substantial computation and memory overhead. Second, attention is diluted because many patches correspond to redundant or homogeneous regions. A natural response is to prune visual tokens, but UI grounding differs from coarse visual understanding tasks because it is a **fine-grained localization task**: small spatial offsets can cause the model to click the wrong button, icon, or text field. In Qwen-style VLMs, visual patches are processed with **multimodal rotary position embeddings (M-RoPE)** over temporal, height, and width dimensions. When tokens are simply removed, the sequence collapses across omitted regions, and neighboring kept patches become adjacent in the sequence even when they were far apart in the original screenshot. The paper attributes large localization offsets to exactly this mechanism, describing direct pruning as creating **positional jumps** in the \((h,w)\) dimensions of the M-RoPE sequence [2601.03928].

PosPad is the mechanism introduced to preserve enough spatial structure after pruning that precise grounding remains possible. It does not determine which tokens are relevant; rather, it preserves the **ordered spatial progression** of screenshot patches once many of them have been removed. This design targets the UI-specific failure mode in which sequence shortening destroys the layout scaffold needed for precise localization.

## 2. Formal definition of the PosPad transformation

Let the original visual token sequence be \(x_{1:M}\), where \(M\) is the number of screenshot patch tokens. FocusUI first computes per-patch saliency scores and retains the top-\(K\) patches under a retention ratio \(r \in (0,1]\), with
\[
K = \lfloor rM \rfloor.
\]
If \(\mathcal{K}\) denotes the kept index set and \(\mathcal{D}\) the dropped index set, the selection rule is defined from the per-patch saliency scores \(s_i\) by thresholding at the \(K\)-th largest score [2601.03928].

PosPad then partitions the dropped set \(\mathcal{D}\) into **maximal contiguous subsequences**
\[
\mathcal{D} = \bigsqcup_{u=1}^{U} \mathcal{R}_u,
\]
where each \(\mathcal{R}_u\) is consecutive in flattened 1D patch order. For each dropped span \(\mathcal{R}_u\), PosPad retains only its **last index**
\[
r_u^{\mathrm{end}} = \max \mathcal{R}_u,
\]
and forms the set of sequence-end indices
\[
\mathcal{E}_{\text{seq-end}} = \{r_u^{\mathrm{end}}\}_{u=1}^{U}.
\]
The preserved index set is then
\[
\mathcal{S} = \mathcal{K} \cup \mathcal{E}_{\text{seq-end}}.
\]

The transformed sequence is
\[
x'_j =
\begin{cases}
<pos> & \text{if } j \in \mathcal{E}_{\text{seq-end}},\\
x_j & \text{if } j \in \mathcal{K},
\end{cases}
\qquad
\mathrm{PosPad}(x_{1:M}) = \{x'_j\}_{j \in \mathcal{S}}.
\]

Each contiguous dropped span is therefore compressed into a single special learnable token, written as `<pos>`, and that marker is placed **at the last index of the dropped span**. All other indices in that span are removed, while kept visual tokens remain unchanged. If a total of \(|\mathcal{D}|\) tokens are dropped and these drops form \(U\) contiguous spans, the final visual length is
\[
M' = M - (|\mathcal{D}| - U).
\]
The number of `<pos>` markers is exactly \(U\) [2601.03928].

The design choice of using the **last index** is the central positional-preservation mechanism. The marker inherits that index’s \((h,w)\) positional information, which preserves continuity more faithfully than removing the entire span or inserting a marker earlier in the span. The paper explicitly states that PosPad alters only **sequence sparsity** and not token indices or rotary bases, so it is compatible with common M-RoPE implementations and requires no modifications to the downstream LM architecture [2601.03928].

## 3. Integration within the FocusUI pipeline

PosPad sits between token selection and multimodal decoding. FocusUI has three major components: **Instruction-to-Patch supervision**, a **Query-Guided Saliency Scorer**, and **PosPad**. The scorer, not PosPad, decides what to keep. It operates over visual patch embeddings \(\{v_i\}_{i=1}^{M}\) and instruction token embeddings \(\{e_j\}_{j=1}^{N}\), computing
\[
\mathcal{P} = \tilde V \tilde E^\top \in \mathbb{R}^{M \times N},
\qquad
s_i = \frac{1}{N}\sum_{j=1}^{N}\mathcal{P}_{i,j}.
\]
These scores drive top-\(K\) selection under the specified retention ratio [2601.03928].

The supervision for the scorer combines instruction-conditioned localization and a rule-based UI prior. The fused supervision is
\[
S_{\mathrm{Ins2Patch}}
=
\lambda S_{\mathrm{bbox}} + (1-\lambda)S_{\mathrm{uig}}.
\]
Here the UI-graph term down-weights large homogeneous regions, while the instruction-conditioned term identifies patches relevant to the task. PosPad is applied **after visual encoding** and **after saliency scoring**, but **before** the visual token sequence is consumed by the downstream LM decoder. In this sense, the scorer answers **which** patches to keep, whereas PosPad determines **how** to represent what was dropped without collapsing the spatial layout [2601.03928].

FocusUI’s downstream grounding component follows the **coordinate-free grounding scheme from GUI-Actor**. The LM outputs an action token representation \(h_{\text{ACTOR}}\), and an action head attends over selected patch features. PosPad is not a grounding head by itself; its contribution is to ensure that the reduced visual sequence still retains enough layout structure for the grounding head to align the instruction-conditioned action representation with candidate UI regions.

This separation of roles is conceptually important. PosPad is neither a saliency estimator nor a positional-embedding rewrite. It is a **sequence-structure transformation** over already selected visual tokens.

## 4. Positional continuity, ablations, and empirical behavior

The clearest empirical finding is that **continuity preservation is crucial**. FocusUI compares four sequence constructions on ScreenSpot-Pro at \(50\%\) retention: the original visual sequence, direct dropping, full padding, and PosPad. The ablation isolates the difference between reducing sequence length and preserving positional continuity [2601.03928].

| Strategy | ScreenSpot-Pro accuracy | Effect |
|---|---:|---|
| Baseline \(100\%\) | 40.9 | No pruning |
| CLIP score + direct drop | 28.5 | Continuity broken |
| CLIP score + full padding | 38.7 | Continuity preserved, no length reduction |
| CLIP score + PosPad | 38.2 | Continuity preserved, compressed |
| Ins2Patch score + direct drop | 29.2 | Continuity broken |
| Ins2Patch score + full padding | 42.1 | Continuity preserved, no length reduction |
| Ins2Patch score + PosPad | 42.3 | Continuity preserved, compressed |

These results show that most of the performance collapse under pruning comes from **breaking positional continuity**, not merely from discarding content. Direct drop is described as catastrophic, whereas continuity-preserving schemes recover most of the lost accuracy. PosPad is slightly better than full padding in the Ins2Patch setting while still reducing sequence length.

The paper also studies marker placement. The appendix compares three variants: **sequence-first**, **sequence-middle**, and **sequence-end**. Sequence-end is best at all tested retention ratios. On ScreenSpot-Pro, at \(r=50\%\), the scores are **39.8** for first, **38.7** for middle, and **40.4** for end; at \(r=25\%\), they are **36.8**, **33.5**, and **37.7**, respectively. This supports the stated interpretation that end placement is most compatible with raster-scan ordering and M-RoPE, whereas earlier placement pulls the omitted region toward earlier positions in the flattened sequence [2601.03928].

At system level, PosPad supports aggressive pruning with modest degradation. For **FocusUI-7B** on ScreenSpot-Pro, accuracy changes from **48.3** at \(r=100\%\) to **46.5** at \(r=50\%\) and **45.1** at \(r=30\%\). In the same comparison, reducing to \(30\%\) retention yields **1.44× faster inference** and about **17% lower peak GPU memory**. The efficiency table notes that the reported number of visual tokens does **not** include `<pos>` tokens, so the actual multimodal sequence is slightly longer than the raw kept-token count, but the end-to-end reduction remains substantial [2601.03928].

Against generic VLM pruning baselines, the gap is larger. At \(30\%\) retention on ScreenSpot-Pro, generic pruning methods such as **Fast-V**, **HiPrune**, and **Vision-Zip** suffer severe drops, while FocusUI with PosPad remains much closer to its dense baseline. The broader implication is that PosPad’s benefit is strongest in **precise localization-heavy UI grounding**, especially under **aggressive token retention ratios** and on **high-resolution screenshots** [2601.03928].

## 5. Implementation characteristics and operational constraints

PosPad introduces a **learnable special token** `<pos>`. The paper specifies that it is inserted only at **sequence-end indices** of dropped spans. It does **not** provide an explicit initialization formula for the `<pos>` embedding, a PosPad-specific attention mask, or any modified rotary-position formula beyond inheriting the original index [2601.03928].

Several implementation details delimit the method’s scope. Dropped spans are defined with respect to the **1D flattened sequence order**, and the appendix ties the sequence-end design to the **raster-scan ordering** used by the vision encoder and M-RoPE. If pruning produces several disjoint contiguous regions, each region receives its own `<pos>` token. PosPad therefore preserves continuity at the level of contiguous omissions rather than at the level of every dropped patch.

FocusUI is trained end-to-end with PosPad present in the loop. The retention ratio \(r\) is sampled uniformly from \((0.1,1.0)\) during training, which exposes the model to varying sparsity patterns. For reproduction, the paper reports **patch size \(p=14\)** for Qwen2.5-VL and **\(p=16\)** for Qwen3-VL, with **\(\lambda = 0.8\)** and **\(\tau = 2\)** in supervision construction. The framework is also described as natively compatible with **FlashAttention-2**, precisely because PosPad changes only the input sequence structure and does not depend on intermediate attention or activation information [2601.03928].

The paper does **not** present PosPad as a purely inference-only plug-in and does **not** test a “PosPad only at inference without retraining” setting. It also does not compare PosPad against interpolation, learned pooling over dropped spans, or explicit index-remapping schemes. Those alternatives therefore remain outside the reported evidence base.

A further operational limitation is that PosPad preserves continuity only **coarsely**. A whole dropped run is summarized by one token. Very aggressive pruning still hurts: at \(25\%\) and especially \(10\%\) retention, accuracy declines noticeably. PosPad thus preserves the layout scaffold more effectively than direct dropping, but it does not reconstruct the omitted fine-grained content [2601.03928].

## 6. Conceptual scope, misconceptions, and related methods

PosPad is best understood as a **sequence-level compression mechanism for preserving positional continuity after visual token pruning**. Several nearby interpretations are inaccurate. It is **not** the token-selection policy itself; selection is handled by the saliency scorer. It is **not** ordinary padding in the NLP sense; the inserted `<pos>` markers summarize omitted contiguous regions and remain tied to inherited spatial indices. It is also **not** a redesign of positional encoding, since FocusUI explicitly leaves token indices and rotary bases unchanged [2601.03928].

A frequent source of confusion is nomenclature. The paper **“Pose-Aware Diffusion for 3D Generation”** uses the exact term **PAD**, not PosPad, for a 3D generation framework that synthesizes geometry directly in observation space using a monocular-depth-derived partial point cloud as a geometric anchor [2605.00345]. That method is pose-aware 3D generation rather than position-preserving token compression, and the overlap is only lexical.

A more substantive conceptual parallel appears in **“Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models”**, which does **not** mention PosPad explicitly but is described as a close conceptual match **if** PosPad is taken to mean adding extra intra-patch positions or latent updates to reduce an information bottleneck inside coarse patches. In that setting, **Scratchpad Patching** inserts **transient internal states inside patches** to refresh patch-level context without increasing the persistent patch sequence length or the inference-time KV-cache footprint. The paper emphasizes, however, that these added positions are **semantic transient compute states**, not dummy padding tokens; it is therefore more accurately a **patch-local latent update mechanism** than a positional padding mechanism [2605.09630].

Within the UI-grounding context, the more precise characterization is narrower and more concrete. PosPad is a method for **compressing dropped contiguous visual-token spans into single learned positional markers placed at sequence ends**, thereby preserving coarse spatial continuity in a reduced multimodal sequence. Its significance lies in showing that efficient screenshot grounding depends not only on selecting the right tokens, but also on representing **where discarded regions used to be**.

Source: https://www.emergentmind.com/topics/pospad