IoU-Augmented Maximum Likelihood for GUI Grounding

• The paper introduces IAML as an IoU-weighted reward-augmented training paradigm for robust and geometry-aware GUI coordinate prediction.
• IAML augments standard MLE by sampling near-correct bounding boxes, which reduces exposure bias and improves performance in low-data regimes.
• Empirical results demonstrate significant F1 and click accuracy gains over traditional MLE on tasks such as Action Space Generation and Single Element Grounding.

{"query":"arXiv (Xu et al., 22 Aug 2025) Structuring GUI Elements through Vision LLMs: Towards Action Space Generation IoU-Augmented Maximum Likelihood RAML Norouzi 2016 reward-augmented maximum likelihood"} IoU-Augmented Maximum Likelihood (IAML) is a training paradigm for multimodal LLMs (MLLMs) applied to GUI grounding and action space generation. It is introduced in the context of generating UI element coordinates from screen contents and user instructions, where precise coordinate prediction is hindered by the fact that off-the-shelf multimodal LLMs are trained as “next-token predictors” over a purely linguistic space. In this setting, numerical bounding-box coordinates $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$ carry almost no semantic meaning in the LLM’s vocabulary, and standard Maximum-Likelihood Estimation (MLE) penalizes every incorrect prediction equally, even when it is almost correct. IAML addresses this by augmenting training with nearby bounding-box variations sampled according to Intersection-over-Union (IoU) and by folding IoU into a reward-augmented MLE framework inspired by RAML, thereby producing a smoother, geometry-aware training signal for coordinate generation (Xu et al., 22 Aug 2025).

1. Definition and motivation

In GUI-grounding tasks, one of the central difficulties is that coordinate prediction is cast into token generation, while the training objective remains the standard next-token objective. The baseline fine-tuning formulation maximizes

$p(y|x)=\prod_i p(y_i|x,y_{<i})$

through the cross-entropy loss

$L_{MLE}(D)=-\sum_{(x,y)\in D}\log p(y|x).$

Within this formulation, the model must rely solely on paired data to align language-based generation with pixel-level grounding, because the coordinate tokens themselves do not encode useful semantic structure. The paper further identifies teacher-forcing in MLE as a source of exposure bias: during training the model always conditions on ground-truth prefixes, but at test time it must condition on its own possibly erroneous outputs, which makes precise number generation brittle (Xu et al., 22 Aug 2025).

IAML is defined as an augmentation of the training distribution with nearby coordinate hypotheses whose weight depends on their IoU with the ground truth. This reshaping has three stated effects: it tolerates small coordinate deviations, increases gradient signal for high-IoU hypotheses, and expands exploration. The intended consequence is partial mitigation of the exposure bias inherent in a pure next-token objective.

2. Mathematical formulation

The formulation follows the Reward-Augmented MLE philosophy, but replaces text-oriented rewards such as BLEU or edit distance with the continuous IoU metric. Let $y$ denote the ground-truth coordinate sequence and $\tilde y$ a perturbed hypothesis. The reward is defined as

$r(y,\tilde y)=IoU(bbox(y),bbox(\tilde y)).$

An exponentiated-payoff distribution over hypotheses is then introduced:

$$q(\tilde y|y;\tau)= \frac{\exp(r(y,\tilde y)/\tau)} {\sum_{\tilde y'}\exp(r(y,\tilde y')/\tau)},$$

where $\tau>0$ is a temperature smoothing parameter.

The IAML objective is the cross-entropy between this IoU-weighted target distribution and the model distribution:

$$L_{IAML}(D)= -\sum_{(x,y)\in D}\sum_{\tilde y}q(\tilde y|y;\tau)\,\log p(\tilde y|x).$$

Because $q$ places more mass on samples with higher IoU, maximizing $p(y|x)=\prod_i p(y_i|x,y_{<i})$0 under $p(y|x)=\prod_i p(y_i|x,y_{<i})$1 encourages the model to produce bounding boxes closer to the ground truth while still exploring slight perturbations. The paper explicitly characterizes this as a smoother, more vision-aware training signal than exact-match MLE (Xu et al., 22 Aug 2025).

3. IoU-based coordinate sampling pipeline

The data smoothing used by IAML is realized through a Monte-Carlo sampler that perturbs ground-truth coordinates, computes IoU, bins perturbations by discretized IoU, and samples according to a softmax over bin statistics plus an IoU penalty. The stated algorithm takes as input $p(y|x)=\prod_i p(y_i|x,y_{<i})$2, temperature $p(y|x)=\prod_i p(y_i|x,y_{<i})$3, perturbation range $p(y|x)=\prod_i p(y_i|x,y_{<i})$4, and sample count $p(y|x)=\prod_i p(y_i|x,y_{<i})$5, and returns one augmented $p(y|x)=\prod_i p(y_i|x,y_{<i})$6.

The procedure is:

1. Initialize an empty map $p(y|x)=\prod_i p(y_i|x,y_{<i})$7.
2. For $p(y|x)=\prod_i p(y_i|x,y_{<i})$8:
   • draw $p(y|x)=\prod_i p(y_i|x,y_{<i})$9,
   • set $L_{MLE}(D)=-\sum_{(x,y)\in D}\log p(y|x).$0,
   • compute $L_{MLE}(D)=-\sum_{(x,y)\in D}\log p(y|x).$1,
   • discretize the index as $L_{MLE}(D)=-\sum_{(x,y)\in D}\log p(y|x).$2,
   • append $L_{MLE}(D)=-\sum_{(x,y)\in D}\log p(y|x).$3 to $L_{MLE}(D)=-\sum_{(x,y)\in D}\log p(y|x).$4.
3. For each reward bin $L_{MLE}(D)=-\sum_{(x,y)\in D}\log p(y|x).$5 in ascending order, compute

$L_{MLE}(D)=-\sum_{(x,y)\in D}\log p(y|x).$6

1. Normalize over bins with $L_{MLE}(D)=-\sum_{(x,y)\in D}\log p(y|x).$7.
2. Draw bin index $L_{MLE}(D)=-\sum_{(x,y)\in D}\log p(y|x).$8.
3. Return a uniformly random bounding box from $L_{MLE}(D)=-\sum_{(x,y)\in D}\log p(y|x).$9.

Several details in this construction are central. Continuous IoU in $y$0 is mapped to an integer reward index $y$1, so higher IoU corresponds to smaller $y$2. The term $y$3 favors bins with higher IoU, while $y$4 accounts for the number of candidates in a bin and is described as stabilizing variance. Softmax over bins yields an approximate exponentiated payoff distribution, and uniform sampling within a chosen bin preserves diversity among equally rewarded candidates (Xu et al., 22 Aug 2025).

4. Exposure bias and target-distribution smoothing

The paper’s rationale for IAML is organized around exposure bias in coordinate generation. Under standard next-token MLE, any deviation from the exact coordinate tokens receives an all-or-nothing penalty, and the decoder is never trained on the near-correct prefixes that arise at inference time. In GUI grounding, this is especially problematic because numerical coordinate tokens do not benefit from the semantic regularities that exist for natural-language tokens.

IAML softens the target distribution by replacing the one-hot ground truth with a distribution $y$5 that assigns non-zero mass to a neighborhood of almost correct outputs. The cross-entropy loss under $y$6 is stated to be equivalent, up to an additive entropy constant, to minimizing $y$7. Because $y$8 includes the ground truth together with nearby high-IoU variants, the model is trained to assign probability mass to those variants as well. The paper characterizes this as effectively training the model to self-correct small errors during generation, thereby alleviating exposure bias by exposing the decoder during training to the kinds of near-correct prefixes it will encounter at inference time (Xu et al., 22 Aug 2025).

This suggests that IAML should be understood not as abandoning MLE, but as reshaping the target distribution inside an MLE-compatible training loop. A plausible implication is that its appeal lies in modifying supervision while retaining familiar optimization machinery.

5. Empirical results

The reported experiments cover two benchmarks and compare IAML against vanilla MLE and random uniform noising. The paper states that IAML consistently outperforms both alternatives.

Task and setting	Baseline	Reported IAML change
Action Space Generation, full train set	MLE	F$y$9 gain of 1.03→2.14 pp absolute
Action Space Generation, 10% data	MLE	Relative F$\tilde y$0 gains of 3.18→6.44%
ScreenSpot, SeeClick	SeeClick-MLE 52.7% avg.	55.0% (+2.3 pp, ≈4.3% rel)
ScreenSpot, OS-Atlas	OS-Atlas-MLE 49.4% avg.	59.8% (+10.4 pp, ≈21.1% rel)

For Action Space Generation on the ScreenAnnotation dataset, evaluated at IoU thresholds $\tilde y$1, IAML raises $\tilde y$2 by 1.03→2.14 percentage points absolute over MLE on the full train set. The example given at IoU $\tilde y$3 is $\tilde y$4, corresponding to $\tilde y$5 relative. In a low-resource setting using 10% of the data, the relative $\tilde y$6 gains increase to 3.18→6.44% across thresholds, with the example at IoU $\tilde y$7 reported as $\tilde y$8.

The paper also reports similar improvements when fine-tuning Qwen2-VL and OS-Atlas, with best $\tilde y$9 gains up to $r(y,\tilde y)=IoU(bbox(y),bbox(\tilde y)).$0 relative at IoU $r(y,\tilde y)=IoU(bbox(y),bbox(\tilde y)).$1 on Qwen2 and $r(y,\tilde y)=IoU(bbox(y),bbox(\tilde y)).$2 relative on OS-Atlas. For Single Element Grounding on the ScreenSpot dataset, click accuracy improves from 52.7% average to 55.0% for SeeClick and from 49.4% average to 59.8% for OS-Atlas (Xu et al., 22 Aug 2025).

6. Role in GUI element structuring and action space generation

The broader setting of IAML is GUI elements structuring through vision-LLMs, specifically for processing user instructions based on screen contents. The paper frames coordinate generation as a critical aspect of GUI understanding and ties IAML directly to action space generation, where the output must correspond to valid and precise UI element locations.

Within that framing, IAML serves as a bridge between linguistic sequence modeling and geometric localization. It uses IoU as a high-resolution, geometry-aware reward and embeds that reward inside a reward-augmented MLE framework. The reported empirical gains are presented as evidence that IoU-aware data smoothing sharpens spatial precision and improves robustness in low-data regimes, while retaining the simplicity of an MLE-based training loop (Xu et al., 22 Aug 2025).

A plausible implication is that IAML is most relevant in settings where outputs are serialized as tokens but evaluated geometrically. In such settings, exact-token supervision and geometric correctness are not identical objectives; IAML directly targets that mismatch by making near-correct coordinate sequences trainable rather than uniformly penalized.

7. Conceptual boundaries and interpretation

IAML is characterized in the paper as “simple, yet effective,” but its conceptual contribution is specific rather than universal. It does not redefine GUI grounding as a fundamentally different task; instead, it alters the supervision signal for coordinate generation. The core change is the substitution of an IoU-weighted hypothesis distribution for the one-hot target distribution used in standard MLE.

This distinction matters for interpretation. The method is not described as replacing cross-entropy, replacing autoregressive decoding, or introducing a separate reinforcement-learning phase. Rather, it augments training data with IoU-based coordinate sampling and trains with a cross-entropy objective defined over the resulting IoU-weighted target distribution. The paper’s stated significance lies in reducing the brittleness of exact coordinate prediction under next-token training, particularly when numerical tokens have little semantic support in the underlying language representation space (Xu et al., 22 Aug 2025).