Agent-ScanKit: Probing Multimodal GUI Agents

• Agent-ScanKit is a diagnostic framework that probes multimodal GUI agents to differentiate robust reasoning from memorized behavior via controlled perturbations.
• It employs three probing paradigms—visual-guided, text-guided, and structure-guided—to evaluate agent sensitivity across diverse GUI benchmarks.
• Empirical findings reveal agents often rely on memorization and short-cutting, suggesting the need for improved reasoning supervision and better generalization.

Searching arXiv for the Agent-ScanKit paper and closely related agent-analysis work. Agent-ScanKit is a systematic probing framework for multimodal GUI agents that is designed to test whether apparent competence arises from robust reasoning or from memorized, training-aligned shortcuts. It operates by applying controlled perturbations to the agent’s visual input, textual instruction, or action structure, and then measuring how the agent’s behavior changes without requiring access to model internals. In the formulation introduced for multimodal LLM agents, the framework is used to analyze 18 open-source GUI agents across five publicly available benchmarks, with the reported result that mechanical memorization often outweighs systematic reasoning and that most models function predominantly as retrievers of training-aligned knowledge with limited generalization (Cheng et al., 1 Oct 2025).

1. Problem setting and conceptual objective

Agent-ScanKit is motivated by a recurrent failure mode in multimodal GUI agents: strong performance on benchmark tasks that are close to the training distribution, followed by sharp degradation on long-horizon, out-of-domain, or cross-platform tasks. The framework treats this discrepancy as evidence that benchmark success may reflect memory-driven spurious reasoning rather than robust inference. In this view, the central question is not merely whether an agent predicts the correct next GUI action, but whether it does so by integrating the available evidence or by relying on spatial priors, instruction templates, and action shortcuts (Cheng et al., 1 Oct 2025).

The paper frames GUI interaction as a partially observable decision problem. An agent observes a screen, a task goal, and sometimes a chain-of-thought trace, then predicts the next action. Two structural difficulties are emphasized. The first is infinite predictive space: GUI actions lie in an open-ended coordinate and vocabulary space, so the agent must choose both an action type and precise coordinates or text strings. The second is finite generalization: even agents with strong in-distribution action prediction often fail as task horizons lengthen or interface conditions shift. Agent-ScanKit is intended as a diagnostic response to that mismatch.

A central interpretive claim of the framework is that high benchmark accuracy does not by itself establish reasoning. If performance remains stable after relevant evidence is corrupted, that stability can indicate shortcut exploitation rather than robustness. Conversely, sharp degradation under perturbation can reveal genuine dependence on the altered signal. The framework therefore studies sensitivity rather than latent mechanism directly.

2. Formalization and perturbation sensitivity

Agent-ScanKit extends the original GUI decision process into a perturbed process

$$\mathcal{M}_p = (G, \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \mathcal{H}, \mathcal{P}),$$

where $\mathcal{P}$ is a perturbation operator on states. Given a goal $g \in G$, the agent observes a perturbed state $s'_t=\mathcal{P}(s_t)$ and acts according to

$a_t \sim \pi(a_t|s'_t,g).$

The core diagnostic quantity is perturbation sensitivity:

$$\Delta_P = \mathbb{E}_{(g,s_t)}[Acc(\pi(s_t, g)) - Acc(\pi(\mathcal{P}(s_t), g))].$$

This quantity compares original and perturbed performance. Larger drops indicate stronger reliance on the perturbed signal. Smaller drops may indicate robustness, but the paper stresses that they can also indicate shortcut behavior depending on the perturbation type and the metric being used (Cheng et al., 1 Oct 2025).

This formalization is deliberately behavioral. Agent-ScanKit does not attempt to recover internal causal structure from attention weights or latent activations. Instead, it infers memory-versus-reasoning tendencies from the way observable action distributions react to controlled changes in evidence. That choice makes the framework model-agnostic and compatible with public agents whose internals are inaccessible.

3. The three probing paradigms

Agent-ScanKit organizes perturbations into three orthogonal paradigms: visual-guided, text-guided, and structure-guided probing. Each targets a different candidate shortcut pathway.

Visual-guided probing tests whether the agent depends on real visual evidence, memorized spatial priors, or local appearance cues. In object masking, the target element $e^*$ is hidden by blacking out its region. If the agent still predicts the same action or nearly the same coordinates, that suggests reliance on memorized location priors rather than current perception. In object editing, the target region is reconstructed by interpolation, which is treated as a stronger corruption than masking. In zoom-in, the screen is cropped to the quadrant containing the target, preserving local detail while removing global layout cues. The paper reports a concrete probing implementation using a 50-pixel black block for masking, 50-pixel image editing for object editing, and quadrant-based cropping for zoom-in (Cheng et al., 1 Oct 2025).

Text-guided probing tests whether the agent reasons over linguistic instructions or keys off familiar instruction patterns. In token-level perturbation, the initial words of the atomic instruction are removed, with the implementation replacing the first word by []. In sentence-level perturbation, the original instruction is replaced by a semantically wrong one; the reported injected wrong instruction is “Click the Amazon APP”. These probes are aimed especially at text-dependent actions such as Type and OpenApp. If the model cannot recover the intended action when early lexical cues disappear, the framework interprets that as brittle dependence on memorized textual triggers rather than robust semantic inference.

Structure-guided probing examines whether reflective or status actions are genuine reasoning outputs or learned shortcuts. The paper distinguishes visual shortcuts, where the model can decide using only the current screen, from action shortcuts, where the model can decide using only the instruction text. This probe focuses particularly on reflective and status actions such as PressBack, PressHome, Wait, and Complete. Visual and textual modalities are corrupted independently to identify where shortcut dependence originates (Cheng et al., 1 Oct 2025).

4. Metrics, benchmarks, and evaluation protocol

The framework uses standard GUI-agent evaluation metrics together with probe-specific diagnostics. The standard metrics are Type for exact action-type match, Grounding for spatial grounding correctness, SR for step-wise success rate, and TSR for task success rate. Probe-specific reporting frequently uses $\Delta P_{\text{Type}}$ and $\Delta P_{\text{SR}}$, capturing changes in action-type accuracy and step success under perturbation (Cheng et al., 1 Oct 2025).

For visual-guided probing, the paper introduces Visual Memory Consistency (VMC):

$$\text{VMC} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I} \left( \left\| \mathbf{p}_i^C - \mathbf{p}_i^O \right\|_2 \leq \gamma \right),$$

where $\mathcal{P}$0 and $\mathcal{P}$1 are the predicted coordinates under perturbed and original conditions, respectively, and $\mathcal{P}$2 is a distance threshold. High VMC indicates that the agent predicts almost the same location even after corruption, which the paper treats as evidence of memory bias. A second probe-specific metric, Reflection Score (RS), measures whether the agent correctly switches to reflection or status behavior under masking or editing; low RS indicates failure to reflect and continued reliance on old memory.

The action-correctness rules are explicitly specified. Click coordinates are normalized to 1000 and counted as correct if they fall within 14% relative distance. Scroll requires exact direction match. Type and OpenApp require exact text match. Other actions, including PressBack, require exact match.

The experimental study covers 18 open-source multimodal GUI agents from 8 institutions, including families such as OS-Atlas, OS-Genesis, UI-TARS, Aguvis, OdysseyAgent, GUI-R1, Mobile-Agent, AgentCPM-GUI, and GUI-Owl. The main text describes five public benchmarks spanning mobile, web, and desktop settings: AndroidControl, AITZ, GUI-Odyssey, GUI-Act-Mobile, and GUI-Act-Web. The appendix additionally discusses OmniAct-Web and OmniAct-Desktop. Unless otherwise noted, the probing evaluation uses low-level settings with samples that have 100% step-wise accuracy, so the analysis begins from trajectories where the unperturbed agent already succeeds (Cheng et al., 1 Oct 2025).

5. Empirical findings on memory, reasoning, and shortcut formation

The main empirical conclusion is that mechanical memorization dominates over systematic reasoning in most current GUI agents. Under visual-guided probing, many models remain strongly attached to original target coordinates even after masking or editing, often yielding high VMC and low RS. The reported interpretation is that these agents rely on strong spatial priors such as recurring interface layouts rather than on current visual evidence. Early SFT models are described as especially memory-driven. Larger or RL-based agents sometimes become more reflective, but they also exhibit over-reflection when spatial cues are disrupted (Cheng et al., 1 Oct 2025).

The text-guided results indicate that agents often cannot recover the intended instruction when early lexical cues are removed, and they remain brittle when the instruction is replaced by a semantically wrong one. A notable claim is that no agent can reliably infer input content from instructions lacking action-start words. The paper further notes that low perturbation sensitivity in some models can itself reflect overdependence on visual memory rather than strong textual inference, because the agent may continue acting from remembered interface priors instead of interpreting the incomplete instruction.

Structure-guided probing shows that reflective and status actions are frequently learned shortcuts rather than deliberate control decisions. Press and Wait often exhibit strong shortcut behavior. Early SFT models tend to rely more on visual shortcuts, whereas later UI-TARS and RL-based models can shift toward action shortcuts in some settings. Scroll is presented as a more mixed case that sometimes reflects more joint visual-semantic decision-making. GUI-Owl is singled out as showing a more multimodal reasoning pattern, plausibly because of redesigned CoT and RL, although even there the behavior is not fully robust.

Across all three paradigms, the framework reports a consistent pattern: many agents succeed through brittle associations rather than generalizable reasoning. The paper summarizes these systems not as agents that merely reason badly, but as systems that often are not reasoning in the intended sense at all. They behave more like retrievers or pattern matchers over training-aligned behaviors than like robust multimodal planners (Cheng et al., 1 Oct 2025).

6. Interpretation, limitations, and place in the broader agent-analysis literature

Agent-ScanKit is diagnostic rather than mechanistic. The paper is explicit that it measures sensitivity, not internal causal mechanisms. Perturbation-based interpretation can remain ambiguous: a small performance drop may indicate robustness, but it may also indicate shortcut dependence. The conclusions are also bounded by the selected benchmarks, perturbation operators, and shared action space. Claims about RL and CoT are similarly qualified: both can help, but neither eliminates spurious reasoning, and in some cases they induce over-reflection. The findings are therefore best read as a structured behavioral analysis of publicly available GUI agents rather than as a complete theory of multimodal reasoning (Cheng et al., 1 Oct 2025).

A recurring misconception is suggested by the framework’s name: Agent-ScanKit is not a security scanner. Its purpose is to probe memory and reasoning in multimodal GUI control. Contemporary work with superficially similar “scan” terminology addresses different objects altogether: Agent Audit focuses on static security analysis of Python agent code, deployment artifacts, prompt construction, credentials, and MCP configurations (Zhang et al., 24 Mar 2026); large-scale skill studies analyze vulnerable third-party agent skills through static and semantic detection pipelines (Liu et al., 15 Jan 2026); multimodal skill-scanning research studies hidden visual instructions and execution-grounded skill auditing (Jia et al., 16 Jun 2026); runtime malware work emphasizes behavior-centric detection resilient to scanner evasion (Ji et al., 2 Jul 2026); and marketplace detectors localize malicious skill spans using attention-based top-$\mathcal{P}$3 retention before judgment (Etteib et al., 22 Jun 2026). Server-side attribution work, by contrast, distinguishes autonomous web agents from humans and crawlers using TLS, HTTP, temporal, and behavioral fingerprints (Kang et al., 18 Jun 2026).

This contrast suggests a useful placement for Agent-ScanKit within the broader agent-evaluation landscape. Security scanners, skill auditors, and fingerprinting systems ask whether an agent or agent artifact is safe, malicious, or attributable. Agent-ScanKit instead asks whether a multimodal GUI agent’s apparent competence is grounded in evidence-sensitive reasoning. Its practical implication is that future GUI agents need better reasoning supervision, stronger out-of-distribution generalization, and more robust handling of corrupted or partial observations. The framework’s broader significance lies in making benchmark success contestable: a high score can coexist with strong memory bias, instruction-template dependence, and shortcut-driven action selection.