Interleaved GUI-Tool Trajectory Scaling Pipeline

• The topic is a system that interleaves screenshots, action histories, tool calls, and reasoning traces into long-horizon trajectories for training and inference.
• It integrates GUI actions and high-level tool invocations using synthesis, calibration, and compression to optimize performance across diverse tasks.
• Techniques such as trajectory reward modeling, visual compression, and continuous memory scaling ensure robust performance and efficient deployment in complex GUI environments.

Searching arXiv for the cited GUI trajectory-scaling and GUI-agent papers to ground the article in current literature. An interleaved GUI-tool trajectory scaling pipeline denotes a class of systems in which screenshots, action histories, reasoning traces, tool calls, tool outputs, and verification signals are organized into long-horizon trajectories and then expanded, filtered, compressed, replayed, or optimized for training and inference. In recent GUI-agent work, the concept is distributed across several complementary components: visual trajectory compression for pure-vision agents, large-scale trajectory synthesis and calibration, hybrid GUI-tool action modeling, trajectory-aware reinforcement learning, and deployment architectures that interleave on-device execution with cloud assistance (Hu et al., 12 May 2026, Xu et al., 26 Feb 2026, Tang et al., 19 Jul 2025, Zhou et al., 26 Dec 2025).

1. Formal setting and representational units

A common starting point is the pure-vision GUI agent, which uses screenshots as the primary state representation, a multimodal LLM with a ViT-like encoder, and an autoregressive trajectory of historical screenshots and actions. In this formulation, the history at time $t$ is written as

$\mathcal{H}_t = \{(X_{t-k}, a_{t-k})\}_{k=1}^{T},$

conditioned together with the current screenshot $X_t$ and instruction $\mathcal{I}$. This formulation already has the structure of an interleaved GUI trajectory—screen $\rightarrow$ action $\rightarrow$ screen—but without explicit external tool channels (Xu et al., 26 Feb 2026).

When reasoning is made explicit, the trajectory expands from state-action pairs into state-thought-action sequences. MagicGUI writes high-level decision making as

$$t_n, a_n = \text{VLM}(task, a_{0:n-1}, t_{0:n-1}, o_n),$$

where $o_n$ is the screenshot observation, $a_{0:n-1}$ the past action sequence, and $t_{0:n-1}$ the past reasoning text or “meta-plan.” This makes the trajectory an interleaved sequence of image tokens, task text, prior thoughts, and executable actions, rather than a purely behavioral trace (Tang et al., 19 Jul 2025).

In hybrid GUI-tool settings, the action space is explicitly partitioned into low-level GUI operations and higher-level tool calls. ToolCUA formalizes this as an MDP $\mathcal{H}_t = \{(X_{t-k}, a_{t-k})\}_{k=1}^{T},$0 with

$\mathcal{H}_t = \{(X_{t-k}, a_{t-k})\}_{k=1}^{T},$1

where states contain screenshots and past tool results, and actions include both computer_use-style GUI primitives and MCP tools. A plausible implication is that interleaving is not merely a logging convention; it is a policy problem over a hybrid action manifold in which each switch between GUI and tool branches alters the remaining execution path (Hu et al., 12 May 2026).

2. Trajectory construction, calibration, and synthesis

A major research axis concerns how such trajectories are created at scale. OS-Genesis reverses conventional task-first collection by first exploring environments through step-wise interactions and only then retrospectively synthesizing low-level and high-level tasks from observed state transitions. It records triplets $\mathcal{H}_t = \{(X_{t-k}, a_{t-k})\}_{k=1}^{T},$2, uses GPT-4o to derive low-level and high-level instructions, executes those synthesized tasks to obtain full trajectories, and then applies a Trajectory Reward Model to score them on a 1–5 scale before reward-weighted supervised training (Sun et al., 2024).

Tree-structured approaches replace flat trajectory corpora with reusable topologies. TreeCUA models exploration as a rooted tree whose nodes are GUI states and whose edges are actions, with an exploration tuple $\mathcal{H}_t = \{(X_{t-k}, a_{t-k})\}_{k=1}^{T},$3. Its multi-agent framework combines an Exploration Agent, Verification Agent, Summary Agent, Evaluation Agent, and Reasoning Improvement Agent. This design reuses shared prefixes, supports replay from canonical paths, and yields step-level, sub-trajectory, and long-trajectory datasets through online exploration followed by offline summarization and filtering (Jiang et al., 10 Feb 2026).

Hardness-aware and robustness-aware synthesis pipelines make trajectory quality depend on failure structure rather than simple completion. HATS defines hardness through alignment between a selected reference sub-trajectory $\mathcal{H}_t = \{(X_{t-k}, a_{t-k})\}_{k=1}^{T},$4 and an execution sequence $\mathcal{H}_t = \{(X_{t-k}, a_{t-k})\}_{k=1}^{T},$5, with action-level reconstruction recall

$\mathcal{H}_t = \{(X_{t-k}, a_{t-k})\}_{k=1}^{T},$6

and hardness reward

$\mathcal{H}_t = \{(X_{t-k}, a_{t-k})\}_{k=1}^{T},$7

This converts semantic ambiguity into an exploration signal for MCTS, so that difficult, context-dependent, and sequentially dependent actions are oversampled rather than suppressed (Shao et al., 12 Mar 2026). RoTS adopts a different tree-based pipeline oriented toward error recovery: it constructs trajectory trees over 20k tasks, uses Fragility-Driven Exploration and Experience-Informed Recovery, and produces 800k training samples after critic filtering and deduplication (Bu et al., 28 May 2026).

Calibration is equally central. UI-Genie uses a reward model, UI-Genie-RM, with an image-text interleaved architecture, controlled trajectory corruption, hard negative mining, and reward-guided beam search, producing UI-Genie-RM-517k and UI-Genie-Agent-16k through three generations of self-improvement (Xiao et al., 27 May 2025). Step-GUI introduces the Calibrated Step Reward System, which converts model-generated trajectories into reliable training signals through trajectory-level calibration, achieving “>90% annotation accuracy with 10-100x lower cost,” and then extracts seven categories of supervision from successful or partially correct traces (Yan et al., 17 Dec 2025). AutoGUI, although centered on functionality grounding rather than long-horizon control, contributes a scalable precursor layer by constructing AutoGUI-704k from pre/post interaction states and LLM-aided rejection and verification, thereby supplying semantically grounded element-function pairs that can serve as action semantics for later trajectory pipelines (Li et al., 4 Feb 2025).

3. Interleaving GUI actions, tool calls, and reasoning

Interleaving becomes explicit once action spaces are unified. MagicGUI defines a unified mobile action space spanning Tap, Scroll, Text Input, Navigation Back, Navigation Home, Long Press, Finish, Wait, Enter, Takeover, Drag, Call API, Screenshot, Long Screenshot, and NoAnswer, and trains on outputs that contain both > ... reasoning and a canonical action string. This turns the trajectory into a ChatML-like sequence where reasoning text and executable actions coexist under a single autoregressive policy (Tang et al., 19 Jul 2025).

MAI-UI extends this further by incorporating user interaction and MCP tools directly into the action space: GUI primitives such as click(x,y) and drag(x1,y1,x2,y2) coexist with answer(text), ask_user(text), and mcp_call(tool_name, arguments). The same self-evolving data pipeline is used to synthesize trajectories where missing information forces ask_user, and tasks that benefit from tools induce mcp_call, so the agent learns when to continue with GUI actions, when to query the user, and when to compress a subroutine into a tool invocation (Zhou et al., 26 Dec 2025).

Step-GUI generalizes the interface layer by proposing GUI-MCP, described as “the first Model Context Protocol for GUI automation,” with a hierarchical architecture combining low-level atomic operations such as click, swipe, drag_to, and input_text with a high-level execute_task(task_description: str) interface backed by local specialist models. ToolCUA operationalizes the same principle for desktop CUAs: it synthesizes grounded tool libraries from GUI-only traces, creates tool-only trajectories, and then randomly replaces selected tool calls with their corresponding GUI sub-sequences while removing those tools from the available library. This yields explicit GUI$\mathcal{H}_t = \{(X_{t-k}, a_{t-k})\}_{k=1}^{T},$8Tool switching boundaries and partial tool-availability contexts, which are crucial for learning path selection rather than isolated tool syntax (Yan et al., 17 Dec 2025, Hu et al., 12 May 2026).

A plausible implication is that “interleaving” has at least three granularities in the literature: intra-step interleaving of thought and action, intra-trajectory interleaving of GUI primitives and external tools, and architectural interleaving of planner, executor, verifier, and memory roles.

4. Context scaling: spatio-temporal pruning, retrieval, and memory

Long GUI trajectories stress both compute and context limits, especially for high-resolution screenshots. GUIPruner addresses the visual side of this problem by identifying temporal mismatch and spatial topology conflict in pure-vision GUI agents. Its Temporal-Adaptive Resolution module assigns a global history budget

$\mathcal{H}_t = \{(X_{t-k}, a_{t-k})\}_{k=1}^{T},$9

and distributes it across frames with a linear decay schedule

$X_t$0

then rescales each historical frame before encoding. Its Stratified Structure-aware Pruning module applies edge-based foreground/background partitioning, shallow-layer attention ranking, and Uniform Grid Sampling to current-frame tokens, preserving layout while pruning aggressively. On Qwen2-VL-2B, GUIPruner reports a 3.4x reduction in FLOPs and a 3.3x encoder speedup while retaining over 94% of original performance (Xu et al., 26 Feb 2026).

A different response to trajectory growth is continuous memory. “Auto-scaling Continuous Memory for GUI Agent” encodes each GUI trajectory into a fixed-length sequence of continuous embeddings $X_t$1 using a Q-Former-style memory encoder, retrieves top-$X_t$2 related trajectories by similarity, and prepends those embeddings directly to the backbone input. The reported scaling law for memory size is

$X_t$3

and performance improves monotonically with larger memory and retrieval depth, unlike text memories that degrade with long prompts. The associated flywheel discovers environments via search, synthesizes tasks with an open-source VLM, rolls out trajectories, verifies them with the same VLM, and collects 100k+ trajectories for about \$4000 (Wu et al., 10 Oct 2025).

Mobile-Agent-v3.5 / GUI-Owl-1.5 adopts a hierarchical context scheme closer to sequence compression than explicit retrieval. It keeps a recent context of the last $X_t$4 multimodal steps in full form, while older steps are folded into a compressed historical context

$X_t$5

where $X_t$6 is an action conclusion. This suggests a broader design split in the literature between budgeted visual compression, fixed-length learned memory, and text-based or conclusion-based summarization (Xu et al., 15 Feb 2026).

5. Optimization, rewards, and policy improvement

Training regimes in this area combine large-scale supervised pretraining with increasingly trajectory-aware reinforcement. MagicGUI uses continued pre-training on 7.8M samples, followed by reinforcement fine-tuning with a spatially enhanced composite reward

$X_t$7

where format, action correctness, and coordinate proximity are scored jointly, and DF-GRPO filters out uninformative rollout groups before optimization (Tang et al., 19 Jul 2025).

Reward modeling and calibrated supervision are prominent alternatives when dense online rewards are unavailable. UI-Genie trains UI-Genie-RM to score both actions and complete trajectories from interleaved screenshot-history-action inputs, treating task completion as a special terminate action. It combines rule-based verification with continuation-based process supervision, while Step-GUI uses CSRS to attach seven categories of structured step-level supervision to a trajectory-level success/failure judgment and then applies RLVR with calibrated rewards over GUI actions and reasoning traces (Xiao et al., 27 May 2025, Yan et al., 17 Dec 2025).

Trajectory-level optimization becomes explicit in long-horizon online RL. GUI-Owl / Mobile-Agent-v3 proposes Trajectory-aware Relative Policy Optimization, in which a trajectory-level advantage

$X_t$8

is applied uniformly across all steps in the trajectory, together with a replay buffer of successful episodes, to cope with sparse success rewards in OSWorld (Ye et al., 21 Aug 2025). ToolCUA adds path-level structure with the Tool-Efficient Path Reward

$X_t$9

where $\mathcal{I}$0 rewards tool use on tool-beneficial tasks and GUI-only behavior on non-tool-beneficial tasks, while $\mathcal{I}$1 favors shorter successful paths. This makes switching between GUI and tools a learned trajectory-level optimization target rather than a purely local choice (Hu et al., 12 May 2026).

DeepTool extends the same logic beyond GUIs to tool-integrated reasoning. It decomposes trajectories into step-wise interleaved triples $\mathcal{I}$2, synthesizes them with a Manager–Actor hierarchy, injects adversarial perturbations, and trains with an Action-Centric Process Reward

$\mathcal{I}$3

showing that process-supervised RL can explicitly reinforce intermediate thought-action-observation structure instead of relying only on terminal outcomes. A plausible implication is that GUI-tool pipelines and general tool-integrated reasoning pipelines are converging on the same optimization abstractions (He et al., 28 May 2026).

6. Robustness, dexterity, deployment, and open problems

The newest trajectory pipelines increasingly treat robustness as a first-class scaling target. GUI-RobustEval measures policy-induced error recovery on 1,216 executable test cases across 11 error categories and four error depths, using Error-Awareness Rate and Post-Error Success Rate. RoTS then uses a tree-based synthesis pipeline to generate 800k high-quality samples aligned with those failure distributions. RoTS-32B reaches 47.4% success rate and 33.8% All-Pass@4 on OSWorld, while RoTS-7B and RoTS-32B both improve substantially on error-awareness and post-error recovery relative to earlier open models (Bu et al., 28 May 2026).

Dexterous control introduces another layer of trajectory complexity. ShowUI-$\mathcal{I}$4 replaces discrete click prediction with unified discrete–continuous action sequences $\mathcal{I}$5, $\mathcal{I}$6, and learns a flow-based action policy for closed-loop drags and fine-grained cursor motion. On ScreenDrag, it achieves 26.98 online with only 450M parameters, outperforming substantially larger proprietary GUI agents on that benchmark. This suggests that future interleaved pipelines may require a separation between a high-level planner and a specialized “dexterous hand” for continuous motor execution (Hu et al., 31 Dec 2025).

Deployment-oriented work adds system constraints absent from benchmark-centric pipelines. MAI-UI couples a local 2B agent, a cloud 32B or 235B-A22B agent, a trajectory monitor, and a privacy monitor inside a native device-cloud collaboration system. The system routes execution according to task state, keeps unified trajectory memory locally, improves on-device performance by 33%, reduces cloud model calls by over 40%, and preserves user privacy. At the same time, multiple papers identify unresolved issues: tool schema coverage, long-context cost, verification noise from LLM-as-a-judge, retrieval drift under UI shift, dependence on strong proprietary models for synthesis, and the difficulty of detecting semantic duplicates in mixed GUI-tool states (Zhou et al., 26 Dec 2025, Jiang et al., 10 Feb 2026, Wu et al., 10 Oct 2025, Shao et al., 12 Mar 2026).

Taken together, these works suggest that an interleaved GUI-tool trajectory scaling pipeline is no longer a single model or dataset recipe. It is an integrated stack in which trajectory synthesis, context compression, hybrid action spaces, critic-guided filtering, trajectory-level rewards, memory retrieval, recovery synthesis, and deployment routing are co-designed. The current literature increasingly treats scale not as raw sample count alone, but as the ability to generate, verify, compress, recover, and optimize trajectories whose structure matches the real long-horizon failure modes of GUI use.