MobileGym: Substrate for Mobile GUI Adaptation

• MobileGym is an interaction-and-evaluation substrate that drives annotation-free adaptation by exploring target app states, mining executable tasks, and providing structured critic feedback.
• It systematically collects app-grounded evidence using goal-oriented exploration that leverages app structure to generate diverse and executable curriculum tasks.
• Its hierarchical critic component evaluates rollout performance step-by-step, generates corrective hints, and supports HiFPO for improved policy optimization.

MobileGym is the interaction-and-evaluation substrate inside MobileForge for annotation-free adaptation of mobile GUI agents. Rather than denoting a single model, it grounds adaptation in real target-app interaction: it explores reachable GUI states, mines executable tasks from those interactions, and evaluates completed attempts with hierarchical feedback. Within the MobileForge loop, MobileGym supplies exploration evidence $\mathcal{Z}$, curriculum tasks $\mathcal{T}$, and critic feedback $\mathcal{F}_k$, while Hierarchical Feedback-Guided Policy Optimization (HiFPO) converts those outputs into hint-contextualized step-level GRPO updates (Liu et al., 18 Jun 2026).

1. System role, problem setting, and architectural position

MobileGym is defined in a setting where a system is given target mobile apps and an initial GUI policy, but not the supervision that many agent methods depend on: no human-written tasks, no expert demonstrations, and no reward labels for those apps. In that setting, an agent cannot simply be trained on curated benchmarks; it must discover what the app can do, generate tasks that are actually executable in that app, attempt those tasks, evaluate its own behavior, and then turn that experience into learning signal. MobileGym is introduced precisely to provide the unified substrate tying together exploration, curriculum generation, rollout execution, and evaluation (Liu et al., 18 Jun 2026).

The system-level loop is written as

$$\mathcal{Z} \leftarrow \operatorname{Explore}(\mathcal{E}),\quad \mathcal{T} \leftarrow \operatorname{Curriculum}(\mathcal{Z}),\quad \{\tau_k\}_{k=1}^{K} \leftarrow \operatorname{Rollout}(\pi_\theta,x,\eta_{<k}),\quad \forall x\in\mathcal{T},\quad \mathcal{F}_{k} \leftarrow \operatorname{Critic}(x,\tau_k),\quad \mathcal{D} \leftarrow \operatorname{HiFPO}(\mathcal{T},\tau,\mathcal{F}),\quad \theta' \leftarrow \operatorname{GRPO}(\theta,\mathcal{D}).$$

In this decomposition, MobileGym is responsible for the first, second, and fourth stages in spirit: exploration evidence $\mathcal{Z}$, curriculum $\mathcal{T}$, and critic feedback $\mathcal{F}_k$. HiFPO then uses those outputs to do feedback-guided policy optimization. This division of labor is central: MobileGym builds the environment-facing learning substrate, and HiFPO converts the substrate’s feedback into policy updates.

Component	Function	Output
Target-app exploration	Explore reachable GUI states	$\mathcal{Z}$
MobileGym-Curriculum	Mine executable tasks from traces	$\mathcal{T}$
MobileGym-Critic	Evaluate completed rollouts hierarchically	$\mathcal{F}_k$

A recurrent misconception is to treat MobileGym as a policy architecture. The paper states the opposite: it is not a single model, but the interaction-and-evaluation substrate inside MobileForge.

2. Exploration as app-grounded evidence collection

The first major piece of MobileGym is target-app exploration. The exploration strategy is function-aware and is described as inspired by GUI-explorer. Instead of random wandering, it uses app-level structural anchors—especially the activities declared in the APK—combined with the current screenshot to generate goal-oriented exploration tasks. It then performs depth-first traversal, restoring parent states when branching, so that the agent systematically explores app functions and screens (Liu et al., 18 Jun 2026).

Each explored transition records the before/after screenshots, executed action, target element, and a short natural-language summary. These transition records become the exploration evidence pool $\mathcal{T}$0. The paper is explicit that this evidence is not treated as expert demonstration; it is raw app-grounded evidence of what the app can do.

This distinction is methodologically important. If the exploration traces were treated as demonstrations, the system would collapse back toward supervised imitation. MobileGym instead uses exploration traces as reachable-behavior evidence. This makes the substrate compatible with annotation-free adaptation, where the central question is not how to replay expert data but how to build training signal directly from target-app interaction.

A plausible implication is that MobileGym’s exploration stage functions as the empirical boundary condition for the rest of the pipeline: only behaviors shown to be reachable during exploration can later be turned into executable tasks and meaningful evaluations.

3. Curriculum mining and task representation

The second major piece is MobileGym-Curriculum, which turns exploration evidence into executable tasks. The paper makes a strong distinction here: a task is not something humans write from scratch, but something mined from observed interaction traces. A generated task is represented as

$\mathcal{T}$1

where $\mathcal{T}$2 is the instruction, $\mathcal{T}$3 is the estimated step budget, $\mathcal{T}$4 is the core functionality, $\mathcal{T}$5 is the variation type, and $\mathcal{T}$6 describes prerequisites (Liu et al., 18 Jun 2026).

The crucial property is that each task is grounded in reachable behavior observed during exploration. The curriculum prompt asks the model both to evaluate the original trajectory and to generate new self-contained task variants. As a result, MobileGym-Curriculum is not abstract task synthesis; it is curriculum mining from real app interactions, with coverage over different functionalities and varying difficulty.

The paper contrasts this trajectory-grounded curriculum with a landing-screen baseline. The landing-screen baseline over-focuses on recipe creation, editing, and deletion, while MobileGym-Curriculum covers more diverse flows such as shopping lists, cooking assistant, meal planning, settings, and media sharing. This comparison clarifies the intended role of MobileGym-Curriculum: it broadens functional coverage by operating on exploration traces rather than on the first visible screen alone.

This directly addresses a common failure mode in annotation-free GUI learning. If generated tasks are not grounded in target-app trajectories, they may describe actions that are linguistically plausible but operationally unreachable. MobileGym-Curriculum is designed to avoid that failure mode by binding the curriculum to observed transitions.

4. Hierarchical rollout evaluation and critic design

The third major piece is MobileGym-Critic, which performs hierarchical rollout evaluation. For a completed rollout $\mathcal{T}$7, it outputs

$\mathcal{T}$8

Here $\mathcal{T}$9 is the trajectory-level outcome label, where $\mathcal{F}_k$0 means the task was completed and $\mathcal{F}_k$1 means failure. The step-level process label is

$\mathcal{F}_k$2

where $\mathcal{F}_k$3 says whether the step is reasonable, and $\mathcal{F}_k$4 is a natural-language rationale. The corrective hint $\mathcal{F}_k$5 is produced when the attempt fails or includes unreasonable steps, summarizing the key mistake, what to avoid, suggested alternatives, and important task insights (Liu et al., 18 Jun 2026).

The paper emphasizes that these three signals serve different purposes. Outcome labels decide whether the task was solved. Process labels identify useful local decisions inside successful and failed attempts. Hints carry reusable failure information across attempts. MobileGym-Critic therefore produces structured supervision rather than a single sparse success/failure number.

Architecturally, the critic is implemented as a hierarchical, agentic evaluator rather than a learned reward model. It first converts the raw execution log into visual action traces by rendering each step as an action-centered screenshot, then a vision-LLM describes the action and relevant screen evidence. A final decision model receives the task, raw logs, step descriptions, and stitched last screenshots and returns a structured JSON verdict.

Two clarifications follow from this design. First, MobileGym-Critic is not a scalar reward head. Second, it is not limited to terminal success detection. Its purpose is to inspect where a rollout went right or wrong, step by step, and to output reusable diagnostic structure.

5. Interaction with HiFPO and empirical role in adaptation

MobileGym and HiFPO are tightly coupled at the repeated-attempt level. HiFPO runs $\mathcal{F}_k$6 attempts per task, and before attempt $\mathcal{F}_k$7 it aggregates hints from earlier attempts: $\mathcal{F}_k$8 It then samples the next attempt as

$\mathcal{F}_k$9

The paper is explicit that the hint is appended to the instruction before the next attempt, while the screenshot and history remain the standard observation (Liu et al., 18 Jun 2026).

HiFPO then filters tasks using the empirical success rate

$$\mathcal{Z} \leftarrow \operatorname{Explore}(\mathcal{E}),\quad \mathcal{T} \leftarrow \operatorname{Curriculum}(\mathcal{Z}),\quad \{\tau_k\}_{k=1}^{K} \leftarrow \operatorname{Rollout}(\pi_\theta,x,\eta_{<k}),\quad \forall x\in\mathcal{T},\quad \mathcal{F}_{k} \leftarrow \operatorname{Critic}(x,\tau_k),\quad \mathcal{D} \leftarrow \operatorname{HiFPO}(\mathcal{T},\tau,\mathcal{F}),\quad \theta' \leftarrow \operatorname{GRPO}(\theta,\mathcal{D}).$$0

removing mastered all-success tasks while retaining all-fail and partially solved tasks. For each retained task, it selects the most informative attempt based on the fraction of reasonable steps: $$\mathcal{Z} \leftarrow \operatorname{Explore}(\mathcal{E}),\quad \mathcal{T} \leftarrow \operatorname{Curriculum}(\mathcal{Z}),\quad \{\tau_k\}_{k=1}^{K} \leftarrow \operatorname{Rollout}(\pi_\theta,x,\eta_{<k}),\quad \forall x\in\mathcal{T},\quad \mathcal{F}_{k} \leftarrow \operatorname{Critic}(x,\tau_k),\quad \mathcal{D} \leftarrow \operatorname{HiFPO}(\mathcal{T},\tau,\mathcal{F}),\quad \theta' \leftarrow \operatorname{GRPO}(\theta,\mathcal{D}).$$1 Only reasonable local decisions from that attempt become step-level training examples.

For each selected step, HiFPO forms a prompt $$\mathcal{Z} \leftarrow \operatorname{Explore}(\mathcal{E}),\quad \mathcal{T} \leftarrow \operatorname{Curriculum}(\mathcal{Z}),\quad \{\tau_k\}_{k=1}^{K} \leftarrow \operatorname{Rollout}(\pi_\theta,x,\eta_{<k}),\quad \forall x\in\mathcal{T},\quad \mathcal{F}_{k} \leftarrow \operatorname{Critic}(x,\tau_k),\quad \mathcal{D} \leftarrow \operatorname{HiFPO}(\mathcal{T},\tau,\mathcal{F}),\quad \theta' \leftarrow \operatorname{GRPO}(\theta,\mathcal{D}).$$2 that includes the task, screenshot, history, and any corrective hint. Candidate responses are scored with a rule-based adaptive GUI action reward: $$\mathcal{Z} \leftarrow \operatorname{Explore}(\mathcal{E}),\quad \mathcal{T} \leftarrow \operatorname{Curriculum}(\mathcal{Z}),\quad \{\tau_k\}_{k=1}^{K} \leftarrow \operatorname{Rollout}(\pi_\theta,x,\eta_{<k}),\quad \forall x\in\mathcal{T},\quad \mathcal{F}_{k} \leftarrow \operatorname{Critic}(x,\tau_k),\quad \mathcal{D} \leftarrow \operatorname{HiFPO}(\mathcal{T},\tau,\mathcal{F}),\quad \theta' \leftarrow \operatorname{GRPO}(\theta,\mathcal{D}).$$3 The resulting objective is

$$\mathcal{Z} \leftarrow \operatorname{Explore}(\mathcal{E}),\quad \mathcal{T} \leftarrow \operatorname{Curriculum}(\mathcal{Z}),\quad \{\tau_k\}_{k=1}^{K} \leftarrow \operatorname{Rollout}(\pi_\theta,x,\eta_{<k}),\quad \forall x\in\mathcal{T},\quad \mathcal{F}_{k} \leftarrow \operatorname{Critic}(x,\tau_k),\quad \mathcal{D} \leftarrow \operatorname{HiFPO}(\mathcal{T},\tau,\mathcal{F}),\quad \theta' \leftarrow \operatorname{GRPO}(\theta,\mathcal{D}).$$4

MobileGym is not inside this formula directly, but the paper states that it supplies the feedback that determines which steps enter the dataset $$\mathcal{Z} \leftarrow \operatorname{Explore}(\mathcal{E}),\quad \mathcal{T} \leftarrow \operatorname{Curriculum}(\mathcal{Z}),\quad \{\tau_k\}_{k=1}^{K} \leftarrow \operatorname{Rollout}(\pi_\theta,x,\eta_{<k}),\quad \forall x\in\mathcal{T},\quad \mathcal{F}_{k} \leftarrow \operatorname{Critic}(x,\tau_k),\quad \mathcal{D} \leftarrow \operatorname{HiFPO}(\mathcal{T},\tau,\mathcal{F}),\quad \theta' \leftarrow \operatorname{GRPO}(\theta,\mathcal{D}).$$5 and which hint context $$\mathcal{Z} \leftarrow \operatorname{Explore}(\mathcal{E}),\quad \mathcal{T} \leftarrow \operatorname{Curriculum}(\mathcal{Z}),\quad \{\tau_k\}_{k=1}^{K} \leftarrow \operatorname{Rollout}(\pi_\theta,x,\eta_{<k}),\quad \forall x\in\mathcal{T},\quad \mathcal{F}_{k} \leftarrow \operatorname{Critic}(x,\tau_k),\quad \mathcal{D} \leftarrow \operatorname{HiFPO}(\mathcal{T},\tau,\mathcal{F}),\quad \theta' \leftarrow \operatorname{GRPO}(\theta,\mathcal{D}).$$6 conditions the prompt. The empirical effect is substantial. In a 200-task rollout study, removing corrective hint context drops overall success from $$\mathcal{Z} \leftarrow \operatorname{Explore}(\mathcal{E}),\quad \mathcal{T} \leftarrow \operatorname{Curriculum}(\mathcal{Z}),\quad \{\tau_k\}_{k=1}^{K} \leftarrow \operatorname{Rollout}(\pi_\theta,x,\eta_{<k}),\quad \forall x\in\mathcal{T},\quad \mathcal{F}_{k} \leftarrow \operatorname{Critic}(x,\tau_k),\quad \mathcal{D} \leftarrow \operatorname{HiFPO}(\mathcal{T},\tau,\mathcal{F}),\quad \theta' \leftarrow \operatorname{GRPO}(\theta,\mathcal{D}).$$7 to $$\mathcal{Z} \leftarrow \operatorname{Explore}(\mathcal{E}),\quad \mathcal{T} \leftarrow \operatorname{Curriculum}(\mathcal{Z}),\quad \{\tau_k\}_{k=1}^{K} \leftarrow \operatorname{Rollout}(\pi_\theta,x,\eta_{<k}),\quad \forall x\in\mathcal{T},\quad \mathcal{F}_{k} \leftarrow \operatorname{Critic}(x,\tau_k),\quad \mathcal{D} \leftarrow \operatorname{HiFPO}(\mathcal{T},\tau,\mathcal{F}),\quad \theta' \leftarrow \operatorname{GRPO}(\theta,\mathcal{D}).$$8, with Pass@3 falling from $$\mathcal{Z} \leftarrow \operatorname{Explore}(\mathcal{E}),\quad \mathcal{T} \leftarrow \operatorname{Curriculum}(\mathcal{Z}),\quad \{\tau_k\}_{k=1}^{K} \leftarrow \operatorname{Rollout}(\pi_\theta,x,\eta_{<k}),\quad \forall x\in\mathcal{T},\quad \mathcal{F}_{k} \leftarrow \operatorname{Critic}(x,\tau_k),\quad \mathcal{D} \leftarrow \operatorname{HiFPO}(\mathcal{T},\tau,\mathcal{F}),\quad \theta' \leftarrow \operatorname{GRPO}(\theta,\mathcal{D}).$$9 to $\mathcal{Z}$0. On AndroidWorld, using only automatically generated annotation-free adaptation data, MobileForge adapts Qwen3-VL-8B to $\mathcal{Z}$1 Pass@3, close to the closed-data GUI-specialized GUI-Owl-1.5-8B base model at $\mathcal{Z}$2. The MobileForge-adapted ForgeOwl-8B further reaches $\mathcal{Z}$3 Pass@3 on AndroidWorld and $\mathcal{Z}$4 success on the out-of-domain MobileWorld GUI-only split.

These results are used in the paper to argue that MobileGym’s target-app grounding plus HiFPO’s feedback reuse creates transferable improvements rather than isolated benchmark gains.

6. Related “MobileGym-style” mobile exercise systems

In the supplied literature, several systems are presented as relevant to a “MobileGym-style system,” although they are distinct from the GUI-agent MobileGym itself. This adjacent usage concerns smartphone-based exercise recognition, exercise learning, physiotherapy supervision, and real-time coaching.

One line of work uses smartphone sensing directly for workout recognition. "Generalization of Fitness Exercise Recognition from Doppler Measurements by Domain-adaption and Few-Shot Learning" describes a mobile phone as a small sonar system that emits a continuous $\mathcal{Z}$5 tone from its loudspeaker and records the returning echo with the microphone. It applies a short-time Fourier transform over sliding windows, using $\mathcal{Z}$6 windows with $\mathcal{Z}$7 overlap, keeps the $\mathcal{Z}$8 band, and studies eight whole-body exercises in controlled and uncontrolled settings. The central result is that a lab-trained model generalizes poorly in realistic use, while domain adaptation and few-shot learning improve recognition accuracy by about $\mathcal{Z}$9 over baseline, depending on the participant and setting (Fu et al., 2023).

A second line emphasizes exercise instruction rather than sensing. "Comparative Study of AR Versus Image and Video for Exercise Learning" presents ARFit, a mobile augmented reality exercise-learning app built in Unity3D with the Vuforia AR SDK. In a within-subject study with $\mathcal{T}$0 participants, mobile AR is reported as better than image-based instruction for all tested exercises, better than video for exercises needing stronger spatial understanding, and associated with a mean System Usability Scale score of $\mathcal{T}$1; $\mathcal{T}$2 participants preferred AR (Burns et al., 2022).

A third line focuses on fully on-device movement analysis. "An M-Health Algorithmic Approach to Identify and Assess Physiotherapy Exercises in Real Time" describes a client-side mobile pipeline based on camera input, pose estimation with PoseNet or MoveNet, $\mathcal{T}$3 distinct body-section angles, lightweight classification with kNN, and sequence matching with a modified Levenshtein distance algorithm for exercise identification, scoring, and localization of inaccuracies (Kandylakis et al., 11 Dec 2025). "Pūioio: On-device Real-Time Smartphone-Based Automated Exercise Repetition Counting System" uses BlazePose, thresholding, optical flow, a state machine, and a counter to support real-time repetition counting for squats, push-ups, and pull-ups, reporting $\mathcal{T}$4 absolute accuracy in real-life testing and up to $\mathcal{T}$5 on a pre-recorded dataset (Sinclair et al., 2023).

A fourth line uses wearable sensing and gamified feedback. "Designing Just-in-Time Detection for Gamified Fitness Frameworks" combines a single wearable IMU on the upper arm, Bluetooth Low Energy transfer to a smartphone, online set segmentation, and a Dynamic Naive Bayes model for early repetition detection. It reports detection as early as $\mathcal{T}$6 before the repetition ends and couples this with an iOS connect-the-dots visualization intended to support just-in-time interventions and form improvement (Milanko et al., 2020).

Taken together, these papers do not define MobileGym as a single exercise platform. Rather, they show that the phrase “MobileGym-style system” has been used to organize a broader family of mobile exercise technologies: unobtrusive sensing, virtual coaching, client-side rehabilitation monitoring, and gamified real-time feedback. This suggests that “MobileGym” operates in the literature both as a concrete substrate for annotation-free mobile GUI adaptation and as a broader conceptual frame for mobile, on-device, interaction-grounded coaching systems.