MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization
Abstract: MLLM-based mobile GUI agents have made substantial progress in UI understanding and action execution, but adapting them to real target apps remains costly because mobile apps are numerous, frequently updated, and hard to cover with human-written tasks, demonstrations, or reward labels. Existing annotation-free GUI learning reduces manual supervision, yet lacks a unified substrate connecting target-app exploration, curriculum mining, rollout execution, and feedback, while policy optimization often relies on isolated rollouts and coarse rewards that are hard to convert into reliable improvement signals. We present MobileForge, an annotation-free adaptation system for mobile GUI agents. MobileForge consists of MobileGym, which grounds task generation and rollout evaluation in real mobile app interaction, and Hierarchical Feedback-Guided Policy Optimization (HiFPO), which turns trajectory outcomes, step-level process feedback, and corrective hints into hint-contextualized step-level GRPO updates. Using only automatically generated annotation-free adaptation data, MobileForge adapts Qwen3-VL-8B to 67.2% Pass@3 on AndroidWorld, close to the closed-data GUI-specialized GUI-Owl-1.5-8B base model at 69.0%. The MobileForge-adapted ForgeOwl-8B further reaches 77.6% Pass@3 on AndroidWorld and 41.0% success on the out-of-domain MobileWorld GUI-only split, establishing the strongest open-data mobile GUI agent in our evaluation. Code, data, and trained models will be released at https://mobile-forge.github.io/.
Summary
- The paper introduces an annotation-free adaptation framework for mobile GUI agents that leverages real app interactions for policy optimization.
- The methodology employs hierarchical feedback via MobileGym and HiFPO, yielding substantial improvements in task success (e.g., +12.5% Pass@3).
- Empirical results demonstrate robust in-domain and cross-domain generalization, narrowing gaps with closed-data models.
MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization
Motivation and Research Problem
Mobile GUI agents require robust adaptation to rapidly evolving app ecosystems. Supervised approaches relying on human-written tasks, demonstrations, and reward labels are expensive and quickly obsolete given the velocity of app updates. Recent annotation-free learning methods partially address data bottlenecks but fail to provide a unified substrate connecting app exploration, task mining, rollout execution, and feedback-driven policy optimization. Isolated rollouts with sparse reward signals leave agents unable to leverage nuanced feedback. The central research question addressed is whether annotation-free adaptation can be achieved by grounding task generation in real app interactions, generating hierarchical feedback, and using self-collected experience for policy improvement without manual supervision (2606.19930).
Figure 1: Motivation and overview of MobileForge, illustrating bottlenecks in prior annotation-free GUI learning and the system's componentsโMobileGym for app exploration and hierarchical feedback, HiFPO for feedback-guided rollout and policy optimization.
System Architecture
MobileForge consists of two synergistic components:
- MobileGym: Functions as the interaction and evaluation substrate. It facilitates target-app exploration, task mining through trajectory analysis (MobileGym-Curriculum), rollout execution, and hierarchical evaluation via MobileGym-Critic, which provides trajectory outcome labels, step-level process feedback, and corrective hints.
- HiFPO (Hierarchical Feedback-Guided Policy Optimization): Schedules hint-guided multi-attempt rollouts, filters tasks using hierarchical feedback, selects optimal steps, and performs policy training using hint-contextualized step-level GRPO.
Figure 2: Overview of MobileForge, depicting its end-to-end adaptation workflow from exploration through feedback-driven policy optimization.
MobileGym: Annotation-Free Substrate for Adaptation
MobileGym grounds adaptation in real app interactions. It generates exploration evidence by traversing reachable GUI states and mining functional anchors from app metadata. Task variants are generated from these trajectories, ensuring grounding in core app functionalities. Hierarchical evaluation by MobileGym-Critic yields labels for overall attempt success/failure, step-level reasonableness, and detailed corrective hints, which are crucial for fine-grained credit assignment.
Figure 3: MobileGym as the adaptation substrate, illustrating interaction-driven curriculum generation and rollout evaluation yielding hierarchical feedback.
HiFPO: Feedback-Guided Rollout and Policy Update
HiFPO leverages hierarchical feedback to maximize experience reuse. For each task, multiple attempts are made, with corrective hints from failed or sub-optimal attempts accumulated and injected into subsequent attempts. Tasks mastered by the current policy are filtered out; all-fail and partially successful tasks are retained due to their value for extracting correct local actions.
Trajectory selection prioritizes attempts with maximum locally reasonable steps. Only reasonable steps are used for training, and each step is contextually conditioned with the latest corrective hint. Hint-contextualized step-level GRPO then performs group-relative optimization, with rewards decomposed into action-type and argument scores, and normalization across candidate responses. KL-regularization ensures consistency with reference policies.
Figure 4: HiFPO pipeline showing rollout, hierarchical evaluation, hint aggregation, task/step filtering, and hint-contextualized policy update.
Figure 5: Case study of corrective hints improving rollout reliability by encoding failed attempt insights for subsequent attempts.
Empirical Results
In-Domain Adaptation (AndroidWorld)
MobileForge adapts generalist and GUI-specialized models. ForgeQwen3-8B achieves 67.2% Pass@3, nearing the performance of the closed-data GUI-Owl-1.5-8B (69.0%). ForgeOwl-8B reaches 77.6% Pass@3 and 67.2% Pass@1, outperforming its base agent by substantial margins (+12.5%, +20.0%, respectively). Gains are concentrated on easy and medium tasks, but hard-task success also improves (from 19.3% to 29.8%).
Cross-Domain Generalization (MobileWorld GUI-only)
Out-of-domain transfer demonstrates MobileForge's robustness, with ForgeOwl-8B achieving 41.0% success, outperforming previous open-data agents and approaching larger, closed-data baselines.
Ablation Studies
- Introducing corrective hints increases overall rollout success (from 52.0% to 77.0%) and Pass@3 (49.0% to 72.5%).
- Hint-contextualized GRPO consistently outperforms SFT and no-hint baselines in step-level policy optimization.
- Task filtering removing mastered tasks, but retaining all-fail and mixed-success tasks, provides maximal policy improvement.
- Curriculum grounding via exploration yields broader functional coverage than landing-screen task generation.
- Superior evaluator models (Gemini 2.5 Pro) marginally improve outcomes, but the adaptation loop remains effective with weaker open models.
Figure 6: Case study demonstrating advanced multi-item deletion in AndroidWorld, highlighting adaptation's impact on task completion.
Figure 7: Tag-wise failure-rate reduction after MobileForge adaptation; largest gains in verification, search, repetition, and UI reading.
Implications and Future Directions
MobileForge demonstrates that annotation-free adaptation with hierarchical feedback is viable for real-world mobile GUI agents. The system narrows the gap between open and closed-data specialized models and transitions self-collected rollout experience into dense step-level training signals. Theoretical implications include improved credit assignment and adaptation loop design for sequential decision-making agents. Practically, MobileForge enables scalable, sustainable policy improvement without costly human annotation. Future developments should extend the substrate to support multi-app workflows, persistent state, and explicit safety constraints on real devices.
Conclusion
MobileForge introduces a formal architecture for annotation-free mobile GUI agent adaptation, substantiated by strong numerical improvements across established benchmarks. Its hierarchical feedback loop enables granular policy optimization and generalizes across domains. The approach is positioned to inspire further work in autonomous adaptation, curriculum generation, and feedback-driven learning for agentic interaction in broader, more dynamic environments.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
MobileForge: A simple guide for teens
What is this paper about?
This paper is about teaching phoneโusing AI assistants (called โmobile GUI agentsโ) to get better at using apps on their own, without people writing practice tasks, giving stepโbyโstep demos, or marking answers. The system they built is called MobileForge. It helps an AI explore apps, create its own practice tasks, learn from its mistakes, and improve with feedbackโkind of like a student who practices on a phone, gets hints after each attempt, and keeps getting better.
What questions were the researchers trying to answer?
They focused on two big questions:
- Can an AI teach itself to use new or changing mobile apps without humans preparing lessons for it?
- If the AI tries a task several times, can it use detailed feedback and hints from earlier tries to do better next time and actually improve its โbrainโ (its policy) instead of just guessing again?
How did they do it? (Methods explained simply)
Think of a school for phoneโusing AIs with two main parts: a playground and a coach.
- The playground: MobileGym
- What it is: A safe place where the AI can tap, type, and swipe in real apps.
- What it does:
- 1. Explores apps to see what screens and buttons exist.
- 2. Turns these explorations into realistic practice tasks (like โfind and delete three expense itemsโ).
- 3. Judges the AIโs attempts in three ways:
- Overall outcome: Did it finish the task or not?
- Step feedback: Which individual actions were good or bad?
- Hints: Short advice about what went wrong and how to fix it next time.
- The coach: HiFPO (Hierarchical FeedbackโGuided Policy Optimization)
- What it is: A training routine that uses the playgroundโs feedback to improve the AIโs decisionโmaking.
- What it does:
- Runs the same task a few times in a row. After each try, it gives the AI a compact hint (like โdonโt open the menu hereโuse the delete button next to the itemโ).
- Keeps tasks that are still hard or only partly solved and drops the ones the AI already masters every time.
- Picks out the good steps from attempts (even from failed tries) and trains the AI on those good steps.
- Uses a learning method (called GRPO) that compares several possible actions for the same situation and prefers the ones that match good examplesโnow with the hint included so the AI learns โwhat to do given this advice.โ
In everyday terms:
- MobileGym is the playground where the AI learns by doing and gets report cards and tips.
- HiFPO is the coach that says โtry again with this hint,โ saves the best moves, and makes the AI practice those moves until they stick.
What did they find, and why is it important?
They tested MobileForge on two benchmarks:
- AndroidWorld (inโdomain: same kind of apps it practiced on)
- MobileWorld GUIโonly (outโofโdomain: new apps it didnโt practice on)
Key results (why they matter):
- A generalโpurpose model (Qwen3โVLโ8B) improved a lot on AndroidWorld after MobileForge training: it reached 67.2% Pass@3 (success within 3 attempts), getting close to a stronger, specialized model.
- A specialized model (GUIโOwlโ1.5โ8B) improved even more, reaching 77.6% Pass@3 on AndroidWorld and 41.0% success on MobileWorld GUIโonly. Thatโs one of the best openโdata results in their tests.
- Hints really helped. In rollout tests with repeated tries, adding corrective hints boosted success rates a lot (for one setup, Pass@3 jumped from about 49% to about 73%).
- The system learned not just to finish tasks, but to make better individual moves (like tapping the right button, swiping in the correct direction, or typing the right words).
- It handled easy and medium tasks much better and made progress on some harder ones that require keeping the task goal in mind across several steps.
Why this matters:
- Apps change all the time. Writing new training tasks and labels for every update is slow and expensive. MobileForge shows an AI can explore, generate its own tasks, and improve with no extra human labeling.
- The AIโs improvement transfers to new apps. It got better on a different test set it never practiced on, which is a key step toward realโworld usefulness.
What does this mean for the future?
- Fewer human labels needed: Systems like MobileForge can reduce the cost and time needed to adapt AI assistants to new or updated apps.
- Smarter practice: Learning from hints and stepโbyโstep feedback makes practice more efficient and helps the AI avoid repeating mistakes.
- Better generalization: Gains on unseen apps suggest this approach can help assistants handle the fastโchanging app ecosystem.
Limitations and next steps:
- The AI is still limited by the apps it explored during training. More diverse exploration could help it handle longer, multiโapp tasks.
- The quality of the automatic โjudgeโ (the evaluator) matters. Stronger, more reliable evaluators could improve safety and accuracy on real devices.
In short: MobileForge is like giving a phoneโusing AI a playground to explore, a coach that provides targeted hints, and a training plan that turns good moves into lasting skillsโall without needing humans to handcraft lessons. It makes AI assistants more adaptable, practical, and ready for the real world.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The paper advances annotation-free adaptation for mobile GUI agents, but leaves several concrete gaps and open questions that future work can address:
- Limited app and task coverage during adaptation: adaptation data is mined from 20 AndroidWorld apps (527 source trajectories; 3,249 candidate tasks, with subsets of 200/400/900 used). It remains unclear how performance scales with substantially larger, more diverse app ecosystems (e.g., hundreds/thousands of apps, different categories, UIs, and design languages).
- Cross-app workflow handling is undeveloped: the system and experiments do not support or evaluate multi-app tasks (acknowledged as a limitation). Extending state tracking, hinting, exploration, and evaluation to cross-app goals and app-switching remains open.
- Persistent user state and account-dependent workflows are unsupported: tasks involving logins, personalized content, user histories, and long-lived sessions are not modeled or evaluated. How to safely and reliably learn with persistent state remains unknown.
- Real-device constraints are unaddressed: evaluation appears emulator-focused; latency, energy, device heterogeneity, OS/permission dialogs, and unreliable network conditions are not studied, leaving real-world feasibility unclear.
- Safety and risk mitigation are not integrated: there are no explicit guardrails (e.g., permission gating, undo/rollback, irreversible-action avoidance, privacy protection), despite plans for future safety constraints. How to enforce and verify safety during exploration, rollout, and training is unresolved.
- Evaluator reliability and bias remain a bottleneck: MobileGym-Critic relies on LLMs for step descriptions and final decisions (best results use proprietary Gemini), with observed performance differences across evaluators. Robustness to evaluator noise/bias, calibration against human-verified labels, and adversarial robustness are not established.
- Lack of verifiable, deterministic rewards: the system avoids learned reward models and uses hierarchical LLM feedback plus rule-based action matching. There is no systematic validation of evaluator correctness on a gold-standard subset, leaving ground-truth alignment uncertain.
- Hint aggregation is simplistic and under-specified: HiFPO aggregates prior-attempt hints by simple concatenation into prompts. There is no principled method for deduplication, contradiction resolution, weighting by reliability, or retrieval of relevant hints from a larger pool.
- Hints are scoped to within-task reuse only: the system does not build a cross-task โhint memoryโ or failure-pattern library. Whether inter-task, app-level, or UI-pattern-level hints can generalize across tasks/apps is an open question.
- Step selection is one-sided (positive-only): only steps labeled โreasonableโ are kept; unreasonable steps are discarded rather than being used as negative signals. It is unknown whether incorporating negative examples (e.g., contrastive or preference learning) would accelerate learning or reduce repeated mistakes.
- Heuristic action-reward design may be brittle: GRPO rewards depend on rule-based action-type and argument matching (e.g., coordinate thresholds, distance measures, token similarity). Sensitivity to device resolution, UI scaling, widget shapes, localization, and non-standard widgets is untested.
- Insufficient treatment of long-horizon credit assignment: the method decomposes trajectories into locally โreasonableโ steps, but does not study whether local reasonableness correlates with global task optimality, especially under partial observability and non-Markovian GUIs.
- Best-attempt-only training may underutilize data: for each task, only one attempt (often the highest-quality or first success) is used. It is unknown whether aggregating multiple informative attempts, or using weighted sampling across attempts, would yield better learning.
- Limited generalization beyond Android GUI-only: experiments cover AndroidWorld and MobileWorld GUI-only; there is no evaluation on iOS, web UIs, or full MobileWorld (including non-GUI or multi-modal interactions). Cross-platform policy/input adaptations are unexplored.
- Accessibility tree and structured UI signals are not leveraged in policy inputs: the policy is driven by screenshots and text; potential gains from UI hierarchies, semantics, accessibility labels, and element metadata (used during exploration) are not studied for decision making and training.
- Handling of dynamic, non-deterministic, or content-rich apps is unclear: variability in content (ads, feeds, latency), pop-ups, or frequent UI updates is not modeled; robustness under UI drift and content churn remains unquantified.
- Continual learning and adaptation to app updates are not addressed: there is no mechanism for monitoring performance decay as apps change, triggering re-exploration, or avoiding catastrophic forgetting during ongoing adaptation.
- Limited exploration strategy analysis: function-aware, depth-first exploration is used, but alternatives (coverage-aware RL, curiosity/novelty, uncertainty-driven exploration) and their impact on task diversity, difficulty, and downstream learning are unexamined.
- Modest cross-domain transfer for generalist base model: ForgeQwen3-8B improves MobileWorld GUI-only only from 7.6% to 10.3%, highlighting base-model dependence. What adaptation signals or curricula better support out-of-domain transfer for generalist VLMs remains open.
- No study of multilingual or localized UIs: robustness to non-English text, mixed-language apps, right-to-left layouts, and locale-specific formats is untested.
- Missing capabilities in known hard categories: game-playing, math/counting, and complex memory remain unsolved. Integrating specialized tools (e.g., OCR/math, programmatic planners, external memory) and measuring their effect within HiFPO is left open.
- Limited metric coverage: evaluation emphasizes Pass@k and single-attempt success. Efficiency (steps to success), safety violations, side effects, intervention frequency, and stability across seeds/sessions are not measured.
- On-policy learning vs. offline step-level GRPO: the training is conducted on logged, feedback-filtered steps with group-relative rewards, not in an online on-policy manner. The trade-offs between offline GRPO and on-policy RL (sample efficiency, stability, bias) are not explored.
- Sensitivity of key hyperparameters is unreported: the number of attempts per task (K), step budget estimation (B), group size (G), reward weights (ฮปtype/ฮปarg), and clipping/KL coefficients lack systematic sensitivity analysis.
- Data contamination safeguards are not documented: since adaptation and evaluation share the AndroidWorld ecosystem, explicit protocols ensuring no overlap or leakage (e.g., tasks, screens, or trajectories) should be clarified and tested.
- Compute and cost scalability are unclear: 900-task 8B runs took ~80 GPU-hours on 8ร80GB GPUs. The cost-quality frontier for larger curricula, models, and evaluators (especially proprietary LLMs) is not characterized.
- Reproducibility with fully open components is uncertain: the best ablations use proprietary Gemini as the decision model. Fully open, reproducible pipelines (open evaluators, open VLMs) and their performance gap remain to be established.
Practical Applications
Below is a concise mapping of practical applications that follow directly from the paperโs findings and innovations (MobileGym substrate, MobileGym-Curriculum/Critic, and HiFPO with hint-contextualized step-level GRPO). Each item states the use case, likely sectors, the tools/products/workflows that could emerge, and key assumptions or dependencies that affect feasibility.
Immediate Applications
- Annotation-free app-specific agent tuning for enterprise apps; sectors: software, enterprise IT
- Tools/products/workflows: โMobileForge-as-a-serviceโ that ingests an APK, explores it in an emulator/device farm (MobileGym), mines tasks, runs hint-guided rollouts (HiFPO), and outputs a tuned agent (e.g., ForgeOwl-8B-like checkpoint) and a reusable task suite.
- Assumptions/dependencies: Android emulator/device access, app ToS-compliant automation, evaluator availability (e.g., Gemini or open VLM), moderate training compute (e.g., 8ร80GB GPUs ~80 hours as reported).
- Self-healing mobile UI test generation and maintenance in CI/CD; sectors: software QA, DevOps
- Tools/products/workflows: CI job that explores nightly builds, mines trajectory-grounded tasks (MobileGym-Curriculum), runs multi-attempt tests with corrective hints, and reports regressions with step-level rationales (MobileGym-Critic).
- Assumptions/dependencies: Stable emulator/device farm; safe sandboxes to prevent destructive actions; permissioned test accounts.
- Repro-case bots for customer support and SRE; sectors: software, customer support
- Tools/products/workflows: Intake a natural-language issue, generate grounded attempts to reproduce in-app flows, and provide step-by-step visual traces plus evaluator rationales to engineers/support agents.
- Assumptions/dependencies: Access to the affected app version and environment; sanitized logs for privacy; evaluator reliability for outcome/process labels.
- Mobile RPA for repetitive single-app back-office tasks; sectors: operations, e-commerce, logistics, finance ops
- Tools/products/workflows: GUI agents that repeatedly perform app-specific form filling, settings updates, content tagging, and data collection with hint-guided robustness against small UI changes.
- Assumptions/dependencies: Narrow task scope (mostly single-app); secure credentials handling; change monitoring to avoid drift.
- Accessibility QA augmentation; sectors: accessibility, compliance
- Tools/products/workflows: Agents that explore screens, identify tappable targets, check label visibility/readability, and produce step-level feedback on confusing flows (e.g., verification, search, screen reading tags improved in the paper).
- Assumptions/dependencies: Evaluatorโs visual-language competence; mappings from process labels to WCAG-style checks.
- Functional coverage mapping and UX documentation; sectors: product, UX research, competitive intelligence
- Tools/products/workflows: Automatic โfunction atlasโ per app using exploration + curriculum grounding; generates human-readable flowcharts, walkthrough screenshots, and coverage summaries (e.g., discovery of shopping list, meal planner, settings in the study).
- Assumptions/dependencies: App exploration coverage; conservative rate limits to respect ToS and avoid anti-bot detection.
- Open-data generation for GUI agent research; sectors: academia, AI labs
- Tools/products/workflows: Reproducible pipelines to generate trajectory-grounded tasks, hierarchical feedback, and hint-contextualized steps for training/evaluation on new app sets; complements existing GUI benchmarks.
- Assumptions/dependencies: Licensing and redistribution constraints for apps/screens; evaluator reproducibility.
- Model selection and benchmarking with hierarchical feedback; sectors: academia, ML platform teams
- Tools/products/workflows: Replace single scalar rewards with MobileGym-Criticโs outcome/process/hint hierarchy to do finer-grained evals and guided ablations (e.g., decision model swaps, task filters).
- Assumptions/dependencies: Availability of a reliable step-description and decision model; standardization of JSON verdict schemas.
- Developer tooling in IDEs (e.g., Android Studio plugin); sectors: software engineering
- Tools/products/workflows: โGenerate Tasksโ and โTry Flowโ buttons that mine tasks from a local build, run hint-guided attempts, surface failing steps with on-screen overlays, and auto-generate regression tests.
- Assumptions/dependencies: Build-time emulator integration; deterministic app states (seeds, mocks).
- In-house fine-tuning of generalist VLMs for mobile UI competence; sectors: ML platform, enterprise IT
- Tools/products/workflows: Start from a generalist 8B VLM (e.g., Qwen3-VL-8B), run annotation-free adaptation on the enterpriseโs app suite, and ship a private, tuned checkpoint for internal automation.
- Assumptions/dependencies: Data governance and security; compute budget; evaluator choice (proprietary vs open).
- Educational labs for RL without human annotation; sectors: higher education, training
- Tools/products/workflows: Course labs where students use MobileGym+HiFPO to study curriculum grounding, hierarchical feedback, and group-relative optimization in GUI control.
- Assumptions/dependencies: Open-source availability (paper provides code/models/datasets links); campus compute resources.
- Simple personal workflow automation in emulators; sectors: daily life, prosumers
- Tools/products/workflows: Desktop companion that automates simple single-app routines (e.g., exporting data, clearing caches, updating preferences) through annotation-free adaptation in a sandboxed emulator before running the same steps on-device.
- Assumptions/dependencies: Transferability from emulator to user device UI; strict safety prompts; user consent and privacy safeguards.
Long-Term Applications
- Cross-app personal mobile assistants that execute end-to-end workflows; sectors: daily life, productivity
- Tools/products/workflows: Agents that book travel, file expenses, or manage subscriptions across multiple apps, reusing corrective hints to adapt to UI changes over time.
- Assumptions/dependencies: Robust multi-app reasoning (not solved in paper), persistent state handling, secure credential vaults, strong safety guardrails; policy notes that multi-app tasks remain challenging.
- Enterprise-grade mobile RPA platforms with continuous self-adaptation; sectors: enterprise IT, operations
- Tools/products/workflows: Fleet management for thousands of workflows where agents continuously explore app updates, mine new tasks, and self-correct via HiFPO; โself-healingโ automations with change reports.
- Assumptions/dependencies: Scalable device farms; automatic drift detection; monitoring/rollback; compliance approvals.
- Safety-certified evaluators and red-teaming frameworks for mobile agents; sectors: policy, security, compliance
- Tools/products/workflows: Standardized hierarchical critics that verify safety, privacy, and compliance constraints in addition to task success; audit logs with step-level rationales and corrective hints.
- Assumptions/dependencies: Third-party certification, adversarial testing, and formalized safety specifications; integration with MDM/enterprise policy engines.
- Government and healthcare app automation co-pilots; sectors: public services, healthcare
- Tools/products/workflows: Agents that assist case workers/patients navigating complex portals and apps; hint-guided robustness to frequent UI changes without retraining on human-annotated data.
- Assumptions/dependencies: Strong privacy (HIPAA/GDPR), strict sandboxing, offline/edge inference options, legal authorization for automated interactions.
- Accessibility co-pilots for end users (assistive control of arbitrary apps); sectors: accessibility, consumer tech
- Tools/products/workflows: On-device agents that interpret screens, propose safe actions, and execute with user confirmation; integrate corrective hints from prior failures to improve reliability for users with visual/motor impairments.
- Assumptions/dependencies: Very high reliability thresholds; local evaluators; explainability and easy undo; OS-level accessibility APIs.
- App-store-scale autonomous exploration and documentation; sectors: app analytics, developer relations
- Tools/products/workflows: Continuous crawling of large app catalogs to auto-generate task atlases, UX health metrics, and change logs; longitudinal studies of UI patterns.
- Assumptions/dependencies: Legal/ToS compliance, rate limiting, scalable storage of screenshots/trajectories, app-specific login handling.
- Closed-loop co-design of UI with agent feedback; sectors: product design, HCI
- Tools/products/workflows: Designers iterate on UI mockups while agents run MobileGym-style evaluations to surface failure-prone interactions; hints guide redesign for agent-and-human usability.
- Assumptions/dependencies: High-fidelity prototyping environments; bridging from mockups to interactive models; design tooling integration.
- Robotics-phone hybrids (robots manipulating smartphones as universal controllers); sectors: robotics, smart home
- Tools/products/workflows: Physical robots use screens as a universal interface (e.g., IoT apps), while MobileForge-derived policies provide robust, hint-accumulating UI control.
- Assumptions/dependencies: Reliable vision/actuation for touchscreens; latency management; safety; alignment between simulated GUI and physical device.
- Financial app testing and compliance bots; sectors: finance, fintech
- Tools/products/workflows: Automated KYC/consent flow checks, edge-case navigation, and multi-factor interactions with step-level audit trails; reduce manual QA burden during frequent releases.
- Assumptions/dependencies: Strict data handling; non-repudiation logs; regulator-acceptable evaluators; careful credential mockups.
- Healthcare mobile app validation and patient-pathway rehearsal; sectors: healthcare IT
- Tools/products/workflows: Agents validate appointment booking, lab result navigation, and medication reminders across app versions, surfacing step-level failures and corrective hints to engineering teams.
- Assumptions/dependencies: De-identified test environments; compliance audits; reliable evaluator calibrations.
- Curriculum and evaluator standards for GUI-agent research; sectors: academia, standards bodies
- Tools/products/workflows: Shared schemas for trajectory-grounded curricula, hierarchical feedback objects, and hint-contextualized optimization to ensure comparability across labs and models.
- Assumptions/dependencies: Community consensus; reference implementations; stable open benchmarks.
- Energy- and cost-aware adaptation pipelines; sectors: MLOps, sustainability
- Tools/products/workflows: Auto-scaling HiFPO loops that prioritize informative tasks/steps (as shown by task filtering and step selection) to reduce compute per unit performance gain.
- Assumptions/dependencies: Scheduling/pacing strategies; dynamic evaluator selection (open vs proprietary) based on budget; hardware availability.
- On-device, privacy-preserving adaptation; sectors: mobile OS, privacy tech
- Tools/products/workflows: HiFPO variants that run locally with compact evaluators and edge models to keep user data on-device while adapting to the userโs specific app versions and settings.
- Assumptions/dependencies: Efficient small evaluators; memory/compute constraints; federated or split-learning schemes; OS-level permissions.
Notes on feasibility across applications:
- The paper demonstrates strong in-domain gains and moderate out-of-domain transfer; multi-app workflows, games, and complex math/memorization remain open challenges.
- Evaluator quality materially affects outcomes; stronger critics help, but MobileForge still performs with weaker open models.
- Safety, privacy, and legal compliance (ToS, anti-bot policies) are critical for deployments beyond closed testbeds.
- Practical deployments depend on robust emulator/device farms, deterministic app states or mocks, and secure credential management.
Glossary
- Adaptive GUI action reward: A reward function that separately scores the correctness of an actionโs type and its arguments during policy optimization. "For parseable responses, we use an adaptive GUI action reward that separates action type from action arguments:"
- Agentic hierarchical evaluator: An evaluation setup where an agent-like process assesses rollouts at multiple levels (trajectory, step, hints), rather than relying on a learned scalar reward model. "MobileGym-Critic is implemented as an agentic hierarchical evaluator rather than a learned reward model."
- APK: Android application package; a file format that contains an Android appโs compiled code and metadata, including declared activities. "app-level structural anchors, such as activities declared in the APK, are combined with the current screenshot to generate goal-oriented exploration tasks."
- Clipped GRPO update: A variation of Group Relative Policy Optimization that uses clipping (akin to PPO-style clipping) to stabilize updates based on group-relative advantages. "HiFPO then applies a clipped GRPO update with KL regularization to a reference policy."
- Corrective hints: Compact feedback summarizing mistakes, what to avoid, and suggested alternatives, used to guide subsequent attempts and training prompts. "The corrective hint is generated when an attempt fails or contains unreasonable steps; it summarizes key mistakes, what to avoid, suggested alternatives, and important task insights."
- Curriculum prompt: A specialized prompt that evaluates explored trajectories and requests grounded, self-contained task variants for the curriculum. "The curriculum prompt jointly evaluates the explored trajectory and asks for self-contained task variants;"
- Depth-first traversal: An exploration strategy that follows one branch of possible interactions to its depth before backtracking to explore alternatives. "The explorer then pursues these goals with depth-first traversal, restoring parent states when branching and recording the resulting interactions."
- Empirical success rate: The observed fraction of attempts on a task that succeed, used to filter tasks before step extraction. "For each task, HiFPO computes the empirical success rate across attempts:"
- Exploration evidence: Recorded transitions and observations from interacting with target apps that ground task generation and evaluation. "It turns target-app interaction into three artifacts needed by policy learning: exploration evidence, executable tasks, and hierarchical evaluation of completed rollouts."
- GRPO: Group Relative Policy Optimization; a critic-free method that normalizes rewards within a group of responses to the same prompt and optimizes based on relative advantages. "GRPO was originally introduced as a critic-free policy optimization method that normalizes rewards within a group of responses to the same prompt"
- Group-relative advantage: The advantage computed by normalizing a responseโs reward relative to the mean and standard deviation of rewards within the same group. "The group-relative advantage is"
- Hierarchical feedback: Multi-level evaluation signals including trajectory outcomes, step-level labels, and corrective hints. "MobileGym-Critic produces hierarchical feedback:"
- Hint context: The aggregated corrective hints from earlier attempts, appended to later prompts to condition behavior and learning. "The hint context is appended to the task instruction before the next attempt,"
- Hint-guided multi-attempt rollout: A rollout protocol that serializes multiple attempts per task, each conditioned on accumulated corrective hints from prior attempts. "HiFPO first runs hint-guided multi-attempt rollouts over the MobileGym curriculum,"
- In-domain setting: Evaluation or adaptation within the same domain or app ecosystem used to generate training or adaptation data. "AndroidWorld~\citep{rawles2024androidworld} is the in-domain setting:"
- Importance ratio: The ratio of the new policyโs probability over the old policyโs probability for a sampled response, used in the update objective. "The response-level importance ratio is"
- KL regularization: A penalty term that constrains the updated policy to remain close to a reference policy via KullbackโLeibler divergence. "HiFPO then applies a clipped GRPO update with KL regularization to a reference policy."
- MobileGym: The interaction and evaluation substrate that explores apps, mines tasks, executes rollouts, and produces hierarchical feedback. "MobileGym is the interaction and evaluation substrate: it explores target apps, mines executable tasks, executes rollouts, and evaluates completed rollouts through a hierarchical critic."
- MobileGym-Critic: The hierarchical evaluator that returns trajectory outcomes, step-level process labels, and corrective hints for completed attempts. "MobileGym-Critic produces hierarchical feedback:"
- MobileGym-Curriculum: The component that converts exploration evidence into executable, app-grounded tasks with variations and prerequisites. "MobileGym-Curriculum converts exploration evidence into executable tasks."
- Out-of-domain setting: Evaluation on a different domain or app ecosystem than the one used for adaptation, testing transfer and generalization. "MobileWorld GUI-only~\citep{kong2026mobileworld} is the out-of-domain setting:"
- Pass@k: A success metric indicating whether a task is solved within k attempts, commonly reported as Pass@1, Pass@2, Pass@3. "evaluates on 116 AndroidWorld tasks with Pass@1, Pass@2, and Pass@3."
- Process label: A step-level evaluation tag indicating whether a step was reasonable and providing a rationale. "The step-level process label summarizes the local quality of step ."
- Reference policy: A baseline model used in KL regularization to keep the updated policy close in distribution. "HiFPO then applies a clipped GRPO update with KL regularization to a reference policy."
- Rollout: A sequence of decisions and actions executed by the policy to attempt a task, often analyzed post hoc for feedback. "A rollout attempt is the sequence"
- Rule-based GUI rewards: Hand-crafted reward functions for GUI actions used to score sampled responses during training without human labels. "by sampling multiple action responses and scoring them with rule-based GUI rewards"
- Step-level GRPO: Applying GRPO at the granularity of individual decision steps rather than full trajectories. "hint-contextualized step-level GRPO."
- Structured GUI action: A formal action representation consisting of a type (e.g., tap, swipe) and arguments (e.g., coordinates, text). "The policy emits a structured GUI action"
- Trajectory outcome label: The binary success/failure label for an entire attempt, used to decide task completion. "The trajectory outcome label is the final decision field:"
- UI affordances: Actionable elements or interaction possibilities exposed by the user interface that the agent can exploit. "reachable screens, UI affordances, and app functions,"
- VLM: Vision-LLM; a model that processes both images and text for perception and decision tasks. "a VLM describes the performed action and task-relevant screen evidence."
Collections
Sign up for free to add this paper to one or more collections.
