MobiFlow Benchmarking Suite Overview
- MobiFlow Benchmarking Suite is a framework that uses DAG-based milestone graphs to evaluate mobile agents under variable real-world conditions.
- It integrates online trace collection with offline replay and multi-level validation—including text, UI, and OCR checks—to robustly verify task execution.
- The suite supports scoring with full and partial credit while addressing issues like environmental variability and non-deterministic verification.
MobiFlow Benchmarking Suite is the benchmarking and evaluation component of the MobiAgent system, introduced as “a DAG-based benchmark framework for mobile agents” and described as a mobile benchmarking framework based on Directed Acyclic Graphs (DAGs) and milestone events. It is intended to evaluate GUI/XML-based mobile agents in real-world mobile environments, especially third-party applications, where environmental variability, multiple correct trajectories, and the lack of deterministic verification mechanisms make single-path or purely live evaluation brittle. Within the broader MobiAgent stack—comprising the MobiMind-series agent models, the AgentRR acceleration framework, and MobiFlow—MobiFlow functions as the evaluation substrate and empirical testbed, combining DAG-based task specification, multi-level verification, partial credit, and both online real-device traces and offline replay traces (Zhang et al., 30 Aug 2025).
1. Position within the MobiAgent framework
MobiFlow is presented as one of the three core components of MobiAgent: the MobiMind-series agent models, AgentRR, and the MobiFlow benchmarking suite. In that architecture, MobiFlow is the part used to assess whether an agent can complete realistic mobile tasks, particularly in third-party applications rather than only in system apps. Its role is evaluative rather than accelerative: AgentRR is aligned with record/replay acceleration, whereas MobiFlow is the framework used to score and compare agent behavior in real-world mobile scenarios (Zhang et al., 30 Aug 2025).
The benchmark is motivated by three properties of practical mobile environments. First, environmental variability arises from app versions, network conditions, recommendation feeds, and context-dependent interfaces. Second, many tasks admit multiple correct trajectories, so a single gold trace is too restrictive. Third, GUI tasks often lack deterministic verification mechanisms analogous to API-level completion signals. MobiFlow is introduced specifically to address those three constraints by defining multiple correct trajectories, using a multi-level verification mechanism, and supporting replay-based evaluation over pre-recorded static or offline traces (Zhang et al., 30 Aug 2025).
A common misconception is to treat MobiFlow as only a task list. Its intended function is broader: it is a benchmarking framework for specifying tasks as milestone graphs, validating whether execution traces satisfy those graphs, and assigning full or partial credit. This suggests an evaluation model centered on milestone satisfaction rather than exact trace imitation, although the paper does not formalize that interpretation mathematically.
2. DAG-based task model and benchmark architecture
MobiFlow’s architecture is described in four stages: Trace Collection, DAG-based Task Definition, Validation Engine, and Result Generation. Trace collection uses two forms of data. Online traces record real interactions between agent and phone, including actions and screenshots from UI pages. Offline traces store pre-defined task executions with recorded action primitives and XML data. Shared steps across different trajectories for the same task are merged, and offline traces are used to stabilize evaluation under dynamic environments (Zhang et al., 30 Aug 2025).
Each task is specified by a configuration file defining a DAG. In that representation, nodes are milestone events, such as key UI states or critical operations, while edges encode dependencies and action transitions between milestones. This is the core formalism that permits multiple acceptable execution paths. The framework also introduces AND and OR logic: AND requires all prerequisite milestone events to be satisfied, whereas OR requires any one of a set of milestone events to be satisfied. That logic allows sequential dependencies, branching alternatives, and conjunctions of milestone requirements to coexist in the same task specification (Zhang et al., 30 Aug 2025).
Task construction follows a template-based process. For each application, test case templates are defined with milestone events structured as a DAG, and three real-world test cases are generated for each type of template before formal testing. MobiFlow also supports task-definition templates with dynamic condition extraction from the task description. The paper’s example is “add milk to the shopping cart,” where the validator dynamically extracts “milk” as the key object and constructs the corresponding milestone nodes and conditions. The paper further states that tasks are designed with varying difficulty levels, from simple to complex, and that evaluation figures distinguish all tasks, easy tasks, and hard tasks, although no explicit rubric for difficulty assignment is given (Zhang et al., 30 Aug 2025).
3. Validation engine and scoring logic
The validation engine parses both the task DAG configuration and the agent’s execution trace, then traverses them together to check whether milestone events and dependencies appear in a way that satisfies the DAG. A fully validated path yields a full score; validation of only part of the milestone structure yields partial credit, according to task-specific criteria. The main quantitative reporting uses task completion score, also described as task completion rates, but the paper does not provide a formal mathematical scoring equation for MobiFlow (Zhang et al., 30 Aug 2025).
Supported checker types are summarized below.
| Checker type | Basis | Typical role |
|---|---|---|
| Text Matching | Specific strings or patterns in the UI | Lightweight verification |
| Regular Expression Matching | More complex text patterns | Flexible text validation |
| UI State Analysis | XML or view hierarchy state | Structured state checking |
| Icon Detection | Visual icon matching | Non-text UI verification |
| OCR Process | Text extracted from screenshots | Fallback when XML is unavailable |
| MLLM-as-a-Judge | Key frames and context | Holistic multimodal judgment |
The engine uses a progressive escalation strategy. It first tries the lightest and fastest checker. If text matching is needed, it first parses XML; only if XML is unavailable or plain-text checking is disabled does it escalate to OCR or LLM-based reasoning. For multi-path matching, MobiFlow uses a synchronized backtracking mechanism: when one DAG path cannot be validated, the checker backtracks to the previous branching milestone in the DAG and simultaneously backtracks in the trace to the corresponding point, then tries an alternative branch. Icon recognition is implemented with OpenCV using multi-scale matching and threshold-controlled similarity, intended to make icon matching robust to minor size or appearance changes (Zhang et al., 30 Aug 2025).
This validation design makes MobiFlow neither purely string-matching nor purely LLM-judged. It is a checker hierarchy that attempts structured validation first and only escalates to costlier or less rigid mechanisms when simpler signals are insufficient.
4. Task construction, scope, and evaluation protocol
MobiFlow targets mobile devices in real-world mobile environments, especially third-party applications. The paper states that the benchmark covers current widely-used mobile applications in China and names application categories including travel, shopping, entertainment, social networking, music and video, and food delivery. The benchmark section does not specify exact OS versions, device models, or whether both Android and iOS are supported, although the broader system discussion heavily references Android/XML dumps, making an Android-centered practical setting plausible rather than formally stated (Zhang et al., 30 Aug 2025).
The evaluation protocol operates per application. The authors define a set of DAG-based task templates, generate three real-world test cases per template, and execute agent models on those tasks under realistic workloads. All failed tasks are manually verified. If failures are caused by app anomalies, network interruptions, or non-existent tasks, testing is repeated. The evaluation also applies a termination penalty for tasks where the agent cannot complete or exit properly. This is significant because the paper highlights non-termination loops as an important behavioral difference among compared agents (Zhang et al., 30 Aug 2025).
Offline replay evaluation is the framework’s main control mechanism for dynamic environments. Correct action sequences are recorded, including multiple valid trajectories for the same task, and models can then replay these traces for evaluation. The advantage is reduced dynamic-environment noise and more deterministic comparison. The paper also states a clear limitation: offline-trace evaluation can produce false negatives when a model discovers a valid new trajectory not present in the offline set (Zhang et al., 30 Aug 2025).
The paper is notably incomplete on benchmark inventory statistics. It reports that three real-world test cases were generated for each type of template, but does not report the total number of benchmark tasks, total number of templates, number of apps, number of devices, OS versions, average or median number of interaction steps, exact difficulty-level counts, or formal train/validation/test split sizes. That omission is central to how MobiFlow should be interpreted: the paper describes a framework and instantiated benchmark process more fully than it quantifies the released benchmark inventory.
5. Baselines, metrics, and reported findings
The primary baselines compared on MobiFlow are GPT-5, Gemini-2.5 Pro, and UI-TARS-1.5-7B, with MobiAgent evaluated as the proposed system. The central reported result is qualitative and comparative: MobiAgent, specifically MobiMind-Decider-7B plus MobiMind-Grounder-3B, achieves the best task completion performance on MobiFlow across most applications, including strong performance on more complex tasks, particularly in shopping and food delivery. The text also attributes this advantage to better task understanding and decomposition, instruction following, exception handling, and task termination reliability, but the paper body does not provide exact numerical task completion scores from Figure 1(a)–(c) (Zhang et al., 30 Aug 2025).
One of the few explicit quantitative comparative findings concerns termination. GPT failed to properly terminate tasks in 11 application categories, Gemini had similar non-termination problems in 3 categories, and MobiAgent achieved correct task termination across all evaluated scenarios. The paper also reports a characteristic baseline behavior: GPT and Gemini often type object descriptions, types, departure time, or locations directly into search boxes. In applications with AI-powered search, that can bypass many interaction steps and sometimes improve completion rates, but it fails badly in applications that do not support AI search. MobiFlow’s diversity of app scenarios is used to expose precisely such strategy-dependent differences (Zhang et al., 30 Aug 2025).
The paper also reports closely related deployment results from AgentRR in realistic mobile scenarios associated with the benchmark setting. Under a uniform task distribution, AgentRR achieves a 30%–60% action replay rate; under a power law distribution where 80% of requests are concentrated in 20% of tasks, the replay rate rises to 60%–85%. Correctness of action replay exceeds 99% in test cases, and in complex scenarios such as food delivery and online shopping, AgentRR yields 2× to 3× average performance improvement over baselines. These results are not intrinsic MobiFlow metrics, but they situate the benchmark within the broader practical use of the MobiAgent system (Zhang et al., 30 Aug 2025).
6. Limitations and relation to adjacent benchmark frameworks
MobiFlow is explicitly presented as best suited for relative comparison rather than absolute leaderboards. Because applications and environments evolve, the paper cautions that absolute MobiFlow scores are mainly for reference, while relative comparisons across agent models are more reliable. Additional stated limitations include dependence on manual verification for failed tasks, false negatives in offline replay when novel valid paths are absent from the recorded set, and the absence of formal benchmark splits and comprehensive inventory statistics (Zhang et al., 30 Aug 2025).
Within the mobile-agent benchmarking landscape, MobiFlow occupies a distinct position. Mobile-Bench emphasizes hybrid UI+API control, starts tasks from the HOME page, includes 103 APIs, and introduces the CheckPoint metric for process-aware scoring of sequential actions (Deng et al., 2024). ColorBench later describes MobiFlow succinctly as modeling tasks as directed acyclic graphs for evaluation purpose, while extending graph-based benchmarking toward a strongly connected directed graph over finite mobile screen states for complex long-horizon tasks with atomic capability analysis (Song et al., 16 Oct 2025). MobiBench, in turn, addresses two limitations that it attributes to existing benchmarks—including single-path offline evaluation and monolithic agent scoring—by proposing a modular and multi path aware offline framework; it reports 94.72 percent agreement with human evaluators and each step has on average 2.95 valid actions (Im et al., 14 Dec 2025).
These comparisons clarify MobiFlow’s historical and technical niche. It is not an API-augmented emulator benchmark in the style of Mobile-Bench, nor a finite-state quasi-dynamic interaction graph like ColorBench, nor a module-standardized offline replay framework like MobiBench. Its defining contribution is the combination of DAG-based milestone specification, multi-level verification, partial-credit scoring, and replay-capable evaluation for third-party mobile applications under realistic but unstable conditions. In that sense, MobiFlow is best understood as a benchmarking framework plus a real-world task suite aimed at fairer evaluation of mobile GUI agents when exact-path supervision is too rigid and fully live evaluation is too unstable.
