ClawMobile: Autonomous LLM-based Smartphone Control

• ClawMobile is a smartphone-native agentic system that integrates LLM-based reasoning with structured, deterministic control tailored for Android’s resource limitations.
• It employs a hierarchical runtime architecture combining deterministic backends, semantic UI agents, and fallback gestures to manage dynamic app states and fragmented UIs.
• Empirical evaluations demonstrate 100% task completion by leveraging explicit state verification and adaptive backend scheduling to optimize reliability under mobile constraints.

Searching arXiv for the specified ClawMobile paper and closely related mobile-agent work to ground the article. ClawMobile is a smartphone-native agentic system for LLM–based autonomy on real Android devices. It is introduced as a case study in how to compose probabilistic language reasoning with structured, deterministic control pathways under mobile-specific constraints such as limited CPU, memory, battery, fragmented UI surfaces, dynamic app states, permission dialogs, backgrounding, and network variability (Du et al., 26 Feb 2026). In this formulation, the smartphone is modeled as a finite but very large state space $S$ and an action space $$A = A_{\text{det}} \cup A_{\text{ui}} \cup A_{\text{fallback}}$$, where deterministic backends, semantic UI-agent primitives, and lower-level UI gestures are explicitly separated. ClawMobile’s central claim is that reliable smartphone-native autonomy requires principled coordination between high-level planning and deterministic system interfaces rather than a monolithic end-to-end UI policy (Du et al., 26 Feb 2026).

1. Problem Setting and Formalization

ClawMobile is motivated by the observation that smartphones present a “uniquely harsh environment” for autonomous agents driven by LLMs (Du et al., 26 Feb 2026). The stated sources of difficulty are resource constraints, fragmented and rapidly changing UI surfaces, and dynamic app states together with OS-level interruptions. These factors jointly make repeated LLM calls, large context payloads, and purely UI-driven control brittle and expensive.

The formalization given for the mobile environment is explicit. The state space is

$$S = \{ s \mid s \text{ encodes (foreground app, UI tree, permission flags, network status, clipboard contents, …)} \},$$

and the action space is partitioned into three modalities:

• $A_{\text{det}}$: deterministic, structured backends such as ADB commands and Termux API calls.
• $A_{\text{ui}}$: semantic UI-agent primitives such as “find field labeled X and type Y”.
• $A_{\text{fallback}}$: lower-level UI gestures such as tap at $(x,y)$, swipe, and long-press (Du et al., 26 Feb 2026).

A smartphone-native agent then executes a trajectory $\tau=(s_0,a_0,s_1,\dots,s_T)$ by alternating reasoning and acting. The explicit optimization target is high success and reproducibility under UI drift and mobile interruptions while also bounding latency and token usage. This framing places ClawMobile within finite-horizon planning, but the practical emphasis is not on dense reward engineering; it is on execution stability under partial observability, volatile interfaces, and constrained runtime budgets. A plausible implication is that ClawMobile treats mobile autonomy less as pure sequential decision-making and more as runtime systems engineering around a planner.

2. Hierarchical Runtime Architecture

ClawMobile adopts a three-tiered hierarchical design consisting of a Chat Channel, an Agent Orchestrator, Control Backends, and a Memory component linked to the smartphone device state (Du et al., 26 Feb 2026). The architectural separation is the defining systems decision: language reasoning is isolated in the orchestrator, while execution is delegated to pluggable modules with explicit return codes and structured results.

The Agent Orchestrator runs an LLM in a control loop: observe state $s_t$, select a backend and action $a_t$, execute, observe $$A = A_{\text{det}} \cup A_{\text{ui}} \cup A_{\text{fallback}}$$0, and repeat. Its objective is cast as

$$A = A_{\text{det}} \cup A_{\text{ui}} \cup A_{\text{fallback}}$$1

where $$A = A_{\text{det}} \cup A_{\text{ui}} \cup A_{\text{fallback}}$$2 is sparse, for example $$A = A_{\text{det}} \cup A_{\text{ui}} \cup A_{\text{fallback}}$$3 on task completion and $$A = A_{\text{det}} \cup A_{\text{ui}} \cup A_{\text{fallback}}$$4 otherwise (Du et al., 26 Feb 2026). The sparse-reward formulation is notable because it underscores that ClawMobile is not presented as a classical RL-trained mobile policy; rather, the LLM planner is embedded in a runtime that obtains reliability through backend design and verification.

Control Backends are described as pluggable modules exposing deterministic APIs, semantic UI agents, or direct UI actions. Each backend returns a structured result $$A = A_{\text{det}} \cup A_{\text{ui}} \cup A_{\text{fallback}}$$5, including success or failure codes, extracted text, and screenshots, so that the orchestrator does not infer success from LLM hallucination alone (Du et al., 26 Feb 2026). This explicit contract between planner and executor is one of the key mechanisms by which the system improves reproducibility.

Memory is implemented as a lightweight key–value store containing mobile-specific knowledge, previous successful workflows (“skills”), and user preferences. Examples include package-name mappings such as “to open Notes on Android 16, use pkg=com.google.android.keep.” Memory biases backend choice and can be retrieved by the orchestrator as context (Du et al., 26 Feb 2026). In systems terms, this makes Memory a runtime substrate for both retrieval augmentation and execution prior selection, rather than merely a conversation log.

3. Execution Modalities and Scheduling Logic

ClawMobile explicitly distinguishes three execution modalities. The first is deterministic backends, the second is UI agents, and the third is direct UI control as a last resort (Du et al., 26 Feb 2026). This decomposition is central to how the runtime orders reliability against latency and token cost.

Deterministic backends include ADB system commands and Termux API calls. The examples given are adb shell settings put system ui_night_mode 2 to turn on dark theme and termux-sms-send --number “+123…” --message “Hello”. These pathways are described as having known semantics and sub-millisecond latency (Du et al., 26 Feb 2026). Their role is to absorb actions whose meaning can be guaranteed independently of a volatile UI surface.

UI-agent execution is based on DroidRun’s program-synthesis approach. Given a UI tree, the UI agent generates a short program that sequences find-by-text, tap, and type instructions. The paper states that reliability is higher than raw taps but slower than $$A = A_{\text{det}} \cup A_{\text{ui}} \cup A_{\text{fallback}}$$6 (Du et al., 26 Feb 2026). This places semantic UI control between deterministic APIs and coordinate-level gestures.

Direct UI control comprises primitive gestures such as tap(x,y) and swipe(x_1,y_1 \to x_2,y_2), and it is “treated as last resort” (Du et al., 26 Feb 2026). In practical terms, this means ClawMobile does not treat pixel-level action as the default representation of mobile interaction.

The orchestrator’s scheduling loop is given procedurally. It queries device state, generates a plan with the LLM, chooses $$A = A_{\text{det}} \cup A_{\text{ui}} \cup A_{\text{fallback}}$$7 if Memory has an API for the operation, otherwise chooses $$A = A_{\text{det}} \cup A_{\text{ui}} \cup A_{\text{fallback}}$$8 if the operation requires UI semantics, and otherwise falls back to $$A = A_{\text{det}} \cup A_{\text{ui}} \cup A_{\text{fallback}}$$9. After execution, it re-queries device state and, if verification fails, triggers replanning on the current context (Du et al., 26 Feb 2026). The important point is not merely conditional backend dispatch; it is that dispatch is always embedded in an explicit verify-and-replan cycle. This suggests that ClawMobile’s operational semantics are closer to guarded command execution than to free-running chain-of-thought acting.

4. Implementation on Android

The runtime is reported to run entirely on-device and was tested on a Google Pixel 9 running Android 16 (Du et al., 26 Feb 2026). The Agent Orchestrator is built atop OpenClaw’s WebSocket-based gateway, but hosted locally. LLM inference with GPT-5.2 runs remotely via a secure channel, while a local “model stub” handles token-efficient context caching (Du et al., 26 Feb 2026). This implementation choice is precise: the orchestration loop is local, but heavy inference is remote.

The control backends are heterogeneous. ADB commands are issued via the Android Debug Bridge API; Termux API commands are invoked through the application’s HTTP interface; and the DroidRun UI agent is compiled into an Android service exposing a protobuf-based tool interface (Du et al., 26 Feb 2026). These details matter because they indicate that ClawMobile is not a single-model mobile assistant but a multi-backend runtime integrating pre-existing Android control channels.

Resource management is explicit. The orchestrator tracks token usage, caps UI-agent calls per minute, and falls back to deterministic pathways when LLM latency or cost exceed thresholds (Du et al., 26 Feb 2026). Stability safeguards are also explicit: every backend call returns a bounded result code; failure triggers re-verification loops such as “is Chrome really in the foreground?”; and permission prompts are handled by a small toolkit of pre-wired ADB scripts stored in Memory (Du et al., 26 Feb 2026). These mechanisms define the runtime as resource-aware and failure-aware at the systems level.

The open-source repository is organized into /orchestrator, /backends/adb, /backends/termux, /backends/droidrun, /memory, and /examples (Du et al., 26 Feb 2026). The repository layout reveals the project’s modularity: planning, backend adapters, memory assets, and example task scripts are distinct components. The paper also states that the implementation is open-sourced at github.com/ClawMobile/ClawMobile (Du et al., 26 Feb 2026).

5. Empirical Evaluation

ClawMobile is evaluated against two baselines: DroidRun running tethered to a PC and ClawMobile without its UI-agent, denoted CM-w/o-DR (Du et al., 26 Feb 2026). The evaluation uses six real-world tasks, each run 10 times. The reported tasks span Settings, Chrome, Play Store, YouTube, and a multi-app workflow.

App	Task	DR / CM-w/o-DR / CM
Settings	Turn on dark theme	100% / 100% / 100%
Chrome	Search “gold price USD”	73% / 100% / 100%
Play Store	Install “RedNote”	33% / 100% / 100%
YouTube (A)	Play “Bad Habits” MV, close ad	100% / 80% / 100%
YouTube (B)	Open comments, post “I love this song”	85% / 50% / 100%
Multi-App	Search PL match results, summarize in Notes	73% / 100% / 100%

The corresponding end-to-end times in seconds are: Settings 26 / 22 / 21; Chrome 26 / 67 / 67; Play Store 29 / 232 / 117; YouTube (A) 66 / 600 / 88; YouTube (B) 121 / 600 / 235; Multi-App 60 / 219 / 145 (Du et al., 26 Feb 2026).

The summary claim is that ClawMobile achieves 100% completion on all tasks, whereas DroidRun and CM-w/o-DR fail on 2–4 tasks (Du et al., 26 Feb 2026). Mean latency of ClawMobile is reported as 91 s, compared with 58 s for DroidRun, and this increase is attributed to verify-and-recover loops while remaining “within acceptable bounds for high-value mobile tasks” (Du et al., 26 Feb 2026).

A more specific qualitative finding is given for the cross-app search-and-summarization case: ClawMobile “never stalls on app launches or ambiguous UI targets because each step is gated by explicit state checks and re-planning when expectations mismatch reality” (Du et al., 26 Feb 2026). Within the logic of the paper, this is the key empirical argument for the architecture: slower execution is accepted in exchange for higher completion and lower brittleness under ambiguous or changing mobile UI conditions.

6. Design Principles, Related Systems, and Scope

Three design principles are explicitly distilled from the system and its evaluation. First, decoupled reasoning versus control: the LLM should remain focused on high-level planning, while guaranteed semantics are delegated to deterministic backends. Second, reproducibility via explicit verification: after every action, the device state is queried rather than trusting probabilistic reasoning. Third, resource-aware hybrid scheduling: backend selection is treated as an online scheduling problem over cost and reliability, defaulting to fast deterministic pathways and falling back to UI agents only when needed (Du et al., 26 Feb 2026).

The paper further states that efficiency, adaptability, and stability form interlocking research axes, and that future systems should jointly optimize token budgets, skill abstractions, persistent memory hierarchies, and privacy-aware state handling (Du et al., 26 Feb 2026). This indicates that ClawMobile is presented not only as an implementation but also as a design-space probe for mobile LLM runtimes.

A useful point of comparison is X-OmniClaw, which describes a unified mobile agent organized around Omni Perception, Omni Memory, and Omni Action, with multimodal ingress, temporal alignment, hybrid grounding, and behavior cloning plus trajectory replay in the Android ecosystem (Ren et al., 7 May 2026). ClawMobile’s architecture is more explicitly framed around a separation between a high-level LLM planner and deterministic control pathways, whereas X-OmniClaw emphasizes a single on-device “agent loop” coupling perception, memory, and action (Ren et al., 7 May 2026). This suggests two neighboring architectural interpretations of smartphone-native agency: one centered on verified control decomposition, the other on unified multimodal ingress and replay-driven interaction.

The term “ClawMobile” may also invite lexical confusion with robotic claw research. In arXiv usage, however, “ClawMobile” in (Du et al., 26 Feb 2026) refers to a smartphone-native agentic runtime, not a mechanical gripper or avian-inspired end-effector such as the UAV perching and walking system described in “Avian-Inspired Claws Enable Robot Perching or Walking” (Askari et al., 2023). The overlap is nominal rather than substantive.

7. Extensions and Research Directions

The repository documentation in the source material identifies three extension paths. New backends can be plugged in by adding a tool descriptor under /backends, with examples including iOS Shortcuts and home-automation APIs. New skill abstractions can be defined in Memory for repeated cross-app workflows. On-device LLM inference can be explored through integrations such as FlexInfer to remove remote-call latency (Du et al., 26 Feb 2026). These are concrete extension mechanisms rather than speculative desiderata.

Within the broader technical narrative, ClawMobile is best understood as a hierarchical runtime for smartphone automation in which deterministic APIs, semantic UI agents, explicit state verification, and LLM planning are composed under one control loop (Du et al., 26 Feb 2026). Its contribution lies less in proposing a novel standalone model than in specifying the control-theoretic and systems interfaces required for reproducible mobile autonomy. A plausible implication is that future smartphone-native agents will need to inherit this emphasis on backend contracts, verification, and resource-aware scheduling even when they adopt richer multimodal perception or more capable on-device models.