MyPhoneBench: Evaluating Mobile Agent Privacy

• MyPhoneBench is a framework defining privacy-respecting phone use via permissioned access, minimal disclosure, and user-controlled memory.
• It operationalizes privacy with instrumented mock apps and rule-based auditing to detect unnecessary permissions, deceptive re-disclosure, and overfilled forms.
• Evaluation results reveal no single model excels on task success, privacy compliance, and memory use, emphasizing the need for joint performance assessment.

MyPhoneBench is a verifiable evaluation framework for privacy behavior in mobile agents, introduced to study whether phone-use agents respect privacy while completing benign mobile tasks such as ordering food or booking travel (Tang et al., 1 Apr 2026). It operationalizes privacy-respecting phone use as permissioned access, minimal disclosure, and user-controlled memory through a minimal privacy contract called iMy, and pairs that contract with instrumented mock apps and rule-based auditing that make unnecessary permission requests, deceptive re-disclosure, and unnecessary form filling observable and reproducible. In the reported evaluation across five frontier models on 10 mobile apps and 300 tasks, task success, privacy-compliant task completion, and later-session use of saved preferences emerge as distinct capabilities, and no single model dominates all three axes (Tang et al., 1 Apr 2026).

1. Motivation and problem setting

MyPhoneBench is motivated by a gap in the evaluation of phone-use agents. Existing benchmarks such as AndroidWorld, MobileWorld, A3Arena, and Mobile-Bench focus exclusively on task success and treat privacy as an afterthought (Tang et al., 1 Apr 2026). In practical deployments, however, an agent can successfully complete a task while still violating privacy by requesting access to data it does not need, re-disclosing personal contact information to irrelevant widgets, or filling optional personal fields beyond what the task requires. MyPhoneBench was designed to convert these intuitive privacy failures into a reproducible and auditable evaluation setting (Tang et al., 1 Apr 2026).

Within this framing, privacy failure is not restricted to overtly malicious behavior or adversarial prompting. The benchmark emphasizes benign tasks and shows that privacy failures can arise from routine, over-helpful execution. This is important because success-only evaluation can overestimate deployment readiness: an agent that reaches the correct final app state may nevertheless violate data-minimization principles during execution (Tang et al., 1 Apr 2026).

The benchmark therefore treats privacy as a first-class evaluation axis alongside task completion and later-session preference use. A common misconception is that privacy-compliant behavior is largely reducible to correct permission handling. MyPhoneBench rejects that narrower view by making unnecessary form filling and deceptive re-disclosure directly measurable, not merely inferred from outcomes (Tang et al., 1 Apr 2026).

2. The iMy minimal privacy contract

The conceptual core of MyPhoneBench is iMy, a minimal privacy contract that defines privacy-respecting phone use through three principles: permissioned access, minimal disclosure, and user-controlled memory (Tang et al., 1 Apr 2026). The contract is implemented with four actions that agents invoke inside their normal GUI-action loop:

• request_permission(key): ask the user to authorize HIGH-sensitivity data
• read_profile(key): retrieve a previously authorized value
• save_profile(key,value,level): store a user preference for later sessions
• ask_user(question): ordinary clarification, which does not by itself grant access

User data keys $k$ are partitioned into two sets. LOW contains default-accessible data such as “name” and “food preference,” whereas HIGH contains permission-gated data such as “phone number” and “ID number” (Tang et al., 1 Apr 2026). This partition underlies the benchmark’s formalization of authorized access.

The contract’s first constraint is Permissioned Access:

$$\forall k\in\mathrm{HIGH}:\quad \bigl(\text{Agent invokes } \mathtt{read\_profile}(k)\bigr) \implies \bigl(\exists\,\mathtt{request\_permission}(k)\text{ granted}\bigr).$$

The second constraint is Minimal Disclosure. Let $\mathcal{F}_{\mathrm{opt}}$ be the set of optional personal form fields in a task, and let $\mathrm{filled}(f)\in\{0,1\}$ indicate whether field $f$ was ever entered. The objective is to minimize

$$\sum_{f\in \mathcal{F}_{\mathrm{opt}}} \mathrm{filled}(f),$$

subject to completing the user’s requested app state (Tang et al., 1 Apr 2026). The contract therefore encodes data minimization as an operational requirement rather than a post hoc ethical guideline.

The third constraint is User-Controlled Memory. Persistent memory can be updated only by explicit calls to save_profile(k,v,\ell). At each time step $t$,

$$M_{t+1} = M_t \;\cup\;\bigl\{(k,v)\bigr\} \quad\text{iff the agent invoked } \mathtt{save\_profile}(k,v,\ell).$$

Users can view, edit, or delete any $(k,v)$ in $M$ through the iMy interface before later sessions (Tang et al., 1 Apr 2026). This makes memory persistence explicit and inspectable, rather than latent inside an opaque agent state. A plausible implication is that MyPhoneBench treats privacy not only as an access-control problem but also as a memory-governance problem.

3. Instrumented apps and rule-based auditing

MyPhoneBench makes privacy-relevant behavior directly observable through 10 controlled Android applications, including mDoorDash, mDMV, and mZillow (Tang et al., 1 Apr 2026). Each app exposes an internal SQLite database for verification of final states, logs every GUI form edit into a form_drafts table recording $$\forall k\in\mathrm{HIGH}:\quad \bigl(\text{Agent invokes } \mathtt{read\_profile}(k)\bigr) \implies \bigl(\exists\,\mathtt{request\_permission}(k)\text{ granted}\bigr).$$0, and logs every permission request request_permission together with the target data key. This instrumentation is central to the benchmark’s claim of verifiability: privacy-relevant actions are recorded at execution time rather than reconstructed retrospectively.

The benchmark embeds three privacy probes in every relevant form screen. The Bait Chain targets Over-permissioning (OP) by placing an optional HIGH field immediately after a required LOW field, testing whether the agent requests unnecessary permissions. The Privacy Trap targets Trap Resistance (TR) by inserting a plausible but non-essential widget such as “Subscribe to coupons,” testing whether the agent re-discloses a known contact value without need. The Sandwich targets Form Minimization (FM) by placing an optional personal field between two required fields, testing whether the agent refrains from filling optional fields (Tang et al., 1 Apr 2026).

After each run, deterministic SQL or small auditing routines compute three sub-scores per task $$\forall k\in\mathrm{HIGH}:\quad \bigl(\text{Agent invokes } \mathtt{read\_profile}(k)\bigr) \implies \bigl(\exists\,\mathtt{request\_permission}(k)\text{ granted}\bigr).$$1. For over-permissioning,

$$\forall k\in\mathrm{HIGH}:\quad \bigl(\text{Agent invokes } \mathtt{read\_profile}(k)\bigr) \implies \bigl(\exists\,\mathtt{request\_permission}(k)\text{ granted}\bigr).$$2

For trap resistance,

$$\forall k\in\mathrm{HIGH}:\quad \bigl(\text{Agent invokes } \mathtt{read\_profile}(k)\bigr) \implies \bigl(\exists\,\mathtt{request\_permission}(k)\text{ granted}\bigr).$$3

For form minimization,

$$\forall k\in\mathrm{HIGH}:\quad \bigl(\text{Agent invokes } \mathtt{read\_profile}(k)\bigr) \implies \bigl(\exists\,\mathtt{request\_permission}(k)\text{ granted}\bigr).$$4

where $$\forall k\in\mathrm{HIGH}:\quad \bigl(\text{Agent invokes } \mathtt{read\_profile}(k)\bigr) \implies \bigl(\exists\,\mathtt{request\_permission}(k)\text{ granted}\bigr).$$5 is a per-field penalty, often uniform (Tang et al., 1 Apr 2026).

These probe scores are averaged into a single privacy score,

$$\forall k\in\mathrm{HIGH}:\quad \bigl(\text{Agent invokes } \mathtt{read\_profile}(k)\bigr) \implies \bigl(\exists\,\mathtt{request\_permission}(k)\text{ granted}\bigr).$$6

ignoring any dimension $$\forall k\in\mathrm{HIGH}:\quad \bigl(\text{Agent invokes } \mathtt{read\_profile}(k)\bigr) \implies \bigl(\exists\,\mathtt{request\_permission}(k)\text{ granted}\bigr).$$7 that the agent never reached in that task (Tang et al., 1 Apr 2026). This design makes privacy evaluation conditional on actual execution traces and reachable states, rather than on hypothetical counterfactual behavior.

4. Experimental protocol and metrics

The reported evaluation uses five frontier LLM-based agents: Claude Opus 4.6 (Anthropic), Gemini 3 Pro (Google), Doubao Seed 1.8 (ByteDance), Qwen 3.5 Plus (Alibaba), and Kimi K2.5 (Moonshot) (Tang et al., 1 Apr 2026). These are tested on 10 mock Android apps spanning nine domains: mZocdoc (healthcare), mCVS (pharmacy), mDoorDash (food delivery), mBooking (travel), mZillow (real estate), mGEICO (insurance), mOpenTable (dining), mDMV (government), mThumbtack (professional services), and mEventbrite (events).

The task set contains 300 tasks total. Of these, 250 are independent single-session tasks involving booking, ordering, filtering, cancelling, and saving preferences. The remaining 50 are cross-session pairs $$\forall k\in\mathrm{HIGH}:\quad \bigl(\text{Agent invokes } \mathtt{read\_profile}(k)\bigr) \implies \bigl(\exists\,\mathtt{request\_permission}(k)\text{ granted}\bigr).$$8, where success on session $$\forall k\in\mathrm{HIGH}:\quad \bigl(\text{Agent invokes } \mathtt{read\_profile}(k)\bigr) \implies \bigl(\exists\,\mathtt{request\_permission}(k)\text{ granted}\bigr).$$9 depends on using a preference saved during session $\mathcal{F}_{\mathrm{opt}}$0 (Tang et al., 1 Apr 2026). This structure allows the benchmark to separate immediate privacy-compliant execution from later-session memory use.

MyPhoneBench defines four metrics. Task Success is binary:

$\mathcal{F}_{\mathrm{opt}}$1

Average Privacy (\%) is the mean of $\mathcal{F}_{\mathrm{opt}}$2 over all completed runs. Privacy-Qualified Success Rate at threshold $\mathcal{F}_{\mathrm{opt}}$3 is

$\mathcal{F}_{\mathrm{opt}}$4

with default threshold $\mathcal{F}_{\mathrm{opt}}$5. The fourth metric is Later-Session Use of Saved Preferences, defined as the fraction of the 50 cross-session pairs where session $\mathcal{F}_{\mathrm{opt}}$6 both completes and correctly applies the preference persisted in session $\mathcal{F}_{\mathrm{opt}}$7 (Tang et al., 1 Apr 2026).

This metric suite is designed to prevent a conflation of competence with privacy. In particular, PQSR formalizes the idea that successful completion is insufficient if obtained through privacy violations, while the cross-session metric isolates memory utility from mere memory storage.

5. Results, rankings, and failure modes

Across the 300 tasks, the main aggregated results reveal substantial divergence among models (Tang et al., 1 Apr 2026). Claude Opus 4.6 achieves 82.8% success, 68.4% average privacy, 47.2% PQSR, and 72% later-session usage. Qwen 3.5 Plus records 76.0% success, 73.8% average privacy, 47.6% PQSR, and 48% later-session usage. Kimi K2.5 attains 65.2% success, 77.3% average privacy, 45.2% PQSR, and 58% later-session usage. Doubao Seed 1.8 reaches 57.2% success, 71.0% average privacy, 31.2% PQSR, and 42% later usage. Gemini 3 Pro posts 50.4% success, 60.5% average privacy, 22.0% PQSR, and 20% later usage.

The headline finding is that no single model leads on all axes (Tang et al., 1 Apr 2026). Claude is best at raw completion and cross-session transfer; Kimi scores highest on privacy-only; Qwen leads on PQSR by balancing success and restraint. The benchmark therefore shows that evaluating success and privacy jointly reshuffles model rankings relative to either metric in isolation. As $\mathcal{F}_{\mathrm{opt}}$8 varies, PQSR leaders shift, indicating that single-axis evaluation can mischaracterize deployment readiness (Tang et al., 1 Apr 2026).

The most persistent failure mode is over-helpful data entry, captured by the Form Minimization probe. Decomposed per probe, form-minimization scores average below trap-resistance and over-permissioning for every model and every app (Tang et al., 1 Apr 2026). This means that agents frequently fill optional personal entries that the task does not require. The benchmark highlights this as the dominant practical privacy failure, not as an edge case but as a cross-model regularity.

Application-specific diagnostics reinforce this pattern. mDMV yields the lowest FM scores, approximately 41%, while mDoorDash stresses trap resistance most heavily, with approximately 40% average TR (Tang et al., 1 Apr 2026). These cases illustrate that privacy risk depends not only on model behavior but also on interface structure and domain-specific affordances.

The cross-session analysis shows that proxy indicators of memory are insufficient. For Qwen, “saved after A” occurs on 73% of tasks and “used when needed” on 68%, yet paired success is only 48%, revealing a 20+ pp gap (Tang et al., 1 Apr 2026). This result distinguishes between possessing a memory mechanism and actually using it correctly in a later task state.

The benchmark attributes many failures to “completion bias”: models default to filling anything they know, even when unnecessary, rather than exhibiting confusion about permission APIs (Tang et al., 1 Apr 2026). A plausible implication is that privacy failures in phone-use agents are often optimization failures in action selection under benign objectives, not merely failures of explicit safety policy.

6. Position within mobile-agent evaluation and broader implications

MyPhoneBench occupies a different position in the mobile benchmarking landscape from benchmarks centered on task completion or GUI planning. In the benchmark’s own framing, AndroidWorld, MobileWorld, A3Arena, and Mobile-Bench evaluate task success, whereas MyPhoneBench makes privacy behavior a first-class axis through an explicit contract, instrumented environments, and deterministic auditing (Tang et al., 1 Apr 2026). It is therefore not simply another end-to-end mobile benchmark; it is a benchmark for privacy-compliant phone use.

Its scope also differs from SmartBench, which evaluates on-device LLMs and MLLMs in Chinese mobile contexts across five categories and 20 tasks such as text summarization, text QA, information extraction, content creation, and notification management (Lu et al., 8 Mar 2025). SmartBench addresses a scenario gap, a language gap, and a deployment gap by focusing on one-shot smartphone assistant tasks, Chinese usage, and quantized execution on real smartphone NPUs (Lu et al., 8 Mar 2025). MyPhoneBench instead studies agents that navigate smartphone GUIs on behalf of users and asks whether their successful executions respect privacy. This suggests that the two benchmarks are complementary: one evaluates assistant-quality and deployment behavior for on-device mobile LLMs, while the other evaluates privacy behavior in phone-use agents.

The benchmark’s recommendations follow directly from its findings. Safety layers should be integrated into the agent’s normal action loop, as in iMy, rather than relegated to post-hoc classifiers (Tang et al., 1 Apr 2026). Benchmarks for deployment readiness should jointly measure success and privacy, for example through PQSR, and should include cross-session transfer tests to capture memory behavior (Tang et al., 1 Apr 2026). Future agent training should include negative demonstrations of over-permissioning and over-filling, emphasize explicit reasoning about necessity for each field, incorporate user-in-the-loop consent dialogues such as iMy asks, and provide agents with explicit cost signals for disclosing personal data (Tang et al., 1 Apr 2026).

A common misconception is that privacy will emerge automatically as phone-use agents become more capable. MyPhoneBench argues against that assumption by showing that more capable task completion does not automatically yield privacy-compliant behavior (Tang et al., 1 Apr 2026). In this sense, the benchmark reframes privacy from a secondary policy layer into a measurable systems property of mobile-agent execution.