KnowU-Bench: Benchmark for Mobile Personalization

• KnowU-Bench is an online benchmark for personalized mobile assistance that evaluates agents’ ability to infer preferences from behavioral logs and interactive cues.
• The benchmark uniquely hides explicit user profiles, requiring agents to derive user preferences solely from observed behavior and clarification dialogues.
• It employs a reproducible Android emulation environment and hybrid scoring methods to assess both execution quality and decision-theoretic calibration.

KnowU-Bench is an online benchmark for personalized mobile agents that evaluates whether a mobile assistant can infer user preferences from history and interaction, ask clarifying questions when needed, proactively intervene only when appropriate, respect user consent and stop after rejection, and carry out personalized mobile tasks correctly in a real GUI environment (Chen et al., 9 Apr 2026). Its defining methodological choice is to hide the user profile from the agent and expose only behavioral logs, so that preference inference must arise from observed behavior or interaction rather than direct profile lookup. The benchmark is built on a reproducible Android emulation environment and is intended to measure not only interface operation, but also the decision-theoretic and interactional prerequisites of trustworthy personal assistance.

1. Problem formulation and benchmark target

KnowU-Bench was introduced in response to a gap in prior mobile-agent evaluation: many earlier benchmarks measure GUI execution under clear instructions, whereas real assistants are expected to operate under vague requests, hidden preferences, routine-based proactive assistance, and consent-sensitive decisions (Chen et al., 9 Apr 2026). The paper distinguishes between being competent at interface manipulation and being competent at personal assistance. In this formulation, a system may be effective at clicking through an application while remaining unreliable about what to do, for whom, and when.

This benchmark therefore reframes personalized mobile assistance as a composite capability spanning three dimensions. First, an agent must recover preferences from behavioral traces and dialogue rather than static context. Second, it must determine when ambiguity warrants clarification. Third, it must calibrate proactivity: whether to intervene, seek consent, remain silent, or stop after rejection. The paper’s central claim is that these are execution-grounded questions, not merely offline intent-prediction problems (Chen et al., 9 Apr 2026).

A common simplification is to treat personalization as a retrieval problem over explicit preferences. KnowU-Bench rejects that assumption by design. The agent receives behavioral logs rather than the latent profile itself, and proactive tasks remove explicit instructions altogether. This makes personalization a partially observed inference-and-action problem rather than a context lookup task. The benchmark’s broader significance lies in showing that personalized assistance requires a tighter coupling between user modeling, dialogue policy, and grounded GUI control than prior evaluation suites typically enforce.

2. Task composition and Android execution substrate

KnowU-Bench contains 192 tasks total, divided into 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks (Chen et al., 9 Apr 2026). The three splits are functionally distinct rather than being mere difficulty tiers.

Task type	Count	Primary requirement
General GUI tasks	42	Explicit instruction following without personalization
Personalized tasks	86	Preference inference from logs or clarification dialogue
Proactive tasks	64	Intervention, consent, or restraint without explicit instruction

General tasks use explicit instructions with no personalization requirement and measure ordinary GUI execution in isolation. Personalized tasks are intentionally ambiguous and require the agent to infer user preferences from logs or clarify with the user simulator; the paper gives “Order me a light lunch” as an example (Chen et al., 9 Apr 2026). Proactive tasks omit explicit instructions entirely, so the agent must infer from current time, location, and user routine whether to act immediately, ask the user, or stay silent.

The execution environment is a containerized Android emulator built around a rooted Pixel 8 AVD with a FastAPI orchestration server (Chen et al., 9 Apr 2026). Reproducibility is enforced operationally rather than informally: each task starts from a fixed emulator snapshot, resets transient state including backend processes, callback files, and interaction history, and may override device time for time-sensitive tasks. A unified controller maps agent outputs to executable ADB operations across the full lifecycle from initialization to evaluation. These design decisions matter because they reduce variance from environmental drift and make live mobile evaluation more repeatable than many prior online benchmarks.

Compared with MobileWorld, the benchmark expands the app ecosystem to 23 applications and explicitly adds service apps for preference-sensitive tasks, including jingdian and Taodian for shopping, and chilemei and tuantuan for food delivery (Chen et al., 9 Apr 2026). This app coverage supports evaluation of shopping platform preference, payment habit, delivery address preference, food choices, and cross-platform ordering routines. The benchmark is therefore centered on daily-service and commerce scenarios in which preference sensitivity is operational rather than decorative.

3. Hidden profiles, behavioral logs, and interactive user simulation

A major architectural feature of KnowU-Bench is the asymmetry between the hidden user profile and the exposed behavioral history. The hidden profile $P$ contains structured fields summarized as

$$\mathcal{F}_{\mathrm{profile}} = \{ identity,\ locations,\ digital\_context,\ habits,\ preferences,\ decision\_criteria,\ social\_graph \},$$

including identity information, home and work locations, device and digital settings, routines and habits, stable preferences, decision criteria or tradeoffs, and social contacts and communication norms (Chen et al., 9 Apr 2026). The GUI agent does not receive this profile.

Instead, the agent receives only exposed behavioral logs $H$, represented as free-text traces of past user actions. Each log entry is defined as

$$\ell_i = \{ time, location, action, label, category \},$$

and the logs are linearized into text of the form

$$\mathrm{fmt}(\ell_i) = [\ell_i.time]\;(\ell_i.location)\;\ell_i.action.$$

Importantly, the agent does not directly see label or category, which prevents reliance on explicit annotations (Chen et al., 9 Apr 2026). This is a central anti-leakage mechanism: the agent must infer, for example, a preference for Tuantuan, a “no peanuts” constraint, or a routine-based trigger from behavior rather than metadata.

The benchmark also introduces controlled noise. Clean logs contain only signal, whereas noisy logs inject about 25% extra irrelevant events, including entertainment activity, accidental interactions, advertisements, scam messages, and other distractors (Chen et al., 9 Apr 2026). This allows evaluation of whether preference inference is robust to the clutter that characterizes realistic mobile traces.

To support multi-turn preference elicitation and proactive consent handling, KnowU-Bench instantiates an LLM-driven user simulator $\pi_u$. When the agent invokes ask_user, the simulator produces a reply according to

$r_t \sim \pi_u(\cdot \mid m_t, P, S),$

where $m_t$ is the agent’s clarification message, $P$ is the hidden profile, and $S$ is the current environment state (Chen et al., 9 Apr 2026). The simulator is thus conditioned on the latent person model and the ongoing interaction context, enabling realistic clarification replies, preference disclosure during personalization tasks, and accept or reject decisions during proactive tasks.

In proactive tasks, the decision space is explicitly triadic: direct execution, ask the user for confirmation, or remain silent (Chen et al., 9 Apr 2026). If the agent asks, the simulator returns explicit accept or reject signals, and the benchmark checks whether the assistant proceeds appropriately or remains restrained after rejection. This operationalizes consent not as a textual nicety, but as a branch in the action policy with downstream execution consequences.

4. Formalization and hybrid evaluation protocol

The paper frames mobile automation as a POMDP,

$$\mathcal{F}_{\mathrm{profile}} = \{ identity,\ locations,\ digital\_context,\ habits,\ preferences,\ decision\_criteria,\ social\_graph \},$$0

with $$\mathcal{F}_{\mathrm{profile}} = \{ identity,\ locations,\ digital\_context,\ habits,\ preferences,\ decision\_criteria,\ social\_graph \},$$1 denoting environment state, $$\mathcal{F}_{\mathrm{profile}} = \{ identity,\ locations,\ digital\_context,\ habits,\ preferences,\ decision\_criteria,\ social\_graph \},$$2 observations including instruction and screenshots, $$\mathcal{F}_{\mathrm{profile}} = \{ identity,\ locations,\ digital\_context,\ habits,\ preferences,\ decision\_criteria,\ social\_graph \},$$3 mobile UI actions, $$\mathcal{F}_{\mathrm{profile}} = \{ identity,\ locations,\ digital\_context,\ habits,\ preferences,\ decision\_criteria,\ social\_graph \},$$4 the transition function, and $$\mathcal{F}_{\mathrm{profile}} = \{ identity,\ locations,\ digital\_context,\ habits,\ preferences,\ decision\_criteria,\ social\_graph \},$$5 the reward or task completion indicator (Chen et al., 9 Apr 2026). Agent actions are modeled as

$$\mathcal{F}_{\mathrm{profile}} = \{ identity,\ locations,\ digital\_context,\ habits,\ preferences,\ decision\_criteria,\ social\_graph \},$$6

where $$\mathcal{F}_{\mathrm{profile}} = \{ identity,\ locations,\ digital\_context,\ habits,\ preferences,\ decision\_criteria,\ social\_graph \},$$7 is the user instruction, $$\mathcal{F}_{\mathrm{profile}} = \{ identity,\ locations,\ digital\_context,\ habits,\ preferences,\ decision\_criteria,\ social\_graph \},$$8 the screenshot, $$\mathcal{F}_{\mathrm{profile}} = \{ identity,\ locations,\ digital\_context,\ habits,\ preferences,\ decision\_criteria,\ social\_graph \},$$9 the past interaction history, $H$0 the exposed user logs, $H$1 the current state, and $H$2 optional feedback, especially from ask_user (Chen et al., 9 Apr 2026). This formalization makes personalization and proactivity first-class inputs to the control policy rather than side channels.

Evaluation is hybrid. Rule-based verification performs deterministic checks on concrete outcomes such as correct recipient, correct event or order creation, correct alarm or setting configuration, valid time window, and safety violations after rejection (Chen et al., 9 Apr 2026). The LLM-as-a-judge component evaluates semantic aspects including preference alignment, trade-off quality, communication style, contextual appropriateness, and clarification quality. The combined task score is defined as

$H$3

A task with $H$4 is fully deterministic, $H$5 is purely semantic, and intermediate values are used for preference-dependent personalized tasks (Chen et al., 9 Apr 2026).

The benchmark further defines $H$6 as the task score, $H$7 as the binary success indicator, $H$8 as the number of actions, and $H$9 as the number of clarification questions (Chen et al., 9 Apr 2026). General tasks measure task success rate and execution efficiency. Personalized tasks measure success rate, average score, and interaction efficiency, with

$$\ell_i = \{ time, location, action, label, category \},$$0

Proactive tasks measure Act rate, Silent rate, and Stop rate, corresponding respectively to intervention when intervention is warranted, restraint when intervention is unnecessary, and stopping after explicit rejection (Chen et al., 9 Apr 2026).

This protocol is notable because it does not reduce evaluation to terminal task success. It also scores whether the agent intervened under the right conditions, negotiated consent appropriately, and used clarification economically rather than indiscriminately. A plausible implication is that KnowU-Bench treats decision quality and execution quality as separable but interacting components of mobile assistance.

5. Empirical results and failure structure

The reported experiments cover 11 models, including GUI-specific models, general open-source models, and closed-source models (Chen et al., 9 Apr 2026). The aggregate pattern is consistent across families: models perform well on general tasks but drop sharply on personalized and proactive tasks. Even strong models fall below 50% on vague or proactive settings, including Claude Sonnet 4.6, and the paper reports roughly 30% average degradation once tasks require personalization or proactivity (Chen et al., 9 Apr 2026).

Claude Sonnet 4.6 achieves the best overall success rate at 60.4%, followed by Seed 2.0 Pro at 51.6% and Gemini 3.1 Pro Preview at 44.3% (Chen et al., 9 Apr 2026). On hard personalized tasks, Claude Sonnet 4.6 reaches 44.2% success rate, while open-source models remain below 12%. The paper argues that the bottleneck is not general GUI navigation, but preference acquisition and intervention calibration.

The personalized-task failure taxonomy makes this point quantitatively explicit. For Claude Sonnet 4.6, failures are distributed as Clarify 66.7%, Partial 27.1%, GUI 4.2%, and Preference 2.1% (Chen et al., 9 Apr 2026). The dominant failure mode is therefore insufficient clarification or inadequate integration of elicited preferences, not low-level interface manipulation. The paper further notes that asking more questions is not enough: Claude Sonnet 4.6 performs best while asking relatively few questions, whereas Seed 2.0 Pro asks more questions but performs worse (Chen et al., 9 Apr 2026). This result argues against equating personalization quality with clarification frequency.

The proactive-task failure analysis likewise emphasizes policy rather than mechanics. For Claude Sonnet 4.6, the failure breakdown is 60.0% Intervention errors, 20.0% Passive errors, 15.0% GUI errors, and 5.0% Rejection errors (Chen et al., 9 Apr 2026). Most failures thus arise from unwarranted action, missed warranted action, or poor calibration of intervention boundaries. The benchmark’s interpretation is that proactive assistance is primarily a calibration problem rather than an execution problem.

The paper also reports memory ablation results over full history, retrieved log snippets, clean logs, and noisy logs. The main conclusion is that the best memory interface is model-dependent: retrieval can help some models but hurt others, and noisy retrieval can destabilize weaker models (Chen et al., 9 Apr 2026). This suggests that better memory access is necessary but not uniformly solved by retrieving more text. Finally, evaluator validation on 26 fixed trajectories rated by four human experts shows that the hybrid evaluator has lower mean absolute error and aligns more closely with human judgment than pure rule-based scoring (Chen et al., 9 Apr 2026).

6. Relation to adjacent benchmarks, limitations, and implications

KnowU-Bench occupies a distinct position in the benchmark landscape. NC-Bench evaluates conversational competence through sequence management patterns such as answering, repair, closing, aborting, RAG-grounded response, and complex request management; it is therefore a strong reference for interactional mechanics, but it does not center live mobile GUI execution, hidden preference inference, or proactive consent calibration (Moore et al., 10 Jan 2026). KnowMe-Bench studies person understanding for lifelong digital companions using autobiographical narratives, flashback-aware temporal realignment, and evidence-linked questions about factual recall, subjective state attribution, and principle-level reasoning; that work is adjacent in its emphasis on user modeling, but it targets narrative person understanding rather than online mobile assistance (Wu et al., 8 Jan 2026). UQABench evaluates whether compact user embeddings can serve as soft prompts for personalized question answering, focusing on sequence understanding, action prediction, and interest perception; it is therefore relevant to the representation of behavioral history, but not to grounded GUI control or intervention policy (Liu et al., 26 Feb 2025). KWBench, by contrast, measures unprompted problem recognition in professional knowledge work, emphasizing framing before execution; a plausible implication is that its notion of latent structure recognition is conceptually related to KnowU-Bench’s hidden-preference inference, even though the deployment setting is entirely different (Maloo, 17 Apr 2026).

Within mobile-agent evaluation itself, KnowU-Bench differs from earlier benchmarks by being online and execution-grounded, by forcing interactive preference elicitation, and by evaluating the full proactive decision chain from intervention choice to consent handling to post-rejection restraint (Chen et al., 9 Apr 2026). The benchmark therefore links personalization research to embodied, policy-sensitive mobile automation rather than to offline prediction or static preference recovery.

The paper also identifies future directions, including stronger long-term memory access, better ambiguity-resolution policies, safer proactive decision boundaries, improved compositional preference modeling, and more robust handling of noisy logs and retrieval (Chen et al., 9 Apr 2026). At the same time, a practical limitation implicit in the benchmark design is that it still relies on synthetic or curated user profiles and simulated dialogues. This suggests that KnowU-Bench should be understood as a controlled approximation of personalized assistance rather than a full substitute for field deployment.

Its main scientific contribution is to show that current agents are substantially better at operating interfaces than at understanding the person behind the interface (Chen et al., 9 Apr 2026). In that sense, KnowU-Bench shifts evaluation from the question of whether an agent can execute a task to whether it can act as the right assistant for the right user at the right time.