Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization

Abstract: Mobile GUI agents powered by Multimodal LLMs (MLLMs) can execute complex tasks on mobile devices. Despite this progress, most existing systems still optimize task success or efficiency, neglecting users' privacy personalization. In this paper, we study the often-overlooked problem of agent personalization. We observe that personalization can induce systematic structural heterogeneity in execution trajectories. For example, privacy-first users often prefer protective actions, e.g., refusing permissions, logging out, and minimizing exposure, leading to logically different execution trajectories from utility-first users. Such variable-length and structurally different trajectories make standard preference optimization unstable and less informative. To address this issue, we propose Trajectory Induced Preference Optimization (TIPO), which uses preference-intensity weighting to emphasize key privacy-related steps and padding gating to suppress alignment noise. Results on our Privacy Preference Dataset show that TIPO improves persona alignment and distinction while preserving strong task executability, achieving 65.60% SR, 46.22 Compliance, and 66.67% PD, outperforming existing optimization methods across various GUI tasks. The code and dataset will be publicly released at https://github.com/Zhixin-L/TIPO.

• The paper proposes the TIPO framework that optimizes trajectory-level preference alignment for mobile GUI agents considering user privacy personas.
• It introduces preference-intensity weighting and padding gating to robustly align privacy-first and utility-first agent trajectories.
• Experimental results show TIPO outperforms baselines, maintaining task feasibility while enhancing privacy-centric behavior.

Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization

Introduction and Motivation

Personalization in Mobile GUI agents, especially with regard to privacy, has become a salient but understudied challenge as MLLM-powered systems transition from laboratory prototypes to deployed assistants on heterogeneous mobile tasks. The central observation in this work is that user privacy preferences induce structural heterogeneity in agent trajectories: privacy-first users manifest protective and defensive actions (e.g., minimizing exposure, denying permissions), whereas utility-first users favor more direct, less restrictive pathways. This divergence cannot be fully addressed by generic preference optimization approaches, as these typically presume homogeneity and aligned length in candidate trajectories. The "Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization" paper introduces an explicit paradigm—trajectory-level optimization conditioned on user privacy personas—culminating in the TIPO framework (2604.11259).

Figure 1: Personalized trajectory selection for GUI agents—different privacy personas condition branches, yielding distinct execution trajectories under identical goals and contexts.

Trajectory-Level Preference Alignment: Problem Definition

Personalization is formalized as trajectory selection, where for each task, the agent must produce an executable trajectory aligning with the target privacy preference. Formally, given the tuple $x = (g, o, h, p)$ encoding the task goal, current UI, interaction history, and user persona, the agent generates $y = (y_1, \ldots, y_{|y|})$. Privacy preference $p$ is operationalized in binary—Privacy-first versus Utility-first—each associated with corresponding reference trajectories. The optimization then targets maximizing the likelihood ratio of persona-aligned over misaligned trajectories for the same context:

$\pi_\theta(y^+|x) > \pi_\theta(y^-|x)$

where $y^+$ is strongly aligned with the specified privacy persona, and $y^-$ is not—even though both are functionally feasible.

Method: Trajectory Induced Preference Optimization (TIPO)

Baseline and Limitations

DPO and kin methods presume that preference pairs are well-aligned at the action level. However, privacy personas induce variable-length, structurally divergent plans—protective actions cannot be linearly matched to utility-centric steps. The necessary padding for alignment (e.g., "no_action" placeholders) introduces noise that DPO uncritically incorporates into objective terms, and uniform weighting across all positions dilutes the rare but critical privacy-relevant step signals.

Figure 2: Persona-induced trajectory divergence: Utility-first and Privacy-first agents diverge at privacy-sensitive decision points, yielding different action sequences despite shared task goals.

TIPO: Preference-Intensity Weighting + Padding Gating

TIPO reformulates the step-level preference signal by decomposing trajectory-level comparisons into action-aligned pairs. For each action step, the preference signal $z_t(\theta)$ is weighted by the “intensity” of its preference relevance:

$$\alpha_t = \text{clip}\left(\frac{\Delta s_t}{\Delta_{\max}}, 0, 1\right)^\gamma$$

where $\Delta s_t$ quantifies the annotated persona relevance differential and $\gamma$ sharpens focus on high-impact steps.

Padding gating is applied by masking out positions corresponding to alignment-induced placeholders. Only semantically meaningful action pairs contribute to the loss:

$y = (y_1, \ldots, y_{|y|})$0

where $y = (y_1, \ldots, y_{|y|})$1 nullifies padded positions. The resulting TIPO loss is a softplus over the valid, weighted preference signals.

Figure 3: Comparison of step-DPO and TIPO—TIPO emphasizes persona-critical steps via preference-intensity weighting and filters alignment noise through padding gating.

Privacy Preference Dataset Construction

A novel Privacy Preference dataset is curated, covering eight high-frequency mobile domains (shopping, browsing, food delivery, accounts, etc.) with two agent personas (Privacy-first and Utility-first) role-played per task instance. Collection is rigorously controlled—aided by ADB instrumentation and human annotators following persona-specific protocols but disallowed from inputting actual personal information. Key steps and trajectory differences are annotated, semantically aligned, and placeholders inserted to synchronize lengths for valid optimization.

Figure 4: Overview of Privacy Preference Dataset construction—tasks are role-played under both privacy personas, trajectories collected, and preference pairs aligned for trajectory-level supervision.

Experimental Results

Quantitative Results and Metrics

Evaluation blends trajectory executability, persona adherence, and persona distinction. TIPO outperforms all baselines across dimensions, notably yielding:

• Overall SR: 65.60% (highest)
• Compliance: 46.22
• Persona Distinction: 66.67%
• PAS-S (Privacy-first adherence): 42.85
• PAS-U (Utility-first adherence): 49.59

Notably, TIPO maintains task feasibility while significantly enhancing persona alignment—critical privacy-related behaviors are more robustly learned without undermining success rates.

Figure 5: Radar chart (higher-is-better) of overall SR, PAS-S (Privacy-first adherence), PAS-U (Utility-first adherence), Compliance, and PD—showing TIPO's superior trade-off between executability and personalization.

Ablation Analysis

Removing preference-intensity weighting or padding gating individually degrades Compliance and Persona Distinction, confirming their orthogonal contributions. The joint application is critical for stable, generalizable trajectory-level preference alignment under structural heterogeneity.

Figure 6: Ablation analysis—TIPO (full model) outperforms step-DPO, TIPO without weighting, and TIPO without padding gating across persona-related metrics.

Category-wise Generalization

TIPO's performance is consistent across browsing/content interaction, account/file management, and transactional task types. It generalizes in both local privacy-critical divergences and more global, structural persona-driven shifts.

Implications and Future Directions

Trajectory-level optimization, as instantiated in TIPO, is inherently extensible beyond privacy to other user-contingent objectives—e.g., cost-sensitivity, accessibility, efficiency trade-offs. TIPO establishes a scalable framework for integrating explicit user personas in Mobile GUI agent platforms. Practically, it supplies a mechanism for agent developers to enforce user trust and satisfaction, as agents can now be tailored not merely to task completion, but to process-level user expectations.

Limitations: TIPO is layered atop existing MLLM-based agents; it does not enhance underlying task understanding or low-level planning, but augments alignment with user-specific process requirements. Improved collection and annotation of nuanced trajectory divergences, alongside richer persona taxonomies, represent promising directions.

Conclusion

This work formally introduces trajectory-level persona alignment for Mobile GUI agents, identifying and addressing the limitations of generic preference optimization in the face of persona-induced structural heterogeneity. TIPO, integrating preference-intensity weighting and padding gating, establishes new benchmarks on the first explicit privacy preference dataset for GUI agents. This trajectory-focused methodology paves the way for practical, user-aware agent personalization, with broader applicability across preference dimensions in real-world AI assistants.