P-GUI-Evo: Benchmark for GUI Privacy Mediation

• P-GUI-Evo is a benchmark for context-sensitive privacy mediation in GUI agents that classifies observations as Allow, Mask, or Ask.
• It leverages real UI patterns, synthetic personas, and scenario variants to simulate privacy decisions under diverse lexical, structural, and visual shifts.
• The benchmark evaluates performance through metrics such as joint decision accuracy, Mask F1, and Ask recall with rigorous edge-side protocols.

P-GUI-Evo is a benchmark for personalized privacy arbitration in GUI agents. Introduced alongside MaskClaw, it formalizes privacy mediation not as generic personally identifiable information detection, nor as downstream task completion, but as a context-conditioned decision over whether the current GUI observation should be Allowed, Masked, or deferred via Ask before any raw screenshot leaves a trusted edge environment. Its construction uses real UI patterns, reconstructed HTML screens, sanitized labels, synthetic personas, and scenario variants designed to preserve privacy boundaries under lexical, structural, and visual changes (Zhao et al., 27 May 2026).

1. Formal decision problem

P-GUI-Evo models GUI privacy as a personalized classification problem. For interaction $i$,

$x_i = (I_i, q_i, a_i, c_i, p_i),$

where $I_i$ is the screenshot, $q_i$ the user instruction or task goal, $a_i$ the candidate GUI-agent intent or next action, $c_i$ the application and relational context, and $p_i$ the persona or policy context. The arbitrator predicts

$$y_i = f(x_i), \qquad \mathcal{Y}=\{\mathrm{Allow}, \mathrm{Mask}, \mathrm{Ask}\}.$$

The benchmark therefore targets policy arbitration over GUI observations, not leakage scoring and not action authorization (Zhao et al., 27 May 2026).

This formulation is explicitly context dependent. The same visible field can require different labels depending on persona, workflow, recipient boundary, and intended action. Static PII detectors are insufficient because they do not capture when a field is task-necessary and should remain visible, when it should be redacted, or when it requires explicit user confirmation. Cloud-side reasoning is also insufficient for the benchmark’s primary threat model, because it can expose the raw screenshot before deciding what should be protected.

The evaluator uses a conservative decision priority: $$\mathrm{Decision}(s)= \begin{cases} \text{Ask}, & u(s)=1,\ \text{Mask}, & u(s)=0 \wedge m(s)=1,\ \text{Allow}, & \text{otherwise}. \end{cases}$$ Here $u(s)$ is the confirmation variable and $x_i = (I_i, q_i, a_i, c_i, p_i),$0 the masking variable. This implements the rule that Ask overrides Mask, and Mask overrides Allow.

A central methodological choice is that downstream task success is not the scoring target. Nor is action authorization. The benchmark evaluates only the privacy decision over the current observation under the deployment-available context.

2. Corpus construction and benchmark organization

P-GUI-Evo contains 832 GUI privacy samples built from 296 normalized scenario cores, averaging 2.81 samples per core. The dataset is described as real-UI-pattern-derived and HTML-reconstructed. Construction proceeds through UI pattern abstraction, LLM-assisted scenario drafting, sensitive-field injection, HTML/UI rendering, and label correction and audit (Zhao et al., 27 May 2026).

The benchmark is explicitly sanitized. It uses placeholders, fake identities, or empty form states as surrogates for private content, and sensitive fields are injected as typed fake values. Real user logs, real chats, real medical records, payment credentials, and human-subject interactions are outside scope.

Aspect	Breakdown	Count
Samples	Active benchmark samples	832
Scenario cores	Normalized cores	296
Benchmark buckets	D1 / D2 / D3	174 / 546 / 112
Personas	User A / User B / User C	234 / 246 / 352
Decisions	Mask / Allow / Ask	438 / 314 / 80

The three personas are synthetic abstractions rather than real users. User A is associated with healthcare and medical workflow, User B with commerce and livestream selling, and User C with office and everyday services. This persona conditioning is integral rather than decorative: it is part of the decision boundary.

The benchmark is partitioned into three buckets. D1 basic contains 174 clean in-distribution cases. D2 generalization contains 546 UI or task variants. D3 stress contains 112 noisy boundary cases, including OCR noise, occlusion, low resolution, mixed languages, popup interference, and multiple sensitive regions.

Variant construction is also explicit. The benchmark includes 250 base screenshots, 201 structural/UI-shift variants, 208 visual-shift variants, and 173 lexical/semantic-shift variants. Among the 296 cores, 144 have at least one expanded variant. This design makes generalization part of the benchmark rather than an afterthought.

3. Annotation schema and evaluation protocol

Each sample includes a GUI screenshot, application platform, persona context, relationship context, user instruction, agent intent, benchmark bucket, and expected Allow/Mask/Ask decision. The construction and audit process checks plausibility, fluency, alignment with candidate agent intent, support for the expected decision, OCR detectability of visible sensitive evidence, and consistency between the rendered screenshot and scenario metadata (Zhao et al., 27 May 2026).

Sample-level PII annotations cover medical, address, financial, identity, contact, credential, compensation, supplier/customer, and security-related signals. The largest localizable evidence categories are medical, financial, contact, address/location, and account or identity credentials. Audit statistics report 715 samples with non-None PII type, 685 location-ready PII evidence samples, 1,529 localizable evidence boxes, and 736 L1/L3 usable samples.

The benchmark’s labels are interpreted operationally:

• Allow: no intervention; screenshot may be exposed.
• Mask: protected content must be redacted before exposure.
• Ask: pause and request user confirmation.

A policy-label consistency pass checks lower-level mask and confirmation fields before deriving the final label. This is important because the final class is not assigned independently of the underlying confirmation and masking conditions.

Reported metrics are joint decision accuracy, Mask F1, Ask recall, leak rate, over-protection, and raw-upload rate. Error notions include Leak for cases where expected Mask is predicted as non-Mask, OverProtect, and AskMiss. SafeScreenshot region mistakes are separated into L3 diagnostics, which makes localization quality analytically distinct from high-level policy classification.

4. Static benchmark behavior and empirical difficulty

P-GUI-Evo is designed to expose contextual privacy boundaries that pattern matching, generic cloud reasoning, and exposure routing do not reliably resolve. In the main benchmark table, all values are reported in $x_i = (I_i, q_i, a_i, c_i, p_i),$1. Policy-grounded MaskClaw reaches $x_i = (I_i, q_i, a_i, c_i, p_i),$2 accuracy, $x_i = (I_i, q_i, a_i, c_i, p_i),$3 Mask F1, $x_i = (I_i, q_i, a_i, c_i, p_i),$4 leak, $x_i = (I_i, q_i, a_i, c_i, p_i),$5 Ask recall, and $x_i = (I_i, q_i, a_i, c_i, p_i),$6 raw-upload rate; the paper reports that this improves over the strongest non-MaskClaw baseline, Static Regex, by 0.160 absolute accuracy, from 0.557 to 0.717 (Zhao et al., 27 May 2026).

Method	Acc.	Raw up.
Policy-grounded MaskClaw	$x_i = (I_i, q_i, a_i, c_i, p_i),$7	0
Static Regex	$x_i = (I_i, q_i, a_i, c_i, p_i),$8	0
Static Policy Corpus	$x_i = (I_i, q_i, a_i, c_i, p_i),$9	0
Cloud Minimal	$I_i$0	1000
Cloud Persona	$I_i$1	1000
Cloud Full-Context	$I_i$2	1000
EdgeClaw-ClawXRouter	$I_i$3	159

The benchmark reveals distinct failure modes. Static Regex is comparatively strong by accuracy because visible PII patterns matter, but it has Ask recall 0 and cannot represent consent-sensitive boundaries. Static Policy Corpus performs poorly because static memory without strong applicability judgment causes severe mismatch. Cloud Minimal, Cloud Persona, and especially Cloud Full-Context expose raw screenshots under the protocol and tend toward over-confirmation or boundary confusion.

The Cloud Full-Context result is especially diagnostic. Despite seeing raw screenshot, app, command, intent, recipient context, and persona, it attains only 0.166 accuracy and predicts Ask for 754/832 samples. Its confusion summary is:

• Allow: 44 Allow / 7 Mask / 263 Ask
• Mask: 2 Allow / 21 Mask / 415 Ask
• Ask: 3 Allow / 1 Mask / 76 Ask

This makes over-confirmation visible as a benchmark-specific pathology rather than a generic model weakness. EdgeClaw-ClawXRouter reduces raw cloud exposure to 15.9%, but routing alone does not resolve the core Allow/Mask/Ask decision boundary.

Error analysis on the 832-example benchmark reports 593 correct and 239 joint errors, with top categories:

• protected content allowed: 95,
• unnecessary masking: 58,
• confirmation replaced by masking: 37.

These three account for 79.5% of errors. Retrieval diagnostics report 239 wrong-boundary retrieval cases, 8 weak retrieval-evidence cases, and 585 neither. This suggests that, once obvious perception failures are separated out, the remaining difficulty is largely boundary confusion across persona, action, and recipient context.

5. Role within MaskClaw and edge-side mediation

Within MaskClaw, P-GUI-Evo serves as the common protocol for evaluating local perception and evidence extraction, retrieval-grounded policy arbitration, SafeScreenshot construction, and alternative baselines such as regex, cloud reasoning, and routing. The online arbitration loop extracts local evidence

$I_i$4

and consults policy memory

$I_i$5

where each rule is

$I_i$6

Here $I_i$7 is scope, $I_i$8 trigger evidence, $I_i$9 recommended action, $q_i$0 conflict-resolution priority, and $q_i$1 rationale (Zhao et al., 27 May 2026).

The paper states that ranking uses facet match and evidence compatibility, with recipient, action, and persona matches weighted above generic field-type matches. After arbitration, protected regions $q_i$2 are redacted to form the mediated screenshot

$q_i$3

Layer-wise diagnostics clarify how P-GUI-Evo decomposes the problem operationally. Table E1 reports:

• L1 OCR coverage: $q_i$4
• L2 Open policy accuracy: $q_i$5
• L2 Closed policy accuracy: $q_i$6
• L3 strict redaction: $q_i$7 strict no-flag rate

In $q_i$8 reporting, L2 Open yields Mask recall $q_i$9, Allow recall $a_i$0, and Ask recall $a_i$1; L2 Closed yields Mask recall $a_i$2, Allow recall $a_i$3, and Ask recall $a_i$4. The improvement from Open to Closed is attributed to retrieval-grounded arbitration and mediator control rather than perception alone. A plausible implication is that P-GUI-Evo is structured not only as a classification benchmark but as a layered evaluation of perception, policy retrieval, and redaction safety.

6. Behavior-driven skill evolution

Although the static benchmark uses fixed policy memory, the same work uses P-GUI-Evo’s scenario design philosophy to study how privacy behavior can evolve from user feedback. MaskClaw records traces

$a_i$5

extracts correction signals

$a_i$6

and updates memory through

$a_i$7

The five designed skill-evolution scenarios are iCloud cleanup, App permission, High-value transfer, Calendar merge, and Driving mode (Zhao et al., 27 May 2026).

These scenarios turn confirmations, cancellations, edits, rejected actions, retries, and explicit skill instructions into candidate reusable skills. Candidate updates are filtered through a schema/coverage gate, a fixed text scorer, and an LLM-Judge or sandbox gate. The objective is

$a_i$8

with

$a_i$9

The sandbox gate checks confirmation timing, safety rationale, and executable state flow for downstream agents. This is necessary because a textual skill can mention the correct privacy rule yet still be unsafe due to wrong action order, incomplete flow, or weak scenario alignment.

Experimental protocol uses 5 scenarios, 3 start conditions, 3 random seeds, and 20 evolution iterations. Full scenario behavior results report that evolved skills raise held-out behavior accuracy from:

• 37.50 to 100.00 in iCloud photo cleanup,
• 25.00 to 100.00 in App permission,
• 20.00 to 100.00 in High-value transfer,
• 12.50 to 100.00 in Calendar merge,
• 25.00 to 100.00 in Driving mode.

Unsafe action rates fall from:

• 62.50 to 12.50,
• 75.00 to 0.00,
• 80.00 to 0.00,
• 87.50 to 0.00,
• 75.00 to 0.00,

with macro averages moving from 24.00 base accuracy to 100.00 evolved accuracy, and from 76.00 base unsafe rate to 2.50 evolved unsafe rate. The paper also reports imperfect correction compliance in transfer and driving mode, which indicates that improved high-level behavior does not eliminate evidence and confirmation-timing failures.

7. Limitations and significance

P-GUI-Evo is intentionally controlled rather than naturalistic. It is sanitized, reconstructed from HTML/UI surrogates, based on synthetic personas rather than real users or organizations, and validated offline rather than through long-term live deployment. The paper is explicit that the skill-evolution results are scenario-level and not evidence of robust lifelong personalization (Zhao et al., 27 May 2026).

These limitations matter because they bound what can be claimed. The benchmark operationalizes a difficult and previously under-specified problem—context-sensitive screenshot exposure arbitration—but it does so under reconstructed conditions. This suggests strong value for controlled comparison, while leaving deployment realism as future work.

Its significance lies in the specificity of the target. P-GUI-Evo bridges GUI-agent evaluation and privacy evaluation by forcing systems to resolve a triage problem that depends jointly on screenshot content, task, persona, recipient, and agent intent. Pattern detectors tend to over-mask and cannot ask; cloud reasoning tends to over-confirm and uploads raw screenshots; routing reduces exposure but does not itself solve contextual arbitration. P-GUI-Evo makes these failure modes directly measurable.

In that sense, P-GUI-Evo is best understood as a benchmark for context-conditioned privacy mediation at the GUI surface. It turns the question “is there sensitive content on screen?” into the more demanding question “given the current user, task, recipient, and intended action, should this screenshot be allowed, masked, or held for confirmation before exposure?”