---
title: 'P-GUI-Evo: Benchmark for GUI Privacy Mediation'
url: https://www.emergentmind.com/topics/p-gui-evo
type: topic
---

# P-GUI-Evo: Benchmark for GUI Privacy Mediation

P-GUI-Evo is a benchmark for personalized privacy arbitration in GUI agents. Introduced alongside MaskClaw, it formalizes privacy mediation not as generic personally identifiable information detection, nor as downstream task completion, but as a context-conditioned decision over whether the current GUI observation should be **Allow**ed, **Mask**ed, or deferred via **Ask** before any raw screenshot leaves a trusted edge environment. Its construction uses real UI patterns, reconstructed HTML screens, sanitized labels, synthetic personas, and scenario variants designed to preserve privacy boundaries under lexical, structural, and visual changes [2605.28646].

## 1. Formal decision problem

P-GUI-Evo models GUI privacy as a personalized classification problem. For interaction \(i\),
\[
x_i = (I_i, q_i, a_i, c_i, p_i),
\]
where \(I_i\) is the screenshot, \(q_i\) the user instruction or task goal, \(a_i\) the candidate GUI-agent intent or next action, \(c_i\) the application and relational context, and \(p_i\) the persona or policy context. The arbitrator predicts
\[
y_i = f(x_i), \qquad \mathcal{Y}=\{\mathrm{Allow}, \mathrm{Mask}, \mathrm{Ask}\}.
\]
The benchmark therefore targets policy arbitration over GUI observations, not leakage scoring and not action authorization [2605.28646].

This formulation is explicitly context dependent. The same visible field can require different labels depending on persona, workflow, recipient boundary, and intended action. Static PII detectors are insufficient because they do not capture when a field is task-necessary and should remain visible, when it should be redacted, or when it requires explicit user confirmation. Cloud-side reasoning is also insufficient for the benchmark’s primary threat model, because it can expose the raw screenshot before deciding what should be protected.

The evaluator uses a conservative decision priority:
\[
\mathrm{Decision}(s)=
\begin{cases}
\text{Ask}, & u(s)=1,\\
\text{Mask}, & u(s)=0 \wedge m(s)=1,\\
\text{Allow}, & \text{otherwise}.
\end{cases}
\]
Here \(u(s)\) is the confirmation variable and \(m(s)\) the masking variable. This implements the rule that **Ask overrides Mask**, and **Mask overrides Allow**.

A central methodological choice is that downstream task success is not the scoring target. Nor is action authorization. The benchmark evaluates only the privacy decision over the current observation under the deployment-available context.

## 2. Corpus construction and benchmark organization

P-GUI-Evo contains **832 GUI privacy samples** built from **296 normalized scenario cores**, averaging **2.81 samples per core**. The dataset is described as **real-UI-pattern-derived** and **HTML-reconstructed**. Construction proceeds through UI pattern abstraction, LLM-assisted scenario drafting, sensitive-field injection, HTML/UI rendering, and label correction and audit [2605.28646].

The benchmark is explicitly sanitized. It uses placeholders, fake identities, or empty form states as surrogates for private content, and sensitive fields are injected as typed fake values. Real user logs, real chats, real medical records, payment credentials, and human-subject interactions are outside scope.

| Aspect | Breakdown | Count |
|---|---|---:|
| Samples | Active benchmark samples | 832 |
| Scenario cores | Normalized cores | 296 |
| Benchmark buckets | D1 / D2 / D3 | 174 / 546 / 112 |
| Personas | User A / User B / User C | 234 / 246 / 352 |
| Decisions | Mask / Allow / Ask | 438 / 314 / 80 |

The three personas are synthetic abstractions rather than real users. **User A** is associated with healthcare and medical workflow, **User B** with commerce and livestream selling, and **User C** with office and everyday services. This persona conditioning is integral rather than decorative: it is part of the decision boundary.

The benchmark is partitioned into three buckets. **D1 basic** contains 174 clean in-distribution cases. **D2 generalization** contains 546 UI or task variants. **D3 stress** contains 112 noisy boundary cases, including OCR noise, occlusion, low resolution, mixed languages, popup interference, and multiple sensitive regions.

Variant construction is also explicit. The benchmark includes **250** base screenshots, **201** structural/UI-shift variants, **208** visual-shift variants, and **173** lexical/semantic-shift variants. Among the 296 cores, **144** have at least one expanded variant. This design makes generalization part of the benchmark rather than an afterthought.

## 3. Annotation schema and evaluation protocol

Each sample includes a GUI screenshot, application platform, persona context, relationship context, user instruction, agent intent, benchmark bucket, and expected **Allow/Mask/Ask** decision. The construction and audit process checks plausibility, fluency, alignment with candidate agent intent, support for the expected decision, OCR detectability of visible sensitive evidence, and consistency between the rendered screenshot and scenario metadata [2605.28646].

Sample-level PII annotations cover medical, address, financial, identity, contact, credential, compensation, supplier/customer, and security-related signals. The largest localizable evidence categories are medical, financial, contact, address/location, and account or identity credentials. Audit statistics report **715** samples with non-None PII type, **685** location-ready PII evidence samples, **1,529** localizable evidence boxes, and **736** L1/L3 usable samples.

The benchmark’s labels are interpreted operationally:

- **Allow**: no intervention; screenshot may be exposed.
- **Mask**: protected content must be redacted before exposure.
- **Ask**: pause and request user confirmation.

A policy-label consistency pass checks lower-level mask and confirmation fields before deriving the final label. This is important because the final class is not assigned independently of the underlying confirmation and masking conditions.

Reported metrics are **joint decision accuracy**, **Mask F1**, **Ask recall**, **leak rate**, **over-protection**, and **raw-upload rate**. Error notions include **Leak** for cases where expected Mask is predicted as non-Mask, **OverProtect**, and **AskMiss**. SafeScreenshot region mistakes are separated into L3 diagnostics, which makes localization quality analytically distinct from high-level policy classification.

## 4. Static benchmark behavior and empirical difficulty

P-GUI-Evo is designed to expose contextual privacy boundaries that pattern matching, generic cloud reasoning, and exposure routing do not reliably resolve. In the main benchmark table, all values are reported in \(10^{-3}\). Policy-grounded MaskClaw reaches \(717 \pm 7\) accuracy, \(819 \pm 14\) Mask F1, \(196 \pm 6\) leak, \(377 \pm 32\) Ask recall, and \(0\) raw-upload rate; the paper reports that this improves over the strongest non-MaskClaw baseline, Static Regex, by **0.160** absolute accuracy, from **0.557** to **0.717** [2605.28646].

| Method | Acc. | Raw up. |
|---|---:|---:|
| Policy-grounded MaskClaw | \(717 \pm 7\) | 0 |
| Static Regex | \(557 \pm 0\) | 0 |
| Static Policy Corpus | \(254 \pm 0\) | 0 |
| Cloud Minimal | \(308 \pm 6\) | 1000 |
| Cloud Persona | \(370 \pm 4\) | 1000 |
| Cloud Full-Context | \(166 \pm 5\) | 1000 |
| EdgeClaw-ClawXRouter | \(401 \pm 6\) | 159 |

The benchmark reveals distinct failure modes. **Static Regex** is comparatively strong by accuracy because visible PII patterns matter, but it has **Ask recall 0** and cannot represent consent-sensitive boundaries. **Static Policy Corpus** performs poorly because static memory without strong applicability judgment causes severe mismatch. **Cloud Minimal**, **Cloud Persona**, and especially **Cloud Full-Context** expose raw screenshots under the protocol and tend toward over-confirmation or boundary confusion.

The **Cloud Full-Context** result is especially diagnostic. Despite seeing raw screenshot, app, command, intent, recipient context, and persona, it attains only **0.166** accuracy and predicts **Ask for 754/832 samples**. Its confusion summary is:
- Allow: 44 Allow / 7 Mask / 263 Ask
- Mask: 2 Allow / 21 Mask / 415 Ask
- Ask: 3 Allow / 1 Mask / 76 Ask

This makes over-confirmation visible as a benchmark-specific pathology rather than a generic model weakness. **EdgeClaw-ClawXRouter** reduces raw cloud exposure to **15.9%**, but routing alone does not resolve the core Allow/Mask/Ask decision boundary.

Error analysis on the 832-example benchmark reports **593 correct** and **239 joint errors**, with top categories:
- protected content allowed: **95**,
- unnecessary masking: **58**,
- confirmation replaced by masking: **37**.

These three account for **79.5%** of errors. Retrieval diagnostics report **239** wrong-boundary retrieval cases, **8** weak retrieval-evidence cases, and **585** neither. This suggests that, once obvious perception failures are separated out, the remaining difficulty is largely boundary confusion across persona, action, and recipient context.

## 5. Role within MaskClaw and edge-side mediation

Within MaskClaw, P-GUI-Evo serves as the common protocol for evaluating local perception and evidence extraction, retrieval-grounded policy arbitration, SafeScreenshot construction, and alternative baselines such as regex, cloud reasoning, and routing. The online arbitration loop extracts local evidence
\[
E(s)=\{e_1,e_2,\ldots,e_n\},
\]
and consults policy memory
\[
C_t=\{r_1,r_2,\ldots,r_m\},
\]
where each rule is
\[
r_j=(s_j,t_j,a_j,p_j,z_j).
\]
Here \(s_j\) is scope, \(t_j\) trigger evidence, \(a_j\) recommended action, \(p_j\) conflict-resolution priority, and \(z_j\) rationale [2605.28646].

The paper states that ranking uses **facet match and evidence compatibility**, with recipient, action, and persona matches weighted above generic field-type matches. After arbitration, protected regions \(B^\ast(s)\subseteq B(s)\) are redacted to form the mediated screenshot
\[
I_{\mathrm{safe}}=\mathrm{redact}(I,B^\ast(s)).
\]

Layer-wise diagnostics clarify how P-GUI-Evo decomposes the problem operationally. Table E1 reports:
- **L1 OCR coverage**: \(0.967\)
- **L2 Open** policy accuracy: \(0.533 \pm 0.010\)
- **L2 Closed** policy accuracy: \(0.717 \pm 0.007\)
- **L3 strict redaction**: \(0.733\) strict no-flag rate

In \(10^{-3}\) reporting, L2 Open yields Mask recall \(770 \pm 7\), Allow recall \(308 \pm 12\), and Ask recall \(113 \pm 22\); L2 Closed yields Mask recall \(804 \pm 6\), Allow recall \(683 \pm 18\), and Ask recall \(377 \pm 32\). The improvement from Open to Closed is attributed to retrieval-grounded arbitration and mediator control rather than perception alone. A plausible implication is that P-GUI-Evo is structured not only as a classification benchmark but as a layered evaluation of perception, policy retrieval, and redaction safety.

## 6. Behavior-driven skill evolution

Although the static benchmark uses fixed policy memory, the same work uses P-GUI-Evo’s scenario design philosophy to study how privacy behavior can evolve from user feedback. MaskClaw records traces
\[
\tau=[(s_1,d_1,y_1),\ldots,(s_T,d_T,y_T)],
\]
extracts correction signals
\[
G=\{g_1,\ldots,g_l\},
\]
and updates memory through
\[
C_{t+1}=\mathrm{AuditMerge}(C_t,\Delta C_t).
\]
The five designed skill-evolution scenarios are **iCloud cleanup**, **App permission**, **High-value transfer**, **Calendar merge**, and **Driving mode** [2605.28646].

These scenarios turn confirmations, cancellations, edits, rejected actions, retries, and explicit skill instructions into candidate reusable skills. Candidate updates are filtered through a schema/coverage gate, a fixed text scorer, and an LLM-Judge or sandbox gate. The objective is
\[
\begin{aligned}
J(\theta) ={}& \lambda_1\,\mathrm{RuleCoverage} + \lambda_2\,\mathrm{TestDecisionAcc}\\
&+ \lambda_3\,\mathrm{SafetyPenalty} + \lambda_4\,\mathrm{SkillQuality},
\end{aligned}
\]
with
\[
\lambda_1=0.40,\quad \lambda_2=0.30,\quad \lambda_3=0.20,\quad \lambda_4=0.10.
\]

The sandbox gate checks confirmation timing, safety rationale, and executable state flow for downstream agents. This is necessary because a textual skill can mention the correct privacy rule yet still be unsafe due to wrong action order, incomplete flow, or weak scenario alignment.

Experimental protocol uses **5 scenarios**, **3 start conditions**, **3 random seeds**, and **20 evolution iterations**. Full scenario behavior results report that evolved skills raise held-out behavior accuracy from:
- **37.50 to 100.00** in iCloud photo cleanup,
- **25.00 to 100.00** in App permission,
- **20.00 to 100.00** in High-value transfer,
- **12.50 to 100.00** in Calendar merge,
- **25.00 to 100.00** in Driving mode.

Unsafe action rates fall from:
- **62.50 to 12.50**,
- **75.00 to 0.00**,
- **80.00 to 0.00**,
- **87.50 to 0.00**,
- **75.00 to 0.00**,

with macro averages moving from **24.00** base accuracy to **100.00** evolved accuracy, and from **76.00** base unsafe rate to **2.50** evolved unsafe rate. The paper also reports imperfect correction compliance in transfer and driving mode, which indicates that improved high-level behavior does not eliminate evidence and confirmation-timing failures.

## 7. Limitations and significance

P-GUI-Evo is intentionally controlled rather than naturalistic. It is sanitized, reconstructed from HTML/UI surrogates, based on synthetic personas rather than real users or organizations, and validated offline rather than through long-term live deployment. The paper is explicit that the skill-evolution results are scenario-level and not evidence of robust lifelong personalization [2605.28646].

These limitations matter because they bound what can be claimed. The benchmark operationalizes a difficult and previously under-specified problem—context-sensitive screenshot exposure arbitration—but it does so under reconstructed conditions. This suggests strong value for controlled comparison, while leaving deployment realism as future work.

Its significance lies in the specificity of the target. P-GUI-Evo bridges GUI-agent evaluation and privacy evaluation by forcing systems to resolve a triage problem that depends jointly on screenshot content, task, persona, recipient, and agent intent. Pattern detectors tend to over-mask and cannot ask; cloud reasoning tends to over-confirm and uploads raw screenshots; routing reduces exposure but does not itself solve contextual arbitration. P-GUI-Evo makes these failure modes directly measurable.

In that sense, P-GUI-Evo is best understood as a benchmark for **context-conditioned privacy mediation at the GUI surface**. It turns the question “is there sensitive content on screen?” into the more demanding question “given the current user, task, recipient, and intended action, should this screenshot be allowed, masked, or held for confirmation before exposure?”

Source: https://www.emergentmind.com/topics/p-gui-evo