EAPrivacy: Embodied Privacy Benchmark

• EAPrivacy is a benchmark that evaluates physical-world privacy awareness by testing how embodied agents recognize and act on privacy-relevant constraints.
• It uses over 400 procedurally generated PDDL-style scenarios across diverse physical scenes to assess sensitivity recognition, context, and social norms.
• The evaluation reveals a gap between language-only privacy competence and embodied task performance, highlighting critical misalignment in current models.

EAPrivacy is an evaluation benchmark for LLM-powered embodied agents that measures physical-world privacy awareness rather than privacy behavior in dialogue alone. It was introduced to test whether a model used as the reasoning or planning core of an embodied system can recognize, infer, and act on privacy-relevant constraints arising from objects, spatial context, environmental change, task demands, and conflicts with social norms. The benchmark contains more than 400 procedurally generated scenarios spanning more than 60 unique physical scenes, and it is organized into four tiers that progress from sensitive-object recognition to high-stakes cases in which privacy must be weighed against safety or welfare (Shen et al., 27 Sep 2025).

1. Conceptual scope and privacy model

EAPrivacy defines its target capability as physical-world privacy awareness: the ability to recognize, infer, and act on privacy-relevant constraints that arise from objects, spatial context, ongoing activities, social cues, and ethical tradeoffs in real environments. The benchmark therefore departs from privacy evaluations that are confined to natural language based scenarios, such as secret leakage or compliance with textual privacy instructions, and instead asks whether an embodied assistant can notice that a passport on a desk is private, infer that a wrapped gift hidden behind a monitor should not be exposed, or override personal privacy in order to report a violent emergency next door (Shen et al., 27 Sep 2025).

The privacy concept used in EAPrivacy is broader than information leakage. The benchmark explicitly frames physical-world privacy using several notions: contextual object privacy, contextual action privacy, physical context privacy, and inferential privacy. This means that privacy violations are not restricted to verbal disclosure. They also include actions such as reading, moving, exposing, entering, interrupting, or reporting, when those actions are inappropriate in light of the environment and the social situation. The benchmark is influenced by contextual integrity and by human judgments of action appropriateness, but it operationalizes these concerns in embodied, physically grounded tasks rather than purely linguistic prompts (Shen et al., 27 Sep 2025).

A central claim of EAPrivacy is that language-only privacy competence is insufficient for embodied deployment. In the appendix, modern post-alignment models such as Gemini 2.5 Pro, Gemini 2.5 Flash, and GPT-5 are reported to achieve 0 secret leak rate on a prior text benchmark, yet perform substantially worse on EAPrivacy. This suggests that privacy behavior in dialogue does not transfer automatically to privacy behavior in environments represented through objects, rooms, actions, and sensory summaries (Shen et al., 27 Sep 2025).

2. Benchmark construction and representation

EAPrivacy is built from procedurally generated scenarios across homes, offices, labs, hospitals, and public spaces. A defining design choice is that the benchmark does not present scenarios only as plain narratives. Instead, it uses structured PDDL-style state descriptions that encode objects, rooms, and spatial relations such as ontop, inside, under, onfloor, and inroom. Inputs also include predefined action lists in PDDL-like form, with action names, parameters, preconditions, and effects, and in later tiers they include action returns that simulate multimodal observations through textual summaries such as Visual: 5 people at table; 1 at whiteboard or Audio Cue: a cry for help (Shen et al., 27 Sep 2025).

This representation is intended to approximate the kind of structured world model an embodied agent would actually receive. Models are asked to produce constrained outputs such as rating(X), selection(X), numbered action lists, or lists of PDDL object identifiers. The benchmark therefore measures privacy reasoning over structured state and action spaces rather than free-form narrative continuation (Shen et al., 27 Sep 2025).

Human validation is provided by five PhD-level raters. In action-rating tasks, the average human rating is used for comparison; in selection tasks, majority or most-frequent human judgments define the correct action. The raters were university-affiliated, familiar with US-based norms, and spent about two hours on the annotation task. The paper treats this as a source of consistency, but also as a limitation for normative diversity (Shen et al., 27 Sep 2025).

3. Tier structure and evaluation criteria

EAPrivacy is organized as a four-tier progression from simpler to harder forms of physically grounded privacy reasoning.

Tier	Focus	Primary evaluation
1	Sensitive Object Identification	object selection, [MOR](https://www.emergentmind.com/topics/model-order-reduction-mor), ONC
2	Privacy in Shifting Physical Environments	rating mode, selection mode
3	Inferential Privacy under Task Conflicts	privacy violation, task completeness, selection
4	Social Norms vs. Personal Privacy	binary rating, binary selection

Tier 1 places a single primary sensitive object on a target surface such as a desk or table, surrounded by non-sensitive distractors. The task is to list all sensitive objects on the specified container. Distractor complexity is varied at 3, 5, 10, and 30 objects. The benchmark gives two explicit metrics. If $I$ is a binary indicator for correct identification of the primary sensitive object and $N$ is the number of objects output as sensitive, the Main Object Ratio is

$MOR = \frac{I}{N}.$

This penalizes over-identification. The appendix also defines Objects Not On Container (ONC). Let $C$ be the target container, $M$ the primary sensitive object, $L=[o_1,\dots,o_{|L|}]$ the model output list, $\mathcal{I}$ the set of valid object instances, and $\text{place}(o)$ the supporting container or surface of object $o$. Then

$$\text{ONC} = \bigl| \{ o \in L : o \in \mathcal{I},\; \text{place}(o) \neq C \} \bigr|.$$

Ideal ONC is 0 (Shen et al., 27 Sep 2025).

Tier 2 evaluates context-sensitive action appropriateness under changing physical environments. The same action may be acceptable in one context and inappropriate in another. There are two modes. In Rating Mode, the model rates an action on a 1–5 appropriateness scale, and performance is measured by Mean Absolute Difference (MAD) from the average human rating. In Selection Mode, the model chooses the best action from a triplet containing one most-appropriate action, one neutral action, and one most-inappropriate action; performance is Selection Accuracy (Shen et al., 27 Sep 2025).

Tier 3 evaluates inferential privacy under task conflicts. Here the privacy constraint is often unstated. The model must infer it from dialogue, spatial hiding, or social behavior, and must then plan around it while still making progress on an explicit task such as “move everything on the desk.” The main metrics are Privacy Violation Rate, Task Completeness, and Selection Accuracy. Task Completeness is defined as the average proportion of the primary task completed without violating privacy (Shen et al., 27 Sep 2025).

Tier 4 evaluates situations in which privacy should be overridden by stronger social norms such as safety, medical response, or child welfare. Evaluation is binary. In Rating Mode, the model labels an action as appropriate (1) or inappropriate (0). In Selection Mode, it chooses the correct action from a pair containing one appropriate and one inappropriate option (Shen et al., 27 Sep 2025).

4. Experimental protocol and headline results

The benchmark evaluates 16 models from several families, including Claude-3.5-haiku, Gemini-2.5-flash, Gemini-2.5-pro, gpt-4o-mini, gpt-4o, gpt-5-low, gpt-5-high, gpt-oss-20b-low, gpt-oss-20b-high, Qwen-30B, Qwen-30B-thinking, Qwen-32B, Qwen-32B-thinking, and Llama-3.3-70B. A major experimental theme is the comparison between default or high-reasoning variants and “w.o.think” or lower-thinking variants (Shen et al., 27 Sep 2025).

Tier 1 already reveals substantial weakness. Main Object Identification ranges from 26% to 96% across models. Mean performance declines from about 63.9% at distractor complexity 3 to 52.0% at complexity 30. MOR is substantially lower: the best observed value is 59.45%, and the average declines from 45.8% at complexity 3 to 28.9% at complexity 30. The interpretation offered is that many models identify the primary sensitive object but also over-label many other objects as sensitive (Shen et al., 27 Sep 2025).

Tier 2 provides the benchmark’s headline dynamic-context result. In Selection Mode, the best model, Gemini 2.5 Pro, reaches only 59% accuracy. In Rating Mode, the best model by MAD is Gemini 2.5 Flash with MAD = 1.32. The paper emphasizes that even the best MAD means the model is off by more than one full point on average on a 1–5 appropriateness scale. Representative selection results include 0.55 for Gemini-2.5-flash, 0.59 for Gemini-2.5-pro, 0.41 for gpt-5-high, 0.18 for gpt-4o-mini, and 0.00 for gpt-4o in the full table (Shen et al., 27 Sep 2025).

Tier 3 exposes the strongest conflict between task-following and privacy preservation. Representative Privacy Violation Rate values include 0.86 for Claude-3.5-haiku, 0.74 for Gemini-2.5-pro, 0.71 for gpt-4o, 0.78 for gpt-5-high, 0.97 for gpt-oss-20b-low, and 0.98 for qwen-30b-thinking. The benchmark summary reports that models prioritized task completion over the privacy constraint in up to 86% of cases. At the same time, selection accuracy can be high: gpt-5-high reaches 1.00 selection accuracy in Tier 3 while still exhibiting 0.78 privacy violation in generated plans. The benchmark treats this gap as evidence that models can often pick the correct answer in simplified multiple-choice form while failing to act on it in physically grounded planning (Shen et al., 27 Sep 2025).

Tier 4 is easier, but not safe. The best models in Selection Mode reach 1.00: both gpt-5-low and gpt-5-high do so. In Rating Mode, the best reported value is 0.95 for gpt-5-low. Yet the paper emphasizes that leading models such as GPT-4o and Claude-3.5-haiku disregarded the relevant social norm in over 15% of rating cases. This is the basis for the abstract’s claim that in high-stakes situations pitting privacy against critical social norms, leading models disregarded the social norm over 15% of the time (Shen et al., 27 Sep 2025).

5. Failure modes and interpretive significance

EAPrivacy identifies several recurrent failure patterns. In Tier 1, many models exhibit biased misinterpretation of sensitivity: they equate “sensitive” with danger or fragility and flag objects such as a knife or glass cup while missing the intended privacy-sensitive item. Others exhibit contextual inappropriateness conflation, marking a book or trophy as sensitive because it is in an unusual location such as a refrigerator. A third pattern is imputed sensitivity, in which anything that might store information, such as a note or laptop, is treated as sensitive without contextual justification (Shen et al., 27 Sep 2025).

Tier 2 reveals what the paper calls asymmetric social conservatism. Models often avoid obviously inappropriate options but still fail to choose the truly appropriate action, producing conservative yet shallow judgments. The benchmark interprets this as brittle social context understanding, because models do not reliably transfer a privacy judgment from one scenario to another structurally similar scenario (Shen et al., 27 Sep 2025).

Tier 3 shows two especially salient errors. The first is literal interpretation over social nuance: all 16 models exhibit cases in which they follow commands like “move everything on the desk” while ignoring cues that one object is a private anniversary gift, secret recipe, confidential prototype, or sensitive medical vial. The second is failure to understand physical occlusion, where models attempt to grasp items physically located beneath other objects, such as a blueprint under a notepad and mug. This means the benchmark is probing not only normative reasoning but also physically plausible action semantics (Shen et al., 27 Sep 2025).

Tier 4 exposes two further patterns. One is underestimation of physical threat, where a model recommends direct confrontation in a dangerous setting rather than a safer escalation path, such as silently alerting hospital security about a visible handgun. The other is literal helpfulness vs. social dignity, where a model tries to help but does so in a privacy-violating or humiliating way, such as returning a lost letter by publicly revealing its sensitive contents (Shen et al., 27 Sep 2025).

The benchmark’s interpretive significance lies in its demonstration that privacy alignment for embodied agents is not equivalent to privacy alignment for chatbots. EAPrivacy explicitly contrasts itself with prior language-only work such as Mireshghallah et al.’s contextual-integrity benchmark, and the appendix reports that models with 0 secret leak rate on that prior benchmark still fail substantially on EAPrivacy. This suggests that current models possess fragments of privacy knowledge, but not robust physically grounded privacy reasoning (Shen et al., 27 Sep 2025).

A further interpretive result is the negative effect of explicit reasoning variants. The benchmark reports that “thinking” often hurts performance, especially for Gemini 2.5 Flash, Gemini 2.5 Pro, and Qwen variants, in Tier 1, Tier 3, and Tier 4. The paper interprets this as a possible over-thinking effect, in which longer reasoning traces make the model too literal, too cautious in the wrong way, or too focused on task completion while missing subtle social constraints (Shen et al., 27 Sep 2025).

6. Limitations, implications, and future directions

EAPrivacy is explicit about its limits. It uses synthetic PDDL scenes and simulated sensory summaries rather than real robot perception, so there remains a deployment gap to systems that operate on actual camera, audio, and sensor streams. It relies on a small annotator pool of five PhD-level raters, which strengthens consistency but limits representativeness. It is grounded in US-based legal and social norms, which provides a stable annotation standard but may not generalize cross-culturally. It also supplies textual summaries of multimodal observations rather than raw images or audio, so it is a benchmark for physically grounded reasoning rather than end-to-end multimodal robotic perception (Shen et al., 27 Sep 2025).

Another limitation is that selection tasks can overestimate competence. The large gap between multiple-choice accuracy and free-form planning performance in Tier 3 shows that choosing the correct action in a constrained format is easier than generating a privacy-preserving plan under realistic task pressure. The appendix also notes that standard deviations are relatively low, which supports result stability, but this does not resolve the underlying normative and ecological constraints of the benchmark (Shen et al., 27 Sep 2025).

Despite these limitations, EAPrivacy has important implications for embodied AI safety. It shows that current LLMs can perform well on language-based privacy tests while still failing when privacy must be inferred from cluttered objects, changing environments, hidden intentions, and conflicts with social norms. The benchmark therefore frames its findings as evidence of a fundamental misalignment in physically grounded privacy. Its concrete agenda for future work includes better spatial grounding of privacy concepts, stronger sensitivity to contextual integrity in embodied settings, more robust social inference, broader and more diverse human annotation, evaluation beyond simulated settings, and alignment methods that jointly reason over world state, action consequences, privacy constraints, and social norms under uncertainty (Shen et al., 27 Sep 2025).