---
title: 'NormAct: Social Norm-Constrained Planning'
url: https://www.emergentmind.com/topics/normact
type: topic
---

# NormAct: Social Norm-Constrained Planning

Searching arXiv for “NormAct” and closely related embodied social norm planning work to ground the article in current literature.
NormAct is a benchmark for embodied, egocentric planning with multimodal large language models in which success requires not only satisfying an explicit task goal but also complying with a hidden social norm that is not stated in the instruction. Its defining feature is the insertion of latent social constraints into otherwise ordinary tasks—such as crossing a road, retrieving an object, asking for directions, or washing hands—so that the evaluation distinguishes literal goal completion from socially appropriate execution. The benchmark therefore operationalizes embodied social competence as constrained action planning rather than as norm classification or free-form moral judgment, and it pairs this evaluation framework with a context-conditioned cue generator, NormPerceptor, for improving hidden-norm planning [2606.27826].

## 1. Conceptual framing

NormAct addresses a setting in which an embodied planner receives an ordinary task instruction in a first-person environment, but the scene contains visual evidence that makes some plans socially unacceptable. The norm is hidden in the main benchmark condition: it is not verbalized in the instruction, and the model must infer it from contextual evidence such as a visible zebra crossing, a queue, an ongoing conversation, a running faucet, or a private bedroom [2606.27826].

This design differentiates NormAct from two common evaluation regimes. First, it is not a benchmark of explicit goal achievement alone, because a plan may complete the stated task while violating ownership, privacy, etiquette, public rules, or resource-responsibility constraints. Second, it is not primarily a benchmark of direct norm knowledge, because the central question is whether a planner can activate and apply a relevant norm inside an action sequence. The paper states this distinction explicitly: existing evaluations typically focus on explicit goal achievement or direct norm knowledge, whereas NormAct tests whether planners can infer and apply hidden constraints within action sequences [2606.27826].

The benchmark is built on TongSim, described as a high-fidelity 3D simulator with photorealistic scenes, physically plausible interactions, and semantic annotations of objects, agents, and affordances. The released benchmark is publicly available at `https://huggingface.co/datasets/Caleb196x/NormAct` [2606.27826].

## 2. Benchmark construction and task taxonomy

Each NormAct episode is generated from a task template that simultaneously guarantees two conditions: the scene contains the physical affordances needed to accomplish the explicit goal, and it also contains the contextual evidence needed to infer the hidden social norm. The generation pipeline applies controlled perturbations including object placement changes, character position variation, viewpoint variation, and irrelevant background object variation while preserving the intended goal–norm relation [2606.27826].

The model’s observation is multimodal and grounded. Each episode provides a first-person concatenated observation containing an RGB image and a paired semantic or instance-segmentation view in which each interactable object is labeled with a numeric ID. This ensures that the task is not merely language-level social reasoning: the planner has access both to scene appearance and to object-level grounding [2606.27826].

The output is a structured high-level action plan rather than an explanation. The required format is a JSON-like sequence of action steps, and the action space is a closed high-level API including `look_at_object`, `look_at_location`, `move_to_object`, `move_and_take_object`, `put_down_to_location`, `speak_to`, `rest`, `wash_hands`, `wash_object_in_hand`, `mop_floor`, `interact`, `open_door`, `close_door`, `knock_door`, `sit_down_to_object`, and `point_at_object`. The paper emphasizes that the primary object of evaluation is the action sequence, not the explanation [2606.27826].

NormAct contains **5 norm dimensions**, **11 task types**, **50 instances per task type**, and **550 evaluation episodes total**. The five dimensions and their associated tasks are as follows.

| Norm dimension | Visible evidence / required adjustment | Task types |
|---|---|---|
| Public rules | Crosswalks, queues, shared service order; follow the public procedure before completing the goal | Road crossing; Queue waiting |
| Etiquette and interaction | Ongoing conversations, narrow shared paths, interpersonal distance; avoid interrupting, yield, approach before speaking | Avoiding interruption; Giving way; Approaching before talking |
| Resource responsibility | Running faucets, used dishes, shared objects or tools; restore the environment after task completion | Turning off faucets; Returning shared objects; Washing used dishes |
| Privacy and ownership | Private rooms, personal belongings in another person’s home; ask permission or choose alternatives | Avoiding private rooms; Respecting belongings |
| Social relationship | Age, need, or social role implied by nearby people; prioritize or assist those with stronger social claims | Giving priority to an elder |

The task taxonomy is intentionally not exhaustive. The paper states that these categories were chosen because they are observable, actionable, and evaluable in embodied settings [2606.27826].

## 3. Formal task definition and evaluation protocol

The paper defines each instance as
\[
x_i = (o_i, g_i, n_i, A, R_{\mathrm{goal}}, R_{\mathrm{norm}})
\]
where \(o_i\) is the egocentric observation, \(g_i\) the explicit task goal, \(n_i\) the hidden social norm, \(A\) the high-level action space, \(R_{\mathrm{goal}}\) the evaluator for goal achievement, and \(R_{\mathrm{norm}}\) the evaluator for norm compliance. At test time, the planner generates an action sequence
\[
\tau_i = \mathrm{MLLM}(o_i, g_i, c_i \mid A).
\]
A successful planner must satisfy both the explicit goal and the hidden norm [2606.27826].

NormAct’s core methodological contribution is its split evaluation. For an action sequence \(\tau_i = (a_1,\ldots,a_{T_i})\), the paper defines three binary rewards:
\[
R_{\mathrm{goal}}(\tau_i) = \mathbb{I}[g_i \text{ is achieved}],
\]
\[
R_{\mathrm{norm}}(\tau_i) = \mathbb{I}[n_i \text{ is complied with}],
\]
\[
R_{\mathrm{success}}(\tau_i) = R_{\mathrm{goal}}(\tau_i) \land R_{\mathrm{norm}}(\tau_i).
\]
These are then averaged over the benchmark to produce **Goal Achieved**, **Norm Compliance**, and **Task Success** [2606.27826].

A technically important detail is that \(R_{\mathrm{norm}}\) is process-oriented rather than purely final-state based. In the road-crossing example, the evaluator checks whether the agent ever steps into the road outside the zebra crossing, not merely whether it eventually reaches the other side. This makes the benchmark sensitive to norm violations that occur transiently during execution [2606.27826].

Primary scoring is automatic and task-specific, using rule-based evaluators for goal and norm. For diagnostic analysis, the paper uses GPT-4o to assign failure-cause labels to generated action sequences. The four failure modes are **Norm inference failure**, **Perception-grounding failure**, **Cue-to-action failure**, and **Goal–norm tradeoff**. For validation, the authors randomly sampled 50 cases per cue condition for manual verification and report that no issues were found [2606.27826].

## 4. Empirical findings and diagnostic interpretation

The benchmark evaluates **GPT-5.4**, **Claude Opus 4.7**, and **Gemini 3 Pro**, with **550 trials per model per cue condition** and **1,650 trials per cue condition** across the three models. In the main **no-cue** setting, the aggregate results are:
- **Goal Achieved:** **67.3%**
- **Norm Compliance:** **26.4%**
- **Task Success:** **21.8%** [2606.27826]

The dominant error pattern is therefore not failure to complete the explicit task but failure to do so appropriately. The paper highlights that the largest outcome class in the no-cue condition is **“goal achieved but norm violating,” 751 of 1,650 trials**. This is the empirical basis for the claim that goal-only evaluation overestimates embodied competence in social environments [2606.27826].

Cueing experiments sharpen that diagnosis. Aggregate results across models show that a **Category cue** raises **Norm Compliance** to **43.5%** and **Task Success** to **33.9%**, while a **Specific cue** raises them to **63.9%** and **49.6%**, respectively. The paper states that explicit cues improve Norm Compliance much more than Goal Achievement, indicating that social behavior is not reducible to generic planning strength [2606.27826].

Additional experiments fix **Gemini 3 Pro** as the planner and compare six cue conditions.

| Cue condition | Norm Compliance | Goal Achieved | Task Success |
|---|---:|---:|---:|
| No cue | 26.7 | 77.3 | 24.2 |
| Category cue | 49.3 | 78.4 | 41.6 |
| Specific cue | **70.7** | 71.6 | **56.5** |
| Evidence cue | 67.1 | 67.5 | 50.2 |
| RAG cue | 24.5 | **83.6** | 23.1 |
| Generated cue | 50.0 | 76.2 | 46.7 |

These numbers support the paper’s central interpretation. **Evidence cue** recovers much of the benefit of **Specific cue**, whereas **RAG cue** improves explicit goal completion but fails on hidden-norm compliance. The authors therefore argue that the bottleneck is not simply missing social knowledge; it is difficulty in activating the relevant norm from scene context and grounding it in perceptual evidence [2606.27826].

The failure-mode analysis is consistent with that reading. **Norm inference failures** drop from **757** under no cue to **204** under specific cue. By contrast, **Cue-to-action failures** remain relatively stable, from **160** under no cue to **126** under specific cue, and **Goal–norm tradeoffs** increase from **111** to **380** when the norm is made salient. The benchmark thus separates at least three distinct difficulties: recognizing which norm applies, mapping that norm into an executable plan, and preserving explicit goal satisfaction once the norm has been enforced [2606.27826].

Task-level heterogeneity is also substantial. Some tasks are almost never solved norm-compliantly without cueing: **Returning shared objects** has **0.0%** no-cue Norm Compliance with **62.7%** no-cue Goal Achieved, **Washing used dishes** has **0.0%** no-cue Norm Compliance with **64.7%** Goal Achieved, and **Avoiding private rooms** has **0.7%** no-cue Norm Compliance with **45.3%** Goal Achieved. Other tasks are easier socially: **Avoiding interruption** reaches **68.7%** no-cue Norm Compliance, rising to **92.7%** under specific cue, and **Giving way** rises from **56.7%** to **98.7%** [2606.27826].

## 5. NormPerceptor

NormPerceptor is the paper’s proposed intervention for improving hidden-norm planning. It is a context-conditioned cue generator that operates before the planner: given the first-person RGB observation and task goal, it generates a short norm-aware cue, which is then prepended to the planner input. Formally,
\[
N_i = P_{\theta}(o_i^{\mathrm{rgb}}, g_i),
\]
and the planner then produces
\[
\tau_i = \pi(o_i, g_i, N_i \mid A).
\]
The module is therefore designed to externalize norm activation and grounding rather than to modify the action model directly [2606.27826].

The generated cue is intended to connect visible evidence, the likely relevant norm, and an action-relevant implication. The appendix example for an avoiding-interruption scenario illustrates the target style: a scene description, the statement that one should not interrupt others who are talking, and the resulting recommendation to wait politely or ask a person who is not engaged in conversation [2606.27826].

NormPerceptor is initialized from **Qwen3-VL-2B-Instruct** and trained by supervised fine-tuning on a cue-generation dataset that is independent of the NormAct test episodes. The training data comprise **11 task types**, **100 diverse first-person RGB images per task type**, and **1,100 training samples total**, generated independently using **Seedance 2.0** with diverse scene layouts, object configurations, viewpoints, and social contexts. Supervision labels are created using a **GPT-4o-series model**, and the reported SFT setup uses **60 training epochs**, **batch size 4**, and **learning rate \(1e^{-4}\)** [2606.27826].

Quantitatively, the main reported NormPerceptor result is the **Generated cue** condition with **Gemini 3 Pro** as planner. Relative to **No cue**, it changes:
- **Norm Compliance:** **26.7% \(\rightarrow\) 50.0%**
- **Goal Achieved:** **77.3% \(\rightarrow\) 76.2%**
- **Task Success:** **24.2% \(\rightarrow\) 46.7%** [2606.27826]

The gain is therefore concentrated in norm-aware planning rather than in raw goal completion. The paper interprets this as evidence that a scene-conditioned front-end can recover a substantial fraction of the benefit of human-authored cues, though generated cues still underperform both **Specific cue** and **Evidence cue** [2606.27826].

## 6. Significance, scope, and limitations

NormAct reframes social competence in embodied AI as a planning problem with implicit constraints. Its central empirical result is the large gap between explicit-goal performance and socially acceptable execution, and its central methodological result is to make that gap measurable through separate **Goal Achievement**, **Norm Compliance**, and **Task Success** metrics. A common misconception is that this is merely a norm-knowledge benchmark; the benchmark design and cue analyses argue otherwise, because generic retrieved norm knowledge does not help in the way scene-relevant evidence does [2606.27826].

A second misconception is that explanation quality is the primary target. The benchmark instead evaluates structured action sequences under a closed high-level API. A model may verbalize the correct rule yet still produce a norm-violating plan, and NormAct is explicitly constructed to expose that failure mode [2606.27826].

The paper also identifies clear limits. The current release has **11 task types** and short high-level action sequences; this improves binary evaluation reliability but does not capture the full space of long-horizon social interaction. Evidence-cue, RAG-cue, and Generated-cue experiments are run only with **Gemini 3 Pro** because it is the strongest base planner. NormPerceptor still trails human-authored **Specific cue** and **Evidence cue**, particularly on tasks involving privacy, queueing, and resource-use constraints. The ethical discussion notes that social norms vary across cultures and communities, and the authors state that NormAct should be used as a **diagnostic tool**, not as a certificate of deployment readiness for real robots [2606.27826].

Within those bounds, the benchmark’s main contribution is to show that embodied social competence depends on the proactive detection of hidden norms, their grounding in visual evidence, and their integration into action planning. In that sense, NormAct situates social norms not as auxiliary annotations on completed behavior but as latent constraints that shape admissible trajectories from the outset [2606.27826].

Source: https://www.emergentmind.com/topics/normact