NormAct: Social Norm-Constrained Planning

• NormAct is a benchmark for embodied planning that integrates hidden social norms into ordinary tasks to evaluate both explicit goal achievement and social compliance.
• It uses high-fidelity 3D simulations with multimodal observations to embed latent social constraints into diverse action sequences.
• Experimental results reveal that context-conditioned cues, such as from NormPerceptor, significantly enhance norm compliance without compromising task completion.

Searching arXiv for “NormAct” and closely related embodied social norm planning work to ground the article in current literature. NormAct is a benchmark for embodied, egocentric planning with multimodal LLMs in which success requires not only satisfying an explicit task goal but also complying with a hidden social norm that is not stated in the instruction. Its defining feature is the insertion of latent social constraints into otherwise ordinary tasks—such as crossing a road, retrieving an object, asking for directions, or washing hands—so that the evaluation distinguishes literal goal completion from socially appropriate execution. The benchmark therefore operationalizes embodied social competence as constrained action planning rather than as norm classification or free-form moral judgment, and it pairs this evaluation framework with a context-conditioned cue generator, NormPerceptor, for improving hidden-norm planning (Zhao et al., 26 Jun 2026).

1. Conceptual framing

NormAct addresses a setting in which an embodied planner receives an ordinary task instruction in a first-person environment, but the scene contains visual evidence that makes some plans socially unacceptable. The norm is hidden in the main benchmark condition: it is not verbalized in the instruction, and the model must infer it from contextual evidence such as a visible zebra crossing, a queue, an ongoing conversation, a running faucet, or a private bedroom (Zhao et al., 26 Jun 2026).

This design differentiates NormAct from two common evaluation regimes. First, it is not a benchmark of explicit goal achievement alone, because a plan may complete the stated task while violating ownership, privacy, etiquette, public rules, or resource-responsibility constraints. Second, it is not primarily a benchmark of direct norm knowledge, because the central question is whether a planner can activate and apply a relevant norm inside an action sequence. The paper states this distinction explicitly: existing evaluations typically focus on explicit goal achievement or direct norm knowledge, whereas NormAct tests whether planners can infer and apply hidden constraints within action sequences (Zhao et al., 26 Jun 2026).

The benchmark is built on TongSim, described as a high-fidelity 3D simulator with photorealistic scenes, physically plausible interactions, and semantic annotations of objects, agents, and affordances. The released benchmark is publicly available at https://huggingface.co/datasets/Caleb196x/NormAct (Zhao et al., 26 Jun 2026).

2. Benchmark construction and task taxonomy

Each NormAct episode is generated from a task template that simultaneously guarantees two conditions: the scene contains the physical affordances needed to accomplish the explicit goal, and it also contains the contextual evidence needed to infer the hidden social norm. The generation pipeline applies controlled perturbations including object placement changes, character position variation, viewpoint variation, and irrelevant background object variation while preserving the intended goal–norm relation (Zhao et al., 26 Jun 2026).

The model’s observation is multimodal and grounded. Each episode provides a first-person concatenated observation containing an RGB image and a paired semantic or instance-segmentation view in which each interactable object is labeled with a numeric ID. This ensures that the task is not merely language-level social reasoning: the planner has access both to scene appearance and to object-level grounding (Zhao et al., 26 Jun 2026).

The output is a structured high-level action plan rather than an explanation. The required format is a JSON-like sequence of action steps, and the action space is a closed high-level API including look_at_object, look_at_location, move_to_object, move_and_take_object, put_down_to_location, speak_to, rest, wash_hands, wash_object_in_hand, mop_floor, interact, open_door, close_door, knock_door, sit_down_to_object, and point_at_object. The paper emphasizes that the primary object of evaluation is the action sequence, not the explanation (Zhao et al., 26 Jun 2026).

NormAct contains 5 norm dimensions, 11 task types, 50 instances per task type, and 550 evaluation episodes total. The five dimensions and their associated tasks are as follows.

Norm dimension	Visible evidence / required adjustment	Task types
Public rules	Crosswalks, queues, shared service order; follow the public procedure before completing the goal	Road crossing; Queue waiting
Etiquette and interaction	Ongoing conversations, narrow shared paths, interpersonal distance; avoid interrupting, yield, approach before speaking	Avoiding interruption; Giving way; Approaching before talking
Resource responsibility	Running faucets, used dishes, shared objects or tools; restore the environment after task completion	Turning off faucets; Returning shared objects; Washing used dishes
Privacy and ownership	Private rooms, personal belongings in another person’s home; ask permission or choose alternatives	Avoiding private rooms; Respecting belongings
Social relationship	Age, need, or social role implied by nearby people; prioritize or assist those with stronger social claims	Giving priority to an elder

The task taxonomy is intentionally not exhaustive. The paper states that these categories were chosen because they are observable, actionable, and evaluable in embodied settings (Zhao et al., 26 Jun 2026).

3. Formal task definition and evaluation protocol

The paper defines each instance as

$$x_i = (o_i, g_i, n_i, A, R_{\mathrm{goal}}, R_{\mathrm{norm}})$$

where $o_i$ is the egocentric observation, $g_i$ the explicit task goal, $n_i$ the hidden social norm, $A$ the high-level action space, $R_{\mathrm{goal}}$ the evaluator for goal achievement, and $R_{\mathrm{norm}}$ the evaluator for norm compliance. At test time, the planner generates an action sequence

$\tau_i = \mathrm{MLLM}(o_i, g_i, c_i \mid A).$

A successful planner must satisfy both the explicit goal and the hidden norm (Zhao et al., 26 Jun 2026).

NormAct’s core methodological contribution is its split evaluation. For an action sequence $\tau_i = (a_1,\ldots,a_{T_i})$, the paper defines three binary rewards: $$R_{\mathrm{goal}}(\tau_i) = \mathbb{I}[g_i \text{ is achieved}],$$

$o_i$0

$o_i$1

These are then averaged over the benchmark to produce Goal Achieved, Norm Compliance, and Task Success (Zhao et al., 26 Jun 2026).

A technically important detail is that $o_i$2 is process-oriented rather than purely final-state based. In the road-crossing example, the evaluator checks whether the agent ever steps into the road outside the zebra crossing, not merely whether it eventually reaches the other side. This makes the benchmark sensitive to norm violations that occur transiently during execution (Zhao et al., 26 Jun 2026).

Primary scoring is automatic and task-specific, using rule-based evaluators for goal and norm. For diagnostic analysis, the paper uses GPT-4o to assign failure-cause labels to generated action sequences. The four failure modes are Norm inference failure, Perception-grounding failure, Cue-to-action failure, and Goal–norm tradeoff. For validation, the authors randomly sampled 50 cases per cue condition for manual verification and report that no issues were found (Zhao et al., 26 Jun 2026).

4. Empirical findings and diagnostic interpretation

The benchmark evaluates GPT-5.4, Claude Opus 4.7, and Gemini 3 Pro, with 550 trials per model per cue condition and 1,650 trials per cue condition across the three models. In the main no-cue setting, the aggregate results are:

• Goal Achieved: 67.3%
• Norm Compliance: 26.4%
• Task Success: 21.8% (Zhao et al., 26 Jun 2026)

The dominant error pattern is therefore not failure to complete the explicit task but failure to do so appropriately. The paper highlights that the largest outcome class in the no-cue condition is “goal achieved but norm violating,” 751 of 1,650 trials. This is the empirical basis for the claim that goal-only evaluation overestimates embodied competence in social environments (Zhao et al., 26 Jun 2026).

Cueing experiments sharpen that diagnosis. Aggregate results across models show that a Category cue raises Norm Compliance to 43.5% and Task Success to 33.9%, while a Specific cue raises them to 63.9% and 49.6%, respectively. The paper states that explicit cues improve Norm Compliance much more than Goal Achievement, indicating that social behavior is not reducible to generic planning strength (Zhao et al., 26 Jun 2026).

Additional experiments fix Gemini 3 Pro as the planner and compare six cue conditions.

Cue condition	Norm Compliance	Goal Achieved	Task Success
No cue	26.7	77.3	24.2
Category cue	49.3	78.4	41.6
Specific cue	70.7	71.6	56.5
Evidence cue	67.1	67.5	50.2
RAG cue	24.5	83.6	23.1
Generated cue	50.0	76.2	46.7

These numbers support the paper’s central interpretation. Evidence cue recovers much of the benefit of Specific cue, whereas RAG cue improves explicit goal completion but fails on hidden-norm compliance. The authors therefore argue that the bottleneck is not simply missing social knowledge; it is difficulty in activating the relevant norm from scene context and grounding it in perceptual evidence (Zhao et al., 26 Jun 2026).

The failure-mode analysis is consistent with that reading. Norm inference failures drop from 757 under no cue to 204 under specific cue. By contrast, Cue-to-action failures remain relatively stable, from 160 under no cue to 126 under specific cue, and Goal–norm tradeoffs increase from 111 to 380 when the norm is made salient. The benchmark thus separates at least three distinct difficulties: recognizing which norm applies, mapping that norm into an executable plan, and preserving explicit goal satisfaction once the norm has been enforced (Zhao et al., 26 Jun 2026).

Task-level heterogeneity is also substantial. Some tasks are almost never solved norm-compliantly without cueing: Returning shared objects has 0.0% no-cue Norm Compliance with 62.7% no-cue Goal Achieved, Washing used dishes has 0.0% no-cue Norm Compliance with 64.7% Goal Achieved, and Avoiding private rooms has 0.7% no-cue Norm Compliance with 45.3% Goal Achieved. Other tasks are easier socially: Avoiding interruption reaches 68.7% no-cue Norm Compliance, rising to 92.7% under specific cue, and Giving way rises from 56.7% to 98.7% (Zhao et al., 26 Jun 2026).

5. NormPerceptor

NormPerceptor is the paper’s proposed intervention for improving hidden-norm planning. It is a context-conditioned cue generator that operates before the planner: given the first-person RGB observation and task goal, it generates a short norm-aware cue, which is then prepended to the planner input. Formally,

$o_i$3

and the planner then produces

$o_i$4

The module is therefore designed to externalize norm activation and grounding rather than to modify the action model directly (Zhao et al., 26 Jun 2026).

The generated cue is intended to connect visible evidence, the likely relevant norm, and an action-relevant implication. The appendix example for an avoiding-interruption scenario illustrates the target style: a scene description, the statement that one should not interrupt others who are talking, and the resulting recommendation to wait politely or ask a person who is not engaged in conversation (Zhao et al., 26 Jun 2026).

NormPerceptor is initialized from Qwen3-VL-2B-Instruct and trained by supervised fine-tuning on a cue-generation dataset that is independent of the NormAct test episodes. The training data comprise 11 task types, 100 diverse first-person RGB images per task type, and 1,100 training samples total, generated independently using Seedance 2.0 with diverse scene layouts, object configurations, viewpoints, and social contexts. Supervision labels are created using a GPT-4o-series model, and the reported SFT setup uses 60 training epochs, batch size 4, and learning rate $o_i$5 (Zhao et al., 26 Jun 2026).

Quantitatively, the main reported NormPerceptor result is the Generated cue condition with Gemini 3 Pro as planner. Relative to No cue, it changes:

• Norm Compliance: 26.7% $o_i$6 50.0%
• Goal Achieved: 77.3% $o_i$7 76.2%
• Task Success: 24.2% $o_i$8 46.7% (Zhao et al., 26 Jun 2026)

The gain is therefore concentrated in norm-aware planning rather than in raw goal completion. The paper interprets this as evidence that a scene-conditioned front-end can recover a substantial fraction of the benefit of human-authored cues, though generated cues still underperform both Specific cue and Evidence cue (Zhao et al., 26 Jun 2026).

6. Significance, scope, and limitations

NormAct reframes social competence in embodied AI as a planning problem with implicit constraints. Its central empirical result is the large gap between explicit-goal performance and socially acceptable execution, and its central methodological result is to make that gap measurable through separate Goal Achievement, Norm Compliance, and Task Success metrics. A common misconception is that this is merely a norm-knowledge benchmark; the benchmark design and cue analyses argue otherwise, because generic retrieved norm knowledge does not help in the way scene-relevant evidence does (Zhao et al., 26 Jun 2026).

A second misconception is that explanation quality is the primary target. The benchmark instead evaluates structured action sequences under a closed high-level API. A model may verbalize the correct rule yet still produce a norm-violating plan, and NormAct is explicitly constructed to expose that failure mode (Zhao et al., 26 Jun 2026).

The paper also identifies clear limits. The current release has 11 task types and short high-level action sequences; this improves binary evaluation reliability but does not capture the full space of long-horizon social interaction. Evidence-cue, RAG-cue, and Generated-cue experiments are run only with Gemini 3 Pro because it is the strongest base planner. NormPerceptor still trails human-authored Specific cue and Evidence cue, particularly on tasks involving privacy, queueing, and resource-use constraints. The ethical discussion notes that social norms vary across cultures and communities, and the authors state that NormAct should be used as a diagnostic tool, not as a certificate of deployment readiness for real robots (Zhao et al., 26 Jun 2026).

Within those bounds, the benchmark’s main contribution is to show that embodied social competence depends on the proactive detection of hidden norms, their grounding in visual evidence, and their integration into action planning. In that sense, NormAct situates social norms not as auxiliary annotations on completed behavior but as latent constraints that shape admissible trajectories from the outset (Zhao et al., 26 Jun 2026).