NormAct: A Benchmark for Hidden Social Norm Compliance in Embodied Planning

Abstract: Multimodal LLMs (MLLMs) are increasingly deployed as embodied planners in egocentric environments, where task success requires not only achieving instructed goals but also acting in socially appropriate ways. While explicit goals may render certain actions optimal, implicit social norms often impose hidden constraints. Existing evaluations typically focus on explicit goal achievement or direct norm knowledge, seldom assessing whether planners can infer and apply these hidden constraints within action sequences. We introduce NormAct, a benchmark for embodied social-norm interactions that evaluates plans on Goal Achievement, Norm Compliance, and overall Task Success. NormAct uniquely embeds hidden norms within ordinary tasks, testing whether models can realize them without explicit instruction. Experiments with state-of-the-art MLLMs (GPT-5.4, Claude Opus 4.7, Gemini 3 Pro) reveal a significant gap: models achieve explicit goals in 67.3\% of cases, but comply with hidden norms in only 26.4\%. Cue-condition experiments indicate that this gap stems not from a lack of general social knowledge, but from challenges in activating and grounding relevant norms in context. To address this, we propose NormPerceptor, a context-conditioned cue generator that infers scene-relevant norms prior to planning, increasing Task Success from 24.2\% to 46.7\%. Our results underscore the importance of enabling embodied agents to proactively detect hidden norms, ground them in visual evidence, and integrate them as action-planning constraints. Our benchmark is publicly available at https://huggingface.co/datasets/Caleb196x/NormAct.

• The paper presents a novel benchmark that distinguishes explicit goal achievement from implicit social norm compliance in embodied AI planning.
• It employs TongSim and progressive cue conditions to diagnose norm activation failures and assess action sequences against hidden social constraints.
• Empirical results reveal that while context-conditioned cues significantly boost norm compliance, translating activated norms into precise actions remains challenging.

Action-Level Benchmarking of Hidden Social Norm Compliance in Embodied Planning

Problem Framing and Benchmark Motivation

NormAct addresses a critical evaluation gap for multimodal LLMs (MLLMs) acting as planners in egocentric embodied environments. While existing benchmarks emphasize physical goal achievement or explicit norm judgments, they neglect implicit social constraints—norms not stated in task instructions yet critical for socially appropriate execution. NormAct formalizes these hidden norms as action-level constraints: completing a goal (e.g., retrieving an object) is insufficient if the action sequence violates situational social norms (e.g., disregarding ownership, interrupting conversations, bypassing queues).

Figure 1: A model may complete the explicit goal while violating an implicit social norm; NormAct distinguishes goal achievement from norm compliance and illustrates cue conditions used for evaluation.

NormAct leverages a high-fidelity simulation platform (TongSim) to construct 550 evaluation episodes covering 11 task types (e.g., road crossing, queue waiting, avoiding interruption, resource responsibility, privacy, and social relationships). Each scene combines explicit goals with latent, scene-encoded social constraints. The benchmark requires models to autonomously infer, ground, and operationalize norms as constraints upon their action plans.

Benchmark Design and Evaluation Methods

NormAct provides for each instance: a first-person RGB observation with paired semantic segmentation, explicit task goals, hidden social norm constraints, and a high-level action API. Tasks span five norm dimensions, ensuring broad coverage of everyday embodied scenarios.

Figure 2: First-person observation format integrates RGB and semantic segmentation views, supporting scene evidence and object-level grounding.

Evaluation is threefold:

• Goal Achieved: binary completion of explicit task goals
• Norm Compliance: binary compliance with hidden social norms
• Task Success: both goal and norm criteria satisfied

Models must output structured high-level action plans. Evaluation is computed at the sequence level, not explanations, ensuring norm compliance is operationalized in actions rather than descriptive text.

Diagnostic Cue Conditions and Failure Analysis

NormAct incorporates progressive cue conditions to diagnose failure points in the norm compliance pipeline:

• No cue: Only task goal and observation, requiring natural norm activation
• Category cue: Abstract social category added
• Specific cue: Human-written, scene-specific norm constraint provided
• Evidence cue: Makes perceptual evidence salient
• RAG cue: Provides retrieved generic norm knowledge
• Generated cue: Employs NormPerceptor for automatic scene-grounded norm cue generation

Error modes are annotated: norm inference (missing the constraint), perception-grounding (scene evidence missed), cue-to-action (wrong action sequence despite cue recognition), and goal–norm tradeoff (preserves norm but abandons goal).

Figure 3: Error signals across cue conditions illustrate reduction in norm-inference and perception-grounding failures as explicit cues are provided.

Empirical Results

Experiments with GPT-5.4, Claude Opus 4.7, and Gemini 3 Pro reveal significant gaps. In the no-cue condition, explicit goals are achieved in 67.3% of cases, but hidden norm compliance is only 26.4%, yielding just 21.8% full Task Success. Most goal-achieving plans violate the social constraint, indicating that goal completion alone is an insufficient metric for embodied competence.

Figure 4: Model performance under different cue conditions, showing sharp divergence between goal achievement and norm compliance.

Explicit cues (category and specific) substantially boost norm compliance (up to 63.9%) and Task Success (up to 49.6%), indicating latent capability if norm constraints are activated. Evidence cues improve norm compliance (67.1%) and Task Success (50.2%) by making scene evidence salient, confirming that failures mainly originate from activation and grounding, not lack of knowledge. RAG cues do not improve norm compliance (24.5%), highlighting the inefficacy of generic norm retrieval without contextual grounding.

NormPerceptor—a context-conditioned cue generator—improves Task Success from 24.2% to 46.7%. Automatic context helps bridge the gap, though it remains inferior to human-written cues, particularly for tasks demanding precise norm grounding.

Figure 5: Screenshots of Task Success in the avoiding interruption task, showing correct action sequence for norm compliance.

NormPerceptor Construction and Impact

NormPerceptor is trained via SFT on independently generated first-person RGB images paired with GPT-4o-generated norm-aware labels, separated from benchmark evaluation scenes to minimize leakage. It produces concise, scene-grounded social cues which, when fed to the planner, significantly increase norm compliance and Task Success. However, effective translation to precise actions remains a bottleneck, especially for nuanced social relations such as privacy or resource responsibility.

Figure 6: Example NormPerceptor training image and label, illustrating context-aware norm grounding derived from visual evidence.

Practical and Theoretical Implications

NormAct demonstrates that action-level evaluation is necessary for assessing embodied AI in social environments. Task completion metrics overestimate competence when hidden norms are not considered. Successful autonomous agents must not only possess normative knowledge but also activate, ground, and operationalize it contextually in first-person environments. Explicit cues are diagnostically useful, and context-conditioned cue generation (as in NormPerceptor) shows promise, but reliable norm activation and translation to executable actions present ongoing challenges.

Benchmarks like NormAct enable systematic diagnosis of social incompetence in embodied planning, facilitating development and assessment of norm-aware AI. They highlight the need for architectures that combine perceptual grounding, norm activation, and robust action translation.

Future Prospects

The results suggest several directions:

• Expansion to long-horizon tasks requiring persistent norm tracking
• Transfer and generalization evaluation of cue-generation across planners and environments
• Integration of causal/situational reasoning for nuanced relations (e.g., overlapping norms, conflicting goals)
• Meta-learning or continual learning for individual or cultural norm adaptation

Automated cue generation modules (e.g., NormPerceptor) will likely become vital in real-world deployed agents, provided their precision and grounding can be improved. Further research is required to operationalize implicit norm compliance at scale, minimizing reliance on explicit annotation.

Conclusion

NormAct establishes implicit social norm compliance as an action-level benchmark for embodied planning, disentangling explicit goal completion from hidden social constraints. Empirical evidence shows a large gap between task achievement and socially appropriate execution, even among state-of-the-art MLLMs. Scene-grounded contextual cues, especially those generated automatically, significantly close this gap but do not fully supplant human-level norm inference. Theoretical and practical advances in activation, grounding, and translation will be necessary to realize robust, socially competent embodied agents.