LoSoNA: A Benchmark for Local Social Norm Adaptation in Group Conversations

Abstract: Online group chats are social spaces with local conversational norms that are rarely stated explicitly. The ability and willingness of LLM-based agents to recognize and adapt to these norms remains mostly unexplored. We introduce LoSoNA, a benchmark for local social norm adaptation in multi-party chat. Each scenario gives a subject model a curated group-chat transcript in which non-subject participants demonstrate a hidden local norm, followed by a final elicitor turn that forces a response revealing whether the subject has inferred that norm. We evaluate eight frontier and open-weight models under four prompting conditions that vary how explicitly the model is told to treat the prior conversation as evidence for how it should answer. Naive prompting remains limited for most models; explicit norm-aware prompting helps unevenly, with Gemini 3.1 Pro reaching $84.2\%$ and Claude Fable 5 reaching $81.6\%$, while several other models show small gains or regressions. LoSoNA contributes to recent calls for evaluating LLM social capabilities by testing whether models can infer local conversational norms from precedent and use them in a one-turn group-chat response.

• The paper introduces LoSoNA, a benchmark testing if LLMs can infer hidden local social norms from group chat transcripts in a single-turn response.
• The paper demonstrates that norm-informed prompting significantly boosts accuracy in models like Gemini 3.1 Pro and Claude Fable 5 while many remain at chance levels.
• The paper reveals that explicit norm guidance is critical for recovering naive failures, highlighting limitations in LLMs' adaptive social reasoning.

LoSoNA: Evaluating Local Social Norm Adaptation in Multi-Party LLM Conversations

Introduction and Motivation

The LoSoNA benchmark ("LoSoNA: A Benchmark for Local Social Norm Adaptation in Group Conversations" (2606.14600)) addresses a significant gap in LLM evaluation: the capacity for LLMs to recognize, infer, and appropriately adapt to local social norms in group chat environments. Whereas most extant social-interaction benchmarks target dyadic contexts, explicit goals, or require only surface-level social knowledge, LoSoNA operationalizes a more nuanced challenge—whether a language agent, as a group participant, can infer a hidden norm from prior conversational precedent and exhibit adaptation in a single, diagnostic response.

LLMs deployed in group chats must demonstrate both behavioral flexibility and context sensitivity, since small-group norms regularly diverge from broader cultural conventions and are often not explicitly articulated. LoSoNA is designed to test next-turn adaptation given only implicit evidence, thus focusing on foundational yet under-explored aspects of LLM social intelligence.

Benchmark Design and Methodological Rigor

LoSoNA formalizes the evaluation as follows: for each scenario, a curated group transcript is constructed such that other participants consistently follow a hidden local norm. The subject agent is exposed to this transcript (never the norm label), then required to respond to an elicitor turn which behaviorally diagnoses adaptation. Scenarios are programmatically generated across a cross-product of event types and norm types, filtered by applicability and then subjected to human curation to enforce contextual plausibility and experiential verisimilitude.

Figure 1: The subject sees precedent-dense transcripts and must produce a next-turn response, without ever observing either the target norm label or an explicit description. Naive models default to generic politeness or neutrality, while effective models exhibit context-specific adaptation.

The taxonomy underlying scenario generation includes 17 common group-chat events and 22 norm types, e.g., bystander intervention, non-affiliative support, binary-only answers, norm-based joke responses, evidence sourcing, and highly group-specific ritual behaviors. The benchmark thus allows for both broad coverage and fine-grained, norm-based diagnosis.

Evaluation is strictly single-turn: models do not observe conversational fallout, reducing response ambiguity and enabling more controlled scoring. The use of a fixed LLM judge (Gemini 3.1 Pro Preview) with explicit, norm-instructed JSON outputs enforces judgment consistency, further augmented by dual-model auditing and targeted human validation.

Models, Prompting Paradigms, and Metrics

Eight contemporary LLMs (OpenAI GPT-5.5, Claude Opus 4.8, Claude Fable 5, Gemini 3.1 Pro, Qwen2.5-72B-Instruct, Llama 3.3-70B-Instruct, Mistral Medium 3.1, and Gemma 3-27B-IT) are benchmarked under four prompt paradigms:

• Naive: No explicit adaptation instruction
• Elicitor-only: Focus on the latest message for response
• Style adaptation: Soft adaptation to context/tone/habits
• Norm-informed: Direct hint that a repeated local norm may exist

For each (model, scenario, prompt) triple, three completions are sampled. The primary metric is accuracy-at-3—majority compliance over three responses per scenario/prompt pair—as judged by the fixed model judge. Recovery from naive failures and introduction of regressions are specifically tracked.

Figure 2: Prompt-wise majority accuracy for each subject model, with sampling variability. Norm-informed prompting yields strong gains for Gemini 3.1 Pro and Claude Fable 5; naive prompting is insufficient for most models.

Figure 3: The paired effect of norm-informed vs. naive prompt: substantial accuracy increases are concentrated in specific models (notably Gemini 3.1 Pro and Claude Fable 5), with wider confidence intervals reflecting small scenario counts and scenario diversity.

Experimental Results

The fine-grained results reveal the following core findings:

• Baseline difficulty is high: Under naive prompting, accuracy-at-3 is generally low, with most models performing below 37%. The best naive performance is observed in Claude Fable 5 (47.4%); several models are at or near chance (21–24%).
• Norm-informed prompting yields selective, strong improvements: Gemini 3.1 Pro exhibits an increase of +47.4 points (to 84.2%), and Claude Fable 5 increases by +34.2 points (to 81.6%) with norm-informed prompting. Other models—Claude Opus 4.8 (+10.5), Llama 3.3-70B (+7.9), Gemma 3-27B (+10.5)—display more muted gains. Qwen2.5-72B and GPT-5.5 are largely unchanged; Mistral Medium 3.1 actually regresses.
• Failure recovery is asymmetric across models: For Gemini 3.1 Pro and Claude Fable 5, most naive failures are recovered under norm-informed prompting, with no new regressions introduced. Conversely, other models show either limited recovery, significant regressions, or both.

  Figure 4: For Gemini 3.1 Pro and Claude Fable 5, gains are realized almost exclusively by recovering naive failures, with no new regressions. Other models have a mixed profile, often introducing regressions when norm-informed prompting is applied.

The effect is therefore both quantitatively strong (in select models) and qualitatively model-specific. Current prompting techniques cannot be assumed to produce universal improvements in local social adaptation; success is contingent on both model architecture and pretraining alignment.

Theoretical Implications and Benchmark Significance

LoSoNA isolates a specific, core element of social reasoning in LLMs: functional Theory of Mind and context-sensitive adaptation, rather than the more commonly tested explicit belief or social knowledge recall. The findings highlight explicit limitations of current LLMs—many top-tier models default to generic behaviors and fail to generalize beyond surface-level politeness even when transcripts provide clear precedent. Moreover, gains from direct norm-informing are non-uniform and can produce unintended regressions in some architectures.

From a theoretical perspective, the benchmark provides evidence that even very large models require explicit priming to access contextually relevant social adaptation strategies, and that the ability to meta-reason (e.g., "a norm is likely present—adapt accordingly") is an emergent property sensitive to both model and prompt design.

Limitations and Future Directions

The authors explicitly rule out broad claims about social intelligence: LoSoNA targets one-turn adaptation to a single focal norm, in synthetic, curated, English-language scenarios. It does not measure longitudinal adaptation, norm repair, or multi-norm inference in naturalistic, messy group chat.

Key limitations include:

• Screening with Gemini-family models may bias baseline and prompting headroom
• Use of an LLM judge could introduce bias, particularly if it shares architecture with subject models
• Current scale (38 curated scenarios) yields wide confidence intervals; per-norm and per-event analyses are not robustly supported

Anticipated future improvements involve naturalistic scenario expansion, multi-language coverage, multi-norm scenarios, and dynamic benchmarks allowing for sanctioning and repair over multiple conversational turns.

Practical and Societal Impact Considerations

Practical deployment of LLMs in group settings—moderation, collaboration, social assistance—will benefit from agents that reliably discern and adhere to local group norms. However, improved norm adherence also introduces dual-use concerns: better social mirroring can facilitate deceptive anthropomorphic behavior or enable adaptation to harmful norms. Disclosure, transparency, and principled deployment constraints are critical as these capabilities advance—aligning with evolving regulatory requirements [EU AI Act Article 50, NIST AI RMF].

Conclusion

LoSoNA represents a rigorous step toward evaluating LLM competence in local norm inference and adaptation. Strong numerical improvements under explicit norm-informed prompting are model-specific and non-uniform. The work demonstrates that current LLMs are not reliably adaptive to local social context unless carefully prompted, and that surface-level politeness is not a proxy for genuine context sensitivity. These findings have immediate implications for benchmarking socially intelligent agents and direct future alignment and evaluation research toward more robust and realistic group-interaction scenarios.