NICE Benchmark: Evaluating Social Intelligence
- NICE Benchmark is a theory-grounded diagnostic tool for evaluating the social intelligence of large language models, structured around Norm, Interaction, Cognition, and Experience.
- It employs a rigorous methodology combining systematic literature reviews, expert interviews, and psychometric validation with a ranking-based evaluation protocol.
- Empirical results reveal that while LLMs outperform humans in aggregate, they exhibit marked weaknesses in communication, highlighting critical areas for improvement in social interaction skills.
NICE is a theory-grounded diagnostic benchmark for evaluating the social intelligence of LLMs, introduced under the acronym Norm, Interaction, Cognition, Experience. It was designed to address a limitation of earlier social-intelligence evaluations: many are either narrowly focused on a single ability or broad but insufficiently structured for fine-grained diagnosis. NICE therefore combines a formal social-intelligence framework, psychometric validation, and a controlled ranking-based evaluation protocol. In its released form, the benchmark contains 137 items written in Simplified Chinese and operationalized through representative Chinese social contexts, with the stated aim of localizing model strengths and weaknesses at the level of dimensions and capability facets rather than only producing an aggregate score (Qi et al., 28 May 2026).
1. Framework and scope
NICE is organized around a social-intelligence framework constructed through a top-down literature review and bottom-up expert validation. The framework is anchored in social information processing theory and integrates classic theories of human social intelligence together with recent AI and social-intelligence reviews. Its final structure contains 4 categories, 11 dimensions, and 34 fine-grained facets (Qi et al., 28 May 2026).
| Category | Dimensions |
|---|---|
| Social Cognition | Social Perception; Social Understanding / Insight |
| Social Interaction | Communication; Emotional Utilization; Relationship Management; Self-consistency |
| Social Learning | Observational Imitation; Adaptive Learning |
| Social Norm | Sociocultural Intelligence; Social Responsibility; Moral / Ethical Intelligence |
The category structure is further mapped onto a processing pipeline. In the authors’ formulation, input corresponds to social cognition, output to social interaction, feedback to social learning, and normative regulation to social norm. This gives NICE a diagnostic organization that is more structured than a flat task collection.
At the facet level, the benchmark is intended to isolate specific capacities. Examples include identifying direct social meaning, integrating multiple cues, inferring unexpressed beliefs and intentions, sustaining multi-turn dialogue with cross-turn consistency, combining verbal and nonverbal skills, learning from feedback, and knowing explicit and implicit social rules. This facet decomposition is central to NICE’s diagnostic role, because performance deficits can be attributed to bounded constructs rather than to a diffuse notion of “social ability.”
2. Construction methodology and psychometric validation
NICE is presented as a full psychometric pipeline rather than a benchmark assembled only from preexisting prompts. Framework construction proceeded through four stages. First, the authors conducted a systematic literature review. Second, they carried out semi-structured expert interviews with 10 experts, refining 1,256 initial codes with thematic analysis; the reported inter-coder agreement was Kappa = 0.96. Third, they performed structured expert validation + Delphi-style refinement: 16 experts rated framework components on necessity, relevance, and discriminability using a 4-point Likert scale, the average coefficient of variation was 19.11%, and the item-level content validity index (I-CVI) reached 0.92. A focus group of 5 experts then resolved remaining disagreements and merged overlapping dimensions. Fourth, the benchmark applied AHP weighting based on pairwise comparisons from 9 experts, of whom 7 with consistency ratio below 0.1 were retained (Qi et al., 28 May 2026).
Item construction was likewise controlled. The benchmark items were human-written, not directly mined from corpora at scale. The rationale given is that direct corpus extraction often embeds too many overlapping cues, which would blur construct boundaries. As development references, the benchmark used 18 adaptable benchmarks selected from 42 candidate papers and 43 psychological tests or paradigms. From these materials and the framework, the authors created 137 items, each aligned to one specific capability facet.
The final item set underwent three rounds of review: internal evaluation by 2 trained interdisciplinary evaluators, external validation by 10 additional expert evaluators, and confirmation or re-scoring after revision. Items were rated for reliability, validity, and neutrality on a 5-point scale, with 3.5 as the retention threshold. The final benchmark retained 137 items, and all retained items satisfied the predefined quality requirements.
3. Item design and evaluation protocol
NICE uses a three-option ranking format rather than ordinary multiple choice. For each item, there is one optimal response, one suboptimal response, and one worst / boundary-violating response. The benchmark is therefore designed to test not only whether a model can identify a preferred answer, but also whether it can detect a quality gradient and recognize violations of social boundaries (Qi et al., 28 May 2026).
Each item contains a scenario, a question, and three candidate responses. During testing, the system samples one option from each of the optimal, suboptimal, and worst pools using a fixed random seed. A prediction is scored as correct only if the full 3-way permutation exactly matches the gold ranking; partial correctness receives no credit. Model outputs are parsed deterministically with a regex that extracts the first three distinct digits in .
The reported evaluation protocol is zero-shot with greedy decoding, temperature = 0, and top_p = 1. Each model was run on the full 137-item benchmark 3 times independently, for a total of 2,055 model calls. Reported statistics include overall accuracy, per-dimension accuracy, and cross-run sample standard deviation as a stability indicator. Confidence intervals are 95% bootstrap CIs computed by resampling the 137 items with 10,000 resamples. The human reference group consisted of 14 native Chinese adults with undergraduate-level or higher education who completed the same ranking task without special tutorials or feedback.
4. Measured abilities and diagnostic granularity
The benchmark’s diagnostic resolution is determined by the relation between dimensions and facets. The Communication dimension, for example, is broken down into appropriate expression style, combining verbal and nonverbal skills, conveying one’s own personality or preferences, emotional synchrony and interaction rhythm, and sustaining multi-turn dialogue with cross-turn consistency. Social Understanding / Insight targets inference of unexpressed beliefs, intentions, needs, and emotional states from context. Sociocultural Intelligence concerns explicit and implicit social rules together with ethnic and national cultural norms. Social Responsibility includes transparent reasoning and protection of privacy, confidential information, and sensitive information (Qi et al., 28 May 2026).
This structure matters because NICE is explicitly diagnostic rather than merely comparative. Each item is tied to a single capability facet, so a low score on a dimension can be decomposed into narrower failure modes. The paper uses this property to localize an overall weakness in communication to specific sub-capabilities rather than treating it as a monolithic defect.
The benchmark is also designed to expose boundary sensitivity. The “worst” response option is not just incorrect; it is intended to represent a socially inappropriate or boundary-violating behavior. That design choice makes the task sensitive to overacceptance of exaggerated politeness, improper disclosure, or contextually inappropriate deference. A plausible implication is that NICE tests normative calibration as well as recognition of the best response.
5. Empirical results
The reported evaluation covers 5 frontier LLMs—GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro Preview, DeepSeek-V4-Pro, and Qwen3.6-Plus—together with the human reference group. On aggregate accuracy, the LLMs scored higher than humans: LLMs: versus Humans: . The difference was reported as statistically significant with Welch’s , , . The best overall model scores were Gemini-3.1-pro-preview: 0.788 and GPT-5.5: 0.786, while Claude Opus 4.7 was lowest among the LLMs overall at 0.711 (Qi et al., 28 May 2026).
The benchmark’s principal diagnostic result is that Communication (D3) was the lowest-scoring dimension for all five models and the only dimension on which humans clearly outperformed the models. The human advantage on Communication was about 9.0 percentage points, with a confidence interval excluding zero. Humans also descriptively outperformed models on Sociocultural Intelligence (D9), though the confidence interval did not clearly exclude zero. By contrast, LLMs robustly outperformed humans on Social Responsibility (D10), Self-consistency (D6), and Emotional Utilization (D4).
Within Communication, the largest reported deficits were sharply localized. Relative to humans, LLMs lagged by 52.4% on multi-turn communication, 24.8% on nonverbal communication, and 10.5% on synchrony. The deficits were uneven across model families: some models achieved perfect performance on some facets and zero on others. The authors interpret this as evidence that communicative competence is not uniformly absent, but unevenly represented.
The paper also includes a qualitative failure case. In a first-meeting scenario, all models identified the polite response, but some did not mark an absurdly exaggerated bow as the worst option. The stated interpretation is that the models overvalued explicit deference and under-penalized boundary-violating behavior. This is consistent with NICE’s goal of measuring not only preference ordering but also recognition of socially inappropriate extremes.
6. Stability, interpretation, and limitations
NICE reports repeated-run stability rather than assuming single-pass performance is sufficient. Across the three repeated evaluations per model, 78.3% of model-item cells were either always correct or always incorrect. Under a binomial null, only 25% would be expected at the extremes, and the reported distribution was highly non-random: . The authors use this to argue that the Communication weakness is systematic rather than noise-driven (Qi et al., 28 May 2026).
The paper also positions NICE as more theory-grounded than earlier benchmark practice in four respects: a unified theoretical structure, item-level construct alignment, psychometric validation throughout the pipeline, and diagnostic rather than aggregate evaluation. On that view, NICE is not primarily a leaderboard instrument. It is an error-localization framework intended to expose socially consequential weaknesses that aggregate accuracy can obscure.
Two limitations are stated explicitly. First, NICE uses a static, text-based format, so it does not capture the full complexity of real-time social interaction. Second, the benchmark is operationalized in a Chinese cultural context, which improves contextual realism for that setting but limits immediate generalizability across cultures. The paper therefore cautions against treating high overall benchmark accuracy as evidence of robust real-world social intelligence or safety for socially sensitive applications such as companionship.
Future extensions proposed in the paper include more complex social scenarios, interactive settings, broader cultural contexts, and larger and more diverse human samples. This suggests that NICE should be understood as a diagnostic baseline within a broader evaluation program: it provides controlled, theory-organized measurement of social intelligence, but not an exhaustive account of social competence in deployment.