Socialization Index (SI): Framework & Benchmarks

• Socialization Index (SI) is a framework defining interpersonal skills by integrating social-psychology theories to categorize both motivational orientations and process abilities.
• The SI evaluation paradigm employs a dual-axis approach, using goal achievement (GAE) and interpersonal ability (IAE) metrics to quantitatively assess social performance.
• Benchmarking with SocialEval utilizes scripted narrative scenarios to compare human and LLM performances, revealing significant gaps in adaptability and prosocial strategy application.

Socialization Index (SI) is defined as the set of interpersonal skills and competencies that enable an individual or agent to understand, manage, and adapt human behaviors in order to behave wisely in social interactions to achieve social goals. Recent operationalizations of SI are grounded in social-psychology theory and formalized in script-based evaluation frameworks, notably SocialEval, which decomposes SI into outcome-oriented achievement of social goals and the process-oriented exercise of interpersonal abilities. SI provides a structured approach to benchmarking both human and artificial agents, particularly LLMs, in their capacity for navigating complex social environments and exhibiting functional social intelligence (Zhou et al., 1 Jun 2025).

1. Theoretical Foundations of Socialization Index

SI builds upon a dual-component model, rooted in established social-psychological frameworks:

• Social Worlds (Orientations): Social goals arise from the interplay of self-interest and altruism, formalized as outcomes $(a, b)$ with $a, b \in \{-1, 0, 1\}$ representing self-interest and altruism, respectively. This Cartesian product generates nine theoretical orientations, with seven retained for operational purposes:
  • Prosocial: Cooperation $(1,1)$, Negotiation $(1,0)$, Assistance $(0,1)$, Altruism $(-1,1)$
  • Proself: Competition $(1,–1)$
  • Antisocial: Induction $(0,–1)$, Conflict $(–1,–1)$
• Interpersonal Abilities (Process Skills): Based on the BESSI framework, SI further decomposes into five high-level domains encompassing 32 fine-grained skills:
  • Social Engagement (e.g., leadership, persuasion, conversational skill)
  • Cooperation (e.g., teamwork, trust, perspective-taking)
  • Self-Management (e.g., task and time management, goal regulation)
  • Emotional Resilience (e.g., stress regulation, impulse control, optimism)
  • Innovation (e.g., creative thinking, cultural competence, abstract thinking)

This decomposition allows SI to systematically characterize both the motivational orientations underlying social exchange and the granular process skills implicated in effective social functioning (Zhou et al., 1 Jun 2025).

2. Operational Evaluation Paradigm

The unified SI evaluation paradigm is instantiated along two primary axes:

• Outcome-Oriented Goal Achievement Evaluation (GAE):
  • Formulated as a goal-conditioned Markov Decision Process (MDP), each step $t$ places the agent in a state $s_t$ corresponding to the current episode $\mathcal{E}_t$. The agent selects an utterance $a_t$ from a discrete set of alternatives $u_{t,1},\ldots,u_{t,m}$.
  • The transition function $F_T$ maps $(s_t, a_t) \rightarrow s_{t+1} = \mathcal{E}_{t+1}$, progressing the narrative along a world tree plot line.
  • A reward function $F_R$ yields $r_{t+1} \in \{0,1\}$, indicating social goal achievement.
  • The principal metric is the goal-achievement ratio: $$\text{GAE} = 100 \times \frac{\sum r_i}{\text{total plot lines}}$$.
• Process-Oriented Interpersonal Ability Evaluation (IAE):
  • At select episode transitions, each choice $u_{t,i}$ is designed to manifest a specific interpersonal ability.
  • A probing question $Q_{t,i}$ (e.g., “Which option best shows stress regulation?”) accompanies the alternatives; one is correct, others are distractors.
  • Correct selection is scored via $F_R(s_t, Q_{t,i}, U_{t,i}) \mapsto (u, r)$ with $r \in \{0,1\}$.
  • The core metric is ability-selection accuracy: $$\text{IAE} = 100 \times \frac{\sum r_i}{\text{total ability questions}}$$.

This dual-axis approach ensures comprehensive assessment of both high-level goal-directed behavior and the deployment of specific social competencies (Zhou et al., 1 Jun 2025).

3. Benchmarking with SocialEval

SocialEval is the canonical large-scale benchmark for SI assessment. It consists of 153 world trees, each a branching narrative scenario constructed as follows:

Component	Structure	Scope/statistics
Characters	Protagonist + supporting roles; public/private profiles	Each tree: multiple roles
Scenarios	Everyday and fictional social settings	Diverse sociocultural content
Episodes	Sequence of dialogue transitions (plot lines)	Avg. 6.5 episodes/tree
Decisions per Transition	2–3 candidate utterances manually crafted	2.17 choices on avg.
Plot Endings	Explicit annotation of goal success/failure	9.46 plot lines/tree
Probing Questions	MCQ for each decision, ability-linked, distractors given	2,493 samples over 32 abilities

Scripts are authored in Chinese and professionally translated to English (97% acceptance), establishing a bilingual evaluation corpus (Zhou et al., 1 Jun 2025).

4. Scoring Functions, Formulas, and Representational Analysis

SI’s evaluation metrics are derived from explicit mapping functions and provide both behavioral and representational insight:

• Goal Achievement Mapping:

  $s_{t+1} = F_T(s_t, a_t), ~ r_{t+1} = F_R(s_{t+1})$

• Interpersonal Ability Mapping:

  $(u, r) = F_R(s_t, Q_{t,i}, U_{t,i})$

• Aggregate SI Metrics:

  $$\text{GAE score} = 100\% \times \frac{\sum_i r_i}{\text{total plot lines}}$$

  $$\text{IAE score} = 100\% \times \frac{\sum_i r_i}{\text{total ability questions}}$$

• Representation and Neuron Analysis: The Wanda score is used to determine per-neuron functional importance for interpersonal abilities. Given a final token hidden state $h_T \in \mathbb{R}^{d_\text{in}}$ and MLP weight $W \in \mathbb{R}^{d_\text{out} \times d_\text{in}}$:

  $I(W) = |W| \odot (1 \cdot \|h_T\|_2^T)$

  where $\odot$ is elementwise multiplication and $1$ is the all-ones vector.

t-SNE projections reveal that as LLM size increases (8B → 70B), the five ability aspects become more segregated in embedding space, while neuron-importance maps indicate denser, functionally isolated regions analogous to cortical partitions (Zhou et al., 1 Jun 2025).

5. Empirical Findings and Behavioral Characteristics

Quantitative and qualitative analysis across LLMs and humans on SocialEval demonstrates several key trends:

• LLM vs. Human Performance: LLMs consistently underperform relative to humans:
  • GAE: Best LLMs score 47–53%, humans average 61–55% (gap ≥ 17–24%).
  • IAE: Best LLMs score 75–77%, humans 80–79% (gap ~4–5%).
• Model Category and Size: Open-source LLMs (DeepSeek-R1, Qwen-2.5-72B, Llama-3.1-70B) achieve slightly higher SI metrics than closed-source models; metrics scale with model size.
• World Orientation Effects: Both groups perform best in prosocial worlds, but LLMs exhibit substantial drops in proself and antisocial worlds, unlike humans, who adapt strategies flexibly.
• Cross-Lingual Variation: Significant discrepancies between Chinese and English prompt scores for LLMs mirror human cross-lingual variability (Wilcoxon $p < 0.001$).
• Prosocial Bias and Strategy Selection: LLMs show a strong bias toward positive/prosocial actions, even to the detriment of goal attainment. Humans proportionally employ neutral or negative strategies to achieve objectives.
• Role-Play Fidelity: In unconstrained generation, LLM outputs align with handcrafted choices at >70% semantic similarity and select the MCQ-matching option >80% of the time.

These results collectively indicate not only existing limitations in LLM social intelligence relative to human baselines but also emerging structure in LLM representations as a function of model scale (Zhou et al., 1 Jun 2025).

6. Implications and Research Trajectories

Systematic evaluation via SI and SocialEval advances understanding of both the behavioral and representational capacity of LLMs for social intelligence tasks. The sustained gaps in goal attainment and adaptive strategy deployment delineate current boundaries and deficiency modes, notably inflexibility in non-prosocial contexts and prosocial overgeneralization. Improved modeling of the underlying world-orientation dynamics and more sophisticated process-skill benchmarks are plausible future directions. The emergent clustering of ability-specific representations and neuron-level specializations with increased scale further suggests capacity for future advances in artificial agents’ social cognition, conditional on targeted training methodologies and expanded evaluative benchmarks (Zhou et al., 1 Jun 2025).