RoleConflictBench: Evaluating Social Dilemmas

• RoleConflictBench is a benchmark evaluating LLMs' ability to navigate ambiguous role-conflict dilemmas by varying role expectations and situational urgency.
• It employs a three-stage pipeline—from role expectation generation to urgency-based situation instantiation and story synthesis—to create realistic social scenarios.
• Empirical findings reveal that LLMs often favor stable role hierarchies and exhibit biases, indicating limited responsiveness to nuanced contextual cues.

Searching arXiv for the benchmark and closely related work. arXiv search query: RoleConflictBench contextual sensitivity role conflicts benchmark RoleConflictBench is a benchmark for evaluating whether LLMs can make socially appropriate decisions in ambiguous role-conflict dilemmas: situations in which one person simultaneously occupies two social roles whose expectations cannot both be satisfied at once. Rather than assuming a single prescriptive answer, it evaluates contextual sensitivity, defined as the ability to recognize and appropriately weigh situational cues that can alter which role should be prioritized, with situational urgency as the main controlled cue. The benchmark consists of 13,914 synthetic-but-audited stories spanning 65 social roles in five domains, and it was introduced to test whether model choices change appropriately as role expectations and urgency levels vary (Shin et al., 30 Sep 2025).

1. Conceptual scope and benchmark rationale

RoleConflictBench addresses a gap in LLM evaluation created by the mismatch between conventional social-reasoning benchmarks and real role conflicts. Prior evaluations often test social norms, commonsense social understanding, moral judgments, or socially appropriate dialogue in settings with a predefined correct answer. Role conflicts are treated differently: they are inherently ambiguous social dilemmas, and appropriateness depends on context rather than on a fixed gold label (Shin et al., 30 Sep 2025).

A role conflict scenario in the benchmark is a first-person story in which one person holds two roles whose expectations are incompatible at a given moment. The paper gives examples such as parent vs. researcher, friend vs. engineer, scientist vs. Muslim, and sibling vs. colleague. Each side of the dilemma is defined by a role, an associated expectation, a specific situation instantiating that expectation, and an urgency score. This design shifts evaluation away from “Did the model pick the right answer?” toward “Did the model’s preference shift appropriately when urgency and role expectations changed?” (Shin et al., 30 Sep 2025).

The central research problem is whether LLMs adapt their decisions to situational context rather than relying on fixed priors about roles. The paper’s core empirical claim is that many models appear somewhat responsive to contextual cues, but their decisions are governed predominantly by an inherent bias related to social roles rather than by situational information. This places RoleConflictBench at the intersection of social reasoning, bias analysis, and preference-structure probing rather than simple moral QA or norm-enforcement evaluation (Shin et al., 30 Sep 2025).

2. Three-stage construction pipeline

RoleConflictBench is built through a three-stage generation pipeline. In Stage 1: Expectation Generation, for each role $r \in R$, GPT-4.1 is prompted with temperature 0 to generate concise expectations: common social obligations or responsibilities associated with that role. The appendix prompt requests 10 expectations per role, while the final dataset description states that each role ends up with three concise role expectations. Expectations are written as single clauses and are human validated/audited for plausibility, neutrality, nonredundancy, and fidelity to common real-world obligations for the role (Shin et al., 30 Sep 2025).

In Stage 2: Situation Instantiation with Urgency Levels, each expectation is converted into concrete situations with urgency $u \in \{1,2,3\}$, where $u=1$ is light, routine, and easy to postpone, $u=2$ is moderately pressing, and $u=3$ is highly pressing and difficult to defer. The paper defines situational urgency as “the level of necessity indicating how strongly an expectation requires action in a given context.” Urgency is manipulated through factors such as rarity of the opportunity, time sensitivity, severity of need, stakes of delay, dependence of others on the actor, and whether the event can be rescheduled or delegated. Every generated situation is human reviewed to ensure alignment with its urgency level (Shin et al., 30 Sep 2025).

In Stage 3: Story Synthesis, two roles $r_i, r_j$ are sampled and combined into a first-person story. Each story contains Role 1 with its expectation and situation, Role 2 with its expectation and situation, emotional tension and inner conflict, and no final decision. The stories are generated with GPT-4.1 at temperature 0, are written in first-person perspective, are 100–200 words long, and are described as vivid but realistic. For every role pair, the benchmark generates the full $3 \times 3$ crossing of urgency combinations, producing all 9 pairwise urgency configurations from $1$ vs. $1$ through $3$ vs. $u \in \{1,2,3\}$0 (Shin et al., 30 Sep 2025).

This pipeline systematically varies both the type of obligation and its situational urgency. The benchmark therefore does not merely compare role names; it compares specific role-bound responsibilities under controlled contextual shifts. That design is methodologically important because it allows the evaluation to test whether model behavior changes with the scenario rather than with the role label alone (Shin et al., 30 Sep 2025).

3. Dataset organization, role inventory, and scenario semantics

The benchmark contains 65 social roles drawn from five domains and organized to maximize cross-domain conflict diversity. Roles are paired only across different domains, and pairing is further restricted so that the two roles share the same gender annotation. The result is 1,546 unique cross-domain role pairs, and with 9 stories per pair from the full urgency crossing, the final dataset contains 13,914 total role conflict stories (Shin et al., 30 Sep 2025).

Domain	Number of roles
Family	18
Occupation	24
Society	5
Interpersonal Relationship	8
Religion	10

The Family domain includes roles such as grandfather, grandmother, father, mother, son, daughter, brother, sister, husband, wife, grandparent, parent, child, sibling, step-parent, step-sibling, step-child, and spouse. The Occupation domain includes 24 occupations, sampled from the U.S. Bureau of Labor Statistics wage survey, with 12 occupations from the top third and 12 from the bottom third of the income distribution; examples include air traffic controller, police officer, subway operator, doctor, pharmacist, judge, lawyer, architect, engineer, accountant, software developer, scientist, ambulance driver, lifeguard, nursing assistant, housekeeping cleaner, construction laborer, carpenter, machine repairer, hairdresser, telemarketer, cashier, taxi driver, and delivery person. The Society domain contains volunteer, activist, citizen, voter, and advocate. The Interpersonal Relationship domain contains boyfriend, girlfriend, friend, mentor, mentee, colleague, acquaintance, and neighbor. The Religion domain contains pastor, christian, priest, nun, imam, muslim, rabbi, jewish, buddhist, and hindu (Shin et al., 30 Sep 2025).

Each benchmark instance is built from two role-expectation-situation bundles. The benchmark’s main contextual cue is urgency, but the underlying story semantics also include emotional tension and incompatibility of expectations. A context-sensitive model is expected to behave roughly as follows: if role $u \in \{1,2,3\}$1 has higher urgency than its competitor, it should usually choose $u \in \{1,2,3\}$2; if both roles have equal urgency, choices should be closer to balanced; if $u \in \{1,2,3\}$3 has lower urgency, it should usually not choose $u \in \{1,2,3\}$4. This operationalization is central to the benchmark because it defines success as appropriate responsiveness to context, not agreement with a hidden normative label (Shin et al., 30 Sep 2025).

The benchmark intentionally avoids trivial contrasts such as life-or-death versus minor inconvenience. Instead, urgency is crossed systematically on both sides of the dilemma so that the decision problem remains nontrivial. This suggests that the benchmark is designed to expose whether models can discriminate nuanced social tradeoffs rather than simply detect the more extreme option (Shin et al., 30 Sep 2025).

4. Decision task, annotation schema, and formal metrics

Given a story, the model is asked: “Which role should I prioritize more in this situation?” with two options, (A) role 1 and (B) role 2. The model is instructed to choose one option only, provide a brief reason, and identify the underlying human value. The appendix specifies a JSON output format:

$u=2$4

The value field is drawn from Schwartz’s ten basic human values: Self-direction, Stimulation, Hedonism, Achievement, Power, Security, Conformity, Tradition, Benevolence, and Universalism. The core evaluation, however, uses the binary choice rather than the rationale or value label (Shin et al., 30 Sep 2025).

RoleConflictBench does not assign a gold-correct label to each scenario. Instead, each item carries structured metadata including role 1, role 2, domain of each role, expectation for each role, situation for each role, urgency for each role, the generated story, and the binary decision question. Additional annotations support bias analysis, including domain, gender, kinship for family roles, income level for occupation roles, and religion group for religion roles (Shin et al., 30 Sep 2025).

The benchmark’s main quantitative measure is the Sensitivity score $u \in \{1,2,3\}$5, where lower is better. For a focal role $u \in \{1,2,3\}$6, the paper defines the urgency-conditioned win probabilities such as

$u \in \{1,2,3\}$7

and similarly for $u \in \{1,2,3\}$8 and $u \in \{1,2,3\}$9. Ideal urgency-sensitive behavior is specified as

$u=1$0

$u=1$1

$u=1$2

Deviation from this ideal is measured by

$u=1$3

for $u=1$4, and the overall sensitivity score is

$u=1$5

Lower $u=1$6 indicates stronger contextual sensitivity; higher $u=1$7 indicates weaker use of urgency cues (Shin et al., 30 Sep 2025).

To quantify stable role preferences, the paper fits a Bradley–Terry pairwise-comparison model and defines the Role Priority Index (RPI) by

$u=1$8

with log-likelihood

$u=1$9

and normalization

$u=2$0

The parameters are estimated using Zermelo’s fixed-point iteration with convergence threshold below $u=2$1. RPI is interpreted as the model’s latent preference hierarchy over roles, independent of any single story (Shin et al., 30 Sep 2025).

The paper also defines domain preference scores and group preference scores based on aggregated RPIs. For group preference, the text notes that the printed formula is garbled in the paper, but the intended definition is explicitly described as a normalized preference of one group over another based on average RPIs. This analytical layer is important because it separates context responsiveness from stable role priors (Shin et al., 30 Sep 2025).

5. Empirical findings

The benchmark evaluates 10 LLMs: GPT-4.1, GPT-4.1-mini, Gemini 2.5 Flash, Gemini 2.5 Flash-Lite, Qwen3-30B-Base, Qwen3-30B-SFT, Qwen3-30B-Instruct, OLMo2-32B-Base, OLMo2-32B-SFT, and OLMo2-32B-Instruct. Sensitivity scores $u=2$2, with lower better and values multiplied by 100 in the paper, are reported as 53.07 for GPT-4.1-mini, 46.28 for GPT-4.1, 49.19 for Gemini 2.5 Flash-Lite, 44.21 for Gemini 2.5 Flash, 44.62 for Qwen3-30B-Base, 51.20 for Qwen3-30B-SFT, 53.10 for Qwen3-30B-Instruct, 55.31 for OLMo2-32B-Base, 48.61 for OLMo2-32B-SFT, and 50.30 for OLMo2-32B-Instruct. The paper’s interpretation is that all tested models show some responsiveness to urgency, but that this sensitivity remains insufficient (Shin et al., 30 Sep 2025).

Model scale sometimes helps, but the effect is not uniform. GPT-4.1 outperformed GPT-4.1-mini, and Gemini 2.5 Flash outperformed Flash-Lite. Training effects are mixed: Qwen3 became worse after SFT and instruction tuning, while OLMo2 improved after SFT and then worsened somewhat with instruction tuning. The paper therefore does not identify a consistent training recipe that reliably fixes the problem (Shin et al., 30 Sep 2025).

Ablation results indicate that the benchmark’s social and contextual design materially changes decisions. In a three-way comparison among role only, role + expectation, and role + expectation + situation, GPT-4.1 showed mean absolute differences in pairwise win probabilities of 23.3 from 1→2, 16.3 from 2→3, and 23.9 from 1→3. GPT-4.1-mini showed 19.6, 15.2, and 20.9 for the same transitions. The paper interprets this as evidence that expectations and urgency-instantiated situations genuinely alter the decision space rather than merely paraphrase the same choice (Shin et al., 30 Sep 2025).

The central empirical result is that role identity matters more than urgency. Conditional win-rate plots show that a role is chosen more often when its urgency is higher than its competitor’s, but all three urgency-conditioned curves descend steeply according to a stable role ranking. High-priority roles remain strong even when they are at a relative urgency disadvantage, while low-priority roles remain weak even when their urgency is higher. The paper describes the pattern as a “steep and monotonic decrease” from high-ranked roles such as air traffic controller to low-ranked roles such as mentee (Shin et al., 30 Sep 2025).

Domain-level analysis shows strong preference toward Occupation and Family. For GPT-4.1 default, normalized domain preference is 16.4 for Family, 70.4 for Occupation, 6.2 for Society, 2.3 for Interpersonal Relationship, and 4.8 for Religion. Across many large models, roughly 70% of domain preference mass is assigned to Occupation. The paper interprets this as an implicit hierarchy in which professional and family obligations are systematically prioritized over wider social, relational, or spiritual commitments (Shin et al., 30 Sep 2025).

At the role level, GPT-4.1 ranked safety-critical occupations such as air traffic controller, ambulance driver, nursing assistant, and lifeguard highly, along with family roles such as parent, mother, father, and spouse. Qwen3-Instruct also favored safety-related and parental roles but placed religious roles higher than GPT-4.1 in parts of its ranking. The paper treats these rankings as evidence that models encode structured, interpretable social priorities rather than merely random noise (Shin et al., 30 Sep 2025).

6. Bias structure, perturbation results, and limitations

RoleConflictBench does not only measure weak contextual sensitivity; it also exposes systematic social bias. For GPT-4.1, male-gendered roles received 53.8% of preference mass versus 46.2% for female-gendered roles. Within Family, neutral roles received 36.2%, male roles 34.5%, and female roles 29.3%. Qwen3-Instruct showed a stronger male preference at 57.7%. The paper emphasizes that this is especially striking because family roles can be assigned identical expectations and situations, yet female roles are still deprioritized (Shin et al., 30 Sep 2025).

Religion analysis shows an Abrahamic-over-Dharmic pattern. For GPT-4.1, normalized preference shares are 29.1% for Christianity, 28.0% for Islam, 29.8% for Judaism, 9.7% for Hinduism, and 3.4% for Buddhism. Qwen3-Instruct exhibits the same broad pattern, although the internal ordering differs. Occupational status also matters: for GPT-4.1, high-income roles receive 57.9% versus 42.1% for low-income roles, while Qwen3-Instruct reverses this with 53.7% for low-income roles and 46.3% for high-income roles. Kinship bias is smaller for GPT-4.1 at 51.1% versus 48.9%, but stronger for Qwen3-Instruct at 67.2% versus 32.8% (Shin et al., 30 Sep 2025).

A further robustness test perturbs the prompt by adding a demographic self-description such as “As a {demographic attribute}, which role should I prioritize?” while leaving the underlying scenario unchanged. The paper reports that a single demographic cue often changes the answer. For GPT-4.1, “man” increases Family preference from 16.4% to 20.6%, “woman” reduces it to 14.0%, “Asian” increases it to 23.2%, and “Hispanic” to 22.6%, while White and Black remain lower at 17.1% and 17.5%. Sensitivity $u=2$3 becomes worse in all demographic-cue settings. Appendix examples show direct answer flips, including a scientist vs. imam scenario in which GPT-4.1 chooses imam as a man and scientist as a woman (Shin et al., 30 Sep 2025).

The collected rationales and value labels reveal a narrow mapping from role domains to values. For GPT-4.1, Family is overwhelmingly associated with Benevolence at 93%, Interpersonal Relationship with Benevolence at 85%, Occupation with Security at 50%, Society with Universalism at 50%, and Religion with Benevolence at 47% and Tradition at 23%. Stimulation, Hedonism, and Power are reported as absent. The paper interprets this as evidence of a narrow and rigid value mapping rather than nuanced pluralistic moral understanding (Shin et al., 30 Sep 2025).

Several limitations are explicit. The stories are synthetic, though human-audited. There is no human answer baseline, because the benchmark intentionally avoids adjudicating a correct answer. Contextual sensitivity is operationalized mainly through urgency and expectation variation, while real social judgment may depend on relationship history, institutional constraints, legal obligations, cultural setting, and available alternatives. The setup is a binary forced-choice task even though real role conflicts may permit compromise, sequencing, delegation, or hybrid solutions. The benchmark also holds gender constant in pair construction, which reduces confounds but may omit some mixed-gender interactions. Finally, at least one printed group-preference formula is corrupted in the provided text, although the intended meaning is recoverable (Shin et al., 30 Sep 2025).

7. Relation to adjacent benchmark families

RoleConflictBench occupies a distinct position within the current benchmark landscape because it focuses on ambiguous social role dilemmas without gold correctness labels, whereas several adjacent benchmarks formalize conflict, roles, or values in different ways. ConflictBench studies human–AI conflict in 150 multi-turn, interactive, optionally visually grounded scenarios centered on self-preservation, resource prioritization, deception, and regret under pressure; its evaluation target is agentic behavioral alignment rather than contextual sensitivity in short first-person social dilemmas (Zhao et al., 9 Mar 2026). Agent-ValueBench shifts value evaluation from text-only judgments to 394 executable environments and 4,335 value-conflict tasks scored at the trajectory level, emphasizing that agent values need not equal underlying LLM values (Dong et al., 11 May 2026).

Other nearby benchmarks make roles more explicit but operationalize them differently. TeamBench evaluates coordination under operating system-enforced role separation among Planner, Executor, and Verifier, showing that pass rate can mask role takeover and false certification under prompt-only role assignment (Kim et al., 8 May 2026). AgentCollabBench is a process-diagnostic benchmark of multi-agent collaboration that isolates instruction decay, false-belief contagion, context leakage, and tracer durability across communication topologies; it is structurally related to role conflict but does not formalize incompatible social role expectations as the primary variable (Mazumder et al., 9 May 2026). OrgAccess benchmarks reasoning under organizational permissions, overlapping or conflicting access constraints, and hierarchical rules, with Full Access, Partial Access, and Rejected Access labels rather than ambiguity-sensitive social judgments (Sanyal et al., 25 May 2025). RoleCDE is closer in spirit to RoleConflictBench because it evaluates role-playing agents under explicit role-versus-alignment dilemmas using a four-way decision taxonomy—Role-Following, Role-Compromise, Alignment-Compromise, and Alignment-Following—and reports the Decision Bias Ratio as a role-oriented preference measure (Lai et al., 1 Jun 2026).

Against this backdrop, RoleConflictBench’s distinctive contribution is methodological rather than agentic: it offers a large-scale, controlled evaluation of whether model choices shift appropriately when role expectations and urgency cues are varied, without presuming that one answer is universally correct. Its main empirical conclusion is correspondingly sharp. Current LLMs show some contextual sensitivity, but their decisions are dominated by stable role hierarchies and social biases, including preference for Occupation and Family, preference for male over female roles, preference for Abrahamic over Dharmic religious roles, and instability under demographic prompt perturbations (Shin et al., 30 Sep 2025).