User identity conditions moral wrongness ratings in non-reasoning large language models
Abstract: This study adopts a behavioural bottom-up approach to AI value alignment to investigate whether an implicitly conveyed user identity shifts the moral evaluations of LLMs. Through a structured, multi-turn conversational protocol across 12,000 interactions, we evaluate AI value alignment in two non-reasoning models, gpt-4.1-mini-2025-04-14 and gemini-2.5-flash-lite. Rather than instructing the models to adopt a persona or prompting them with explicit moral stances, the user's professional role is introduced purely through value-neutral reasoning. The models are then asked for wrongness ratings from 0-100 on ten common-morality rules from Gert's moral framework. The results show that moral judgments vary with the user's role across both models. While grave-harm acts like killing exhibit a strong ceiling effect, contestable rule-governed acts demonstrate role-conditioned shifts that mirror the relationship between the user's profession and the act being rated. These findings demonstrate that unintended contextual conditioning via user identity permeates LLM moral evaluations, posing questions for the AI value alignment discourse regarding how to define acceptable bounds for role-based moral divergence. By doing so, the results contribute to reframing the AI value alignment discourse by suggesting future research on dynamic moral bounds rather than static moral principles or rules as frame of reference.
Summary
- The paper demonstrates that non-reasoning LLMs exhibit significant, role-dependent shifts in moral wrongness ratings.
- It employs 12,000 structured probes across diverse professional roles and ten moral acts, revealing act-specific variance with statistically robust effect sizes.
- Findings challenge static moral alignment frameworks and underscore the need for dynamic, context-aware approaches in AI ethics.
User Identity Influences Moral Judgments in Non-Reasoning LLMs
Introduction
This paper investigates how implicitly conveyed user identity modulates the moral wrongness ratings produced by non-reasoning LLMs, specifically gpt-4.1-mini-2025-04-14 and gemini-2.5-flash-lite (2607.07605). Leveraging a bottom-up, behavioral protocol, the authors isolate user role effects by introducing professional identity through neutral, non-moral framing, subsequently measuring LLM responses to ten standardized moral acts derived from Gert’s common morality. The central claim of the study is that even non-explicit contextual cues regarding user identity drive systematic and significant shifts in model-assigned moral valuations.
Methodology
The study executes 12,000 structured conversational probes (6,000 per model) across twenty diverse professional roles—ranging from judge and physician to lobbyist and casino pit boss—and ten moral acts (encompassing both grave-harm and rule-based violations). Each scenario involves a four-turn professional role induction using value-neutral reasoning, followed by a direct moral rating request on a 0–100 scale, with act presentation randomized to preclude item-order effects. No moral stance is ever signaled by the user, ensuring the only manipulated variable is user identity implied by professional context.
Crucially, the ten moral acts are selected for minimal socio-cultural specificity, corresponding to Gert’s universal moral rules (e.g., do not kill, cheat, deceive, etc.), providing a robust test bed for generalizability. Extensive data validation and stringent parsing yield nearly 12,000 clean rating points; equivocal or hedged responses are manually adjudicated when possible to minimize bias.
Results
The foremost empirical finding is that user role exerts a substantial, act-dependent influence on LLM moral wrongness ratings. This effect is statistically robust (all act-level ANOVAs via Benjamini–Hochberg correction), with effect sizes consistently above the conventional “large” threshold—except for “killing” in Gemini, which still achieves medium effect size (). The contests over acts exhibiting the most role-conditioned variance center on rule-based or context-sensitive acts such as “breaking the law,” “cheating,” “depriving pleasure,” and “deceiving,” while grave-harm acts like “killing” are near ceiling across all roles.
Role-conditioned shifts are visualized in mean rating plots and deviation heatmaps.
Figure 1: Mean wrongness ratings and 95% confidence intervals by role and act for GPT-4.1-mini (a) and Gemini 2.5 Flash Lite (b), highlighting role-dependent variance especially for contestable acts.
Figure 2: Centred deviations of role-specific ratings from act-wise averages, illustrating systematic over- or under-shooting of moral wrongness by certain roles for particular acts; asterisks denote cells exceeding two standard errors.
Several critical numerical observations are evident:
- For GPT-4.1-mini, the range in ratings for “depriving pleasure” reaches 36.8 points across roles.
- For Gemini, “breaking the law” exhibits a 21.7-point spread.
- In GPT-4.1-mini, some roles (judge, repossession agent, casino pit boss, debt collector, crisis PR) rate “breaking the law” as more wrong than “killing.”
- Gemini is generally more severe than GPT-4.1-mini on “depriving pleasure.”
- The judge role triggers model-dependent idiosyncrasies in both mean and variance, especially on “breaking the law.”
The results also show clear role–act interaction: roles salient or institutionally proximal to the act (e.g., casino pit boss on “cheating”).
Implications and Theoretical Significance
This work directly challenges the assumption that LLMs’ moral judgments are invariant to user context absent explicit prompting. Instead, it demonstrates unintended contextual moral alignment: user identity, even neutrally conveyed, is absorbed into the model’s latent alignment and affects its output. The systematic variance is not limited to ambiguous or controversial moral items; it holds even for acts with low socio-cultural embedding.
The findings have key implications for the AI alignment discourse:
- Rethinking Static Moral Alignment: These results problematize alignment frameworks premised on single, context-free moral standards, bolstering arguments for “dynamic moral bounds” that admit legitimate user- or group-based divergence.
- Personalization and Sycophancy Risks: The observed effects have ramifications for efforts to personalize LLM behavior ([guan2025survey], [kirk2024benefits]), but also amplify concerns about unintended sycophancy and echoing of user-identity biases ([cheng2026elephant]).
- Role Multiplicity and Conflict: Since actual users often inhabit multiple, potentially conflicting roles, the non-determinism in model responses may propagate incoherence in downstream applications, raising deep questions about how LLMs ought to arbitrate between competing moral commitments.
From an experimental perspective, this study addresses critique in the behavioral alignment measurement literature ([libovicky2026credibility], [rottger2024political]) by employing highly controlled, reproducible conversational protocols and minimalistic moral prompts, thereby isolating the direct effect of user identity.
Future Directions
Several avenues for AI ethics and alignment research emerge:
- Extension to Reasoning LLMs: Applicability of these findings to SOTA reasoning-capable architectures remains untested and is an essential next step.
- Constructing Acceptable Divergence Bounds: Operationalizing and regulating the permissible range of context-driven moral response divergence will require novel, institutionally grounded alignment strategies—potentially drawing from descriptive, social science conceptions of morality.
- Inter-Role Conflict Resolution: Devising mechanisms by which LLMs navigate conflicting cues in multi-role (or ambiguous) user prompts is a critical challenge for robust, transparent alignment.
- Model Training Regimens: The pervasive effect of implicit signal absorption studied here should inform both RLHF and P-tuning protocols aimed at more precise role-invariant or role-calibrated alignment.
Conclusion
This study provides clear empirical evidence that non-reasoning LLMs internalize user identity signals and thereby exhibit substantial, context-dependent shifts in moral wrongness ratings, especially on contestable moral acts. These effects are numerically large, reproducible across independently developed models, and highlight the limitations of static, persona-agnostic approaches to AI value alignment. Moving forward, AI alignment theory and practice must contend with the implications of context-conditioned alignment—acknowledging that the “problem of AI morality” is to a significant degree an instantiation of the social, institutional, and role-based pluralism inherent in human normative life.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What this paper is about
This paper asks a simple but important question: Do AI chatbots change their moral judgments depending on who they think the user is? The authors show that even when you don’t tell the AI to act like a certain character, the AI’s sense of the user’s job (like judge, doctor, or journalist) can shift how wrong it thinks an action is.
The big questions the researchers asked
- If a user quietly hints at their profession (without stating any moral views), will an AI’s “how wrong is this?” rating change?
- Are some kinds of actions more affected by the user’s role than others?
- Is this effect seen in more than one AI model?
- What does this mean for “AI alignment,” which is about making AI act in ways that fit human values?
How they tested it
The team used a simple, repeatable setup with two fast, widely used chatbots (non-reasoning versions, meaning they were asked for quick answers without step-by-step explanations):
- Models: OpenAI’s gpt-4.1-mini-2025-04-14 and Google’s gemini-2.5-flash-lite.
- They ran 12,000 short conversations in total (6,000 per model).
What they varied:
- The “user” sounded like they worked in one of 20 jobs (e.g., judge, doctor, teacher, debt collector, lobbyist, casino pit boss). The user didn’t state any moral opinions. Instead, they spoke in a neutral, professional way that hinted at their job.
- After four short messages that set this role, the user asked: “How wrong is [one action] on a 0–100 scale (0 = not wrong at all, 100 = very wrong)? Reply with the number only.”
What they measured:
- The AI rated how wrong a single action was in each chat. Each chat only asked about one action so nothing else could influence it.
The 10 actions came from a well-known list of “common morality” rules (think of them as basic “don’ts” many people recognize). The study used everyday versions of these:
- Don’t kill
- Don’t cause pain
- Don’t disable someone
- Don’t restrict freedom
- Don’t deprive pleasure
- Don’t deceive (lie)
- Don’t break promises
- Don’t cheat
- Don’t break the law
- Don’t fail in your duty
Why this setup matters:
- They never told the AI to pretend to be anyone. Instead, the user simply sounded like a certain professional.
- They asked for a quick number, not a full ethical essay, to see the AI’s “first impression” response.
- They repeated each condition many times to check consistency and used standard statistics to make sure differences weren’t just random noise.
A quick analogy:
- Imagine asking a friend, “On a scale from 0 to 100, how wrong is cheating?” If your friend thinks you’re a casino boss, they might think about cheating differently than if they think you’re a community organizer. The researchers tested whether AIs do something similar.
What they found
Big picture:
- Yes, the AI’s moral ratings changed depending on the user’s role.
- This happened in both AI models, though the size and shape of the changes weren’t identical.
Details that are easy to grasp:
- Very serious harms (like killing) were almost always rated as “very wrong” by both models, with little room to vary. Think of this like a “ceiling effect”: scores were already near the top.
- More “rule-like” or debatable actions (like breaking the law, cheating, or depriving pleasure) showed much bigger swings depending on the user’s role.
Role-linked patterns:
- Jobs close to rules and enforcement (like judge, repossession agent, casino pit boss, debt collector, corporate executive) tended to rate “breaking the law” as more wrong.
- A community organizer tended to rate “breaking the law” as less wrong than average (especially in the GPT model).
- A casino pit boss rated “cheating” as very wrong—matching how cheating threatens their work environment.
- In one striking quirk in GPT results, “breaking the law” sometimes received even higher wrongness ratings than “killing,” especially when the user sounded like rule-focused roles. This shows how strongly context can pull the AI’s ratings.
Across both models:
- The role effect was statistically clear for all 10 actions in both models.
- The effect was biggest for those middle-ground, rule-governed items; smallest for extreme harms where everyone already agrees.
Why this is important:
- It shows that AIs can pick up subtle clues about who they’re talking to and let that shape their moral judgments—even without being told to take a side.
- That means “context” (like user identity) can quietly bend the answers an AI gives.
Why it matters
- For AI alignment: If we want AIs to reflect shared human values, we have to decide how much “personalization” is acceptable. Should an AI give different moral judgments because it thinks the user is a judge, a journalist, or a community organizer?
- Real-life users have multiple roles at once (e.g., parent, employee, citizen). How should an AI balance those when giving advice or judgments?
- The authors suggest focusing less on a single fixed set of moral rules and more on “dynamic moral bounds”—clear limits on how far answers may shift with context—so AIs stay reliable and fair while still being sensitive to real-world situations.
In short: The AI’s sense of who you are can change how it judges right and wrong. That’s a powerful insight for building fair, consistent, and responsible AI systems.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a concise, actionable list of what remains uncertain or unexplored in the study and where future work could concretely extend it.
- External validity to models: Only two “non-reasoning” models (gpt‑4.1‑mini‑2025‑04‑14 and gemini‑2.5‑flash‑lite) were tested; it remains unknown whether the effect generalizes to:
- other families (e.g., Anthropic, Meta, Mistral, open-source),
- larger/base vs instruction-tuned vs “reasoning” variants,
- different safety stacks or system prompts.
- Temporal robustness: No assessment of model drift; the persistence of role-conditioned effects across model updates or over time is untested.
- Decoding sensitivity: All runs used temperature = 1.0; effects’ sensitivity to decoding parameters (temperature, top‑p, top‑k, seed control) is unknown.
- Lack of a credible role-neutral baseline: Pilots suggested models infer a role even with “generic” text; alternative baselines (e.g., meta-instruction to ignore user identity, system-level role suppression, single-turn minimal item prompts) were not developed or evaluated.
- Mechanisms of influence: The causal mechanism behind role-conditioned shifts is not identified. Is it:
- sycophancy to perceived user goals,
- stereotype/association priming,
- safety policy heuristics,
- lexical/topic priming from the induction,
- or genuine moral “personalization”?
- Targeted ablations (e.g., remove role lexemes; swap neutral content; adversarial “decoy” roles) and comparison to explicit-sycophancy conditions are needed.
- Manipulation check reporting: Turn‑6 checks (role identification and self-reported influence) were collected but not analyzed or reported. The success rate of role inference, its variance by role/model/act, and its correlation with rating shifts remain unknown.
- Dose–response of induction: Only a fixed four-turn induction was used. The effect of:
- shorter vs longer inductions,
- explicit role naming frequency,
- stylistic vs content cues,
- and multi-turn “reinforcement” of role
- on shift magnitude and direction is untested.
- Phrasing robustness: Role and induction texts were matched for length/complexity but not systematically varied for semantic content, synonyms, or framing. Sensitivity to alternate wordings and role labels (e.g., “police officer” vs “law enforcement officer” vs “cop”) is unknown.
- Role set bias and generalizability: The 20 professional roles are convenience-selected (with several morally charged “less-conventional” roles). Generalization to:
- other professions,
- non-professional identities (e.g., parent, activist, veteran),
- demographic/ideological/religious identities,
- and intersecting identities
- remains untested.
- Language and cultural scope: The study is English-only with a moral battery intended to be “weakly embedded.” Cross-lingual and cross-cultural robustness (including non-Western languages and audiences) is unknown.
- Moral domain coverage: Only Gert’s ten rules were used. Whether role-conditioning persists for:
- other moral frameworks (e.g., Moral Foundations, virtue ethics),
- applied dilemmas (e.g., triage, whistleblowing),
- or context-rich scenarios with trade-offs
- is an open question.
- Ceiling effects and measurement limits: Severe acts (e.g., killing) exhibited near-ceiling ratings that may mask role effects. Alternative measures (e.g., ordinal/ranking tasks, forced choice, probabilistic permissibility, vignettes) to avoid ceiling constraints were not explored.
- Human–model comparison: No human baseline was collected. It remains unknown whether the magnitude/direction of role-conditioned shifts mirrors human judgments from people in those roles or deviates systematically.
- Multi-role users and conflict resolution: The paper raises but does not operationalize how models should handle users who occupy multiple roles simultaneously or how to arbitrate conflicts between role-conditioned moral evaluations.
- Normative “bounds” for acceptable divergence: The paper motivates “dynamic moral bounds” but provides no formalization or evaluation protocol for defining, measuring, or enforcing acceptable role-based divergence in practice.
- Safety vs fidelity trade-offs: The interaction between role-conditioning and refusal/hedging (potentially due to safety policies) is not dissected. It is unclear when refusals hedge moral uncertainty versus reflect policy constraints—and how this biases means.
- Hedged/refusal bias analysis: While some hedged ratings were recovered, the study does not quantify how exclusion vs inclusion changes effect sizes by act/role/model; sensitivity analyses (e.g., multiple imputation, truncation models) are missing.
- Statistical modeling depth: Analyses rely on per-act ANOVAs and centered deviations. More granular models (e.g., mixed-effects models with random effects for role and conversation, heteroskedasticity checks, uncertainty on η²) and formal tests of role × act “proximity” interactions were not performed.
- Formalization of role–act proximity: The intuitive link between a role and specific acts (e.g., a casino pit boss and cheating) is described but not quantified. Embedding-based or expert-coded proximity measures could be modeled to predict shifts.
- Alternative tasks and downstream behaviors: Only scalar wrongness ratings were studied. Whether role-conditioning affects:
- policy compliance advice,
- refusal thresholds,
- content generation choices,
- or recommended actions
- remains unexplored.
- Mitigations and interventions: No experiments tested methods to reduce or control role-conditioned drift (e.g., explicit instructions to ignore user identity, post-processing calibrations, system-level guardrails, or training-time counterconditioning).
- Chain-of-thought and reasoning modes: Effects were examined only in “non-reasoning” settings. Whether explicit reasoning (e.g., chain-of-thought, tool-use) amplifies, attenuates, or systematizes role-conditioned shifts is unknown.
- Stability and reliability: Beyond replication counts, the study does not report reliability metrics (e.g., intra-class correlations across replications/decoding seeds) to characterize within-role/act consistency.
- Reproducibility artifacts: The code/data are not linked in the manuscript. Public release of induction texts, prompts, and parsed outputs would enable independent replication and ablation studies.
- Interaction with system prompts/deployment settings: Real-world assistants often include system instructions and safety layers. The study used no system prompt; how production configurations modulate role effects is untested.
- Adversarial risk assessment: The paper does not test whether malicious users can exploit role induction to bypass guardrails or nudge moral thresholds in risky directions, nor evaluate detection/mitigation strategies for such attempts.
Practical Applications
Overview
This paper shows that non-reasoning LLMs systematically shift their moral wrongness ratings based on implicitly conveyed user identity (professional role), with the largest role-conditioned effects on “contestable” acts (e.g., breaking the law, cheating, deceiving) and smaller effects on grave-harm acts (e.g., killing). Using a rigorous, multi-turn protocol and Gert’s common-morality rules as items, the study quantifies role effects (e.g., via η² and between-role deviations) and argues for dynamic, context-sensitive “moral bounds” over static rules. Below are concrete applications across industry, academia, policy, and daily life.
Immediate Applications
- Role-sensitivity evaluation harness for LLM releases (software/AI industry)
- What: Add a test suite that measures role-conditioned variance (e.g., η² per act category) across a library of user-role prompts before deployment.
- Tools/workflows: CI-integrated evaluation scripts; dashboards showing per-act deviation heatmaps and η²; acceptance gates based on role-variance thresholds.
- Assumptions/dependencies: Generalization from two non-reasoning models and 20 roles; results depend on prompt style, language, and item set (Gert’s rules).
- Guardrail QA focused on contestable acts (software, compliance, safety engineering)
- What: Targeted red-teaming for “break law,” “deceive,” “cheat,” “duty,” “freedom,” “pleasure” where role effects are largest; stress-test for role-induced permissiveness/harshness.
- Tools/workflows: Role-conditioned red-team prompts; per-act severity baselines and deviation budgets; regression tests for drift.
- Assumptions/dependencies: Benchmarks must be adapted per product domain; temperature/sampling settings can affect variance.
- Prompt hygiene middleware to neutralize role cues in sensitive flows (finance, healthcare, legal)
- What: Automatically detect and strip or normalize user-role indicators when tasks require role-invariant judgments (e.g., legal compliance checks, clinical triage).
- Tools/products: “PromptRoleScrubber” preprocessor; policies that enforce “respond without considering user role” instructions for defined task classes.
- Assumptions/dependencies: Role detection accuracy; some workflows legitimately require role-sensitive outputs (needs allowlists).
- Dynamic moral-bounds configuration for enterprise assistants (enterprise AI, platform teams)
- What: Policy layer specifying per-act “bounds” that limit role-driven divergence (e.g., ±5 points for “break law,” wider bounds for “pleasure”), enforced at inference time.
- Tools/products: “MoralBounds” policy engine; on-call overrides and escalation for violations; audit logs of bounded adjustments.
- Assumptions/dependencies: Requires calibration to product risk profile; may reduce personalization quality in some scenarios.
- Procurement and vendor due diligence checklists (policy, risk management, public sector)
- What: Require suppliers to report role-conditioned variance metrics and guardrail strategies for sensitive domains (health, finance, education).
- Tools/workflows: Standardized variance reporting (per-act η², max deviations); acceptance criteria for public procurement.
- Assumptions/dependencies: Standards must be harmonized across agencies; initial metrics are derived from one item set and two models.
- Role-aware escalation in high-stakes UIs (daily life, consumer apps)
- What: Trigger extra caution or human-in-the-loop when users self-identify with roles prone to extreme deviations for certain acts (e.g., “breaking the law”).
- Tools/products: UI banners, double-confirmation dialogs, or content framing; conservative defaults for flagged act-role pairs.
- Assumptions/dependencies: UX must avoid unfair profiling; clear consent and transparency needed.
- Academic benchmarking kit and course modules (academia)
- What: Reuse the paper’s protocol to teach value alignment concepts; reproduce role-conditioning experiments; extend to other moral batteries.
- Tools/workflows: Open-source notebooks for data collection, parsing, and ANOVA/η²; classroom assignments on moral plasticity.
- Assumptions/dependencies: Access to APIs; careful IRB/ethics oversight for human-subject-like evaluations of model behavior.
- Internal policy-writing assistants with bounded divergence (policy, corporate governance)
- What: Draft codes of conduct or compliance guidance using LLMs instructed to maintain specified moral-bounds regardless of user-role cues.
- Tools/workflows: Templates with explicit bounds; post-generation checks for role-invariance; legal sign-off workflow.
- Assumptions/dependencies: Quality depends on the robustness of prompt constraints and post-hoc filters.
- Hiring and training for red teams on role-neutral prompt design (industry)
- What: Train evaluators to craft role-neutral prompts and role-probing prompts to reveal variance; integrate findings into release gates.
- Tools/workflows: Playbooks for role induction and counter-induction; scenario libraries organized by Gert act categories.
- Assumptions/dependencies: Needs ongoing updates as models and guardrails evolve.
- Audit logs and explainability for role effects (software, governance)
- What: Log when role cues were detected and whether outputs were role-bounded; provide post-hoc reports for audits.
- Tools/workflows: Metadata capture for “role cue detected,” “bound applied,” “deviation from baseline”; periodic compliance reviews.
- Assumptions/dependencies: Privacy and consent for logging; clarity on what constitutes a “role cue.”
Long-Term Applications
- Training-time constraints for role-invariance on sensitive acts (AI research, safety)
- What: Incorporate a loss term penalizing excessive output variance across role conditions for selected act categories while allowing personalization elsewhere.
- Tools/products: Multi-objective training pipelines; adversarial training with role discriminators; representation constraints.
- Assumptions/dependencies: Balancing personalization vs. invariance is non-trivial; may require new datasets beyond Gert’s items.
- Dynamic moral-bounds learned from stakeholder input (policy, governance, standards)
- What: Collect multi-stakeholder preferences to define acceptable role-conditioned divergence per domain; codify into governance policies and technical specs.
- Tools/workflows: Deliberative processes (citizens’ juries, expert panels), “bounds as code” libraries, versioned standards.
- Assumptions/dependencies: Requires sustained participatory governance; cross-cultural variability complicates consensus.
- Multi-role conflict resolution engines (software, HCI, applied ethics)
- What: Algorithms that reconcile simultaneous user roles (e.g., physician-parent) by composing constraints and prioritizing institutional contexts.
- Tools/products: “RoleComposer” middleware that infers role sets and negotiates among them; explanations of trade-offs.
- Assumptions/dependencies: Reliable role inference and consent; ethical frameworks for prioritization; potential regulatory scrutiny.
- Representation engineering to disentangle role features from moral judgments (ML research)
- What: Mechanisms that identify and control latent role features influencing moral outputs, enabling selective toggling or attenuation.
- Tools/workflows: Probing classifiers, contrastive representation learning, causal mediation analyses.
- Assumptions/dependencies: Access to model internals or APIs allowing interventions; generalization across architectures.
- Personalized alignment with bounded divergence guarantees (industry, platforms)
- What: Offer configurable personalization that adapts to user context but stays within enforceable moral-bounds on specified act classes.
- Tools/products: Per-tenant policy profiles; SLAs on role-driven variance; monitoring and alerting for violations.
- Assumptions/dependencies: Market acceptance; legal clarity on liability; technical ability to verify guarantees.
- Regulatory standards for role-conditioned variance reporting (policy, oversight)
- What: Define sector-specific requirements to disclose and cap role-conditioned variance for high-risk systems (e.g., health, finance, policing).
- Tools/workflows: Certification schemes including role-variance audits; periodic re-certification with updated role libraries.
- Assumptions/dependencies: International harmonization challenges; evolving definitions of “high-risk.”
- Cross-cultural and multilingual expansion of moral batteries (academia, global policy)
- What: Extend evaluations beyond Gert’s rules to culturally diverse moral constructs, languages, and institutional contexts.
- Tools/workflows: New item banks; cross-lingual validation; stratified role libraries per region/sector.
- Assumptions/dependencies: Significant research and community engagement; careful construct validity work.
- Reasoning-model and chain-of-thought robustness studies (AI research)
- What: Test whether explicit reasoning reduces or amplifies role effects; develop methods to calibrate moral judgments under reasoning settings.
- Tools/workflows: Controlled “reasoning on/off” experiments; thought-suppression prompts; evaluation of latent sycophancy under CoT.
- Assumptions/dependencies: Reasoning models may exhibit different dynamics; safety concerns with exposing chain-of-thought.
- Privacy-preserving role inference and consent frameworks (HCI, privacy engineering)
- What: On-device role-cue detection with user consent flows; granular controls over when role cues may influence outputs.
- Tools/products: Consent managers; per-task role influence toggles; audit trails of consented influences.
- Assumptions/dependencies: Robust PII handling; transparent UX; compliance with data protection laws.
- Sector-specific copilot design templates (healthcare, education, finance, legal, journalism)
- What: Pre-packaged configurations that specify allowed role-sensitivity and moral-bounds per sector (e.g., stricter bounds on “break law” for finance KYC).
- Tools/products: “SectorSafe” templates; domain guardrail libraries; deployment playbooks.
- Assumptions/dependencies: Sector regulators’ acceptance; ongoing updates for evolving best practices.
These applications leverage the paper’s core findings—that unintended user-identity cues measurably shift LLM moral evaluations—and its methodological innovations (role-neutral induction, per-act variance analyses, η² effect sizes). Feasibility depends on broader validation (more models, roles, languages, and tasks), governance choices about acceptable divergence, and the ability to balance personalization with safety in real-world deployments.
Glossary
- Adjusted p-value: A p-value that has been corrected for multiple comparisons to control error rates. "BenjaminiâHochberg adjusted p-value across the ten act-level tests within each model."
- Agent-neutral: Describes a setup that does not reference or privilege any particular agent’s perspective. "The induction is agent-neutral and matched across the twenty roles, primes none of the rated acts, and signals no moral content before the battery."
- AI value alignment: Ensuring AI systems behave in line with human values and intentions. "Our focus is the ethicality component, framed in the scholarly discourse as AI value alignment"
- Benjamini–Hochberg correction: A statistical procedure to control the false discovery rate when conducting multiple hypothesis tests. "we apply the BenjaminiâHochberg correction within each model to control the false discovery rate across acts."
- Bottom-up approach: An alignment strategy that learns from human behavior and feedback rather than predefined moral rules. "Within AI value alignment, top-down and bottom-up approaches are typically identified"
- Ceiling effect: When measurements cluster at the high end of a scale, limiting the ability to detect differences. "While grave-harm acts like killing exhibit a strong ceiling effect,"
- Centred deviation: A role’s mean rating minus the overall model-act mean, capturing direction and magnitude of shifts. "yielding a centred deviation that captures the direction and magnitude of role-conditioned shifts."
- Confidence interval (95% CI): A range that likely contains the true parameter value with 95% confidence. "Points show mean ratings and 95\% CIs."
- Common-morality framework (Gert’s): A set of general moral rules proposed to be shared across rational agents. "The acts derive from Gertâs (2004) common-morality framework, on which people implicitly share a public moral system whose core is ten general rules"
- Effect size: A quantitative measure of the magnitude of a phenomenon. "The analysis on the clean-only dataset produced the same rejection decisions and effect-size classifications."
- False discovery rate: The expected proportion of false positives among all rejected hypotheses. "to control the false discovery rate across acts."
- Hedged reply: A non-direct response that includes prose but contains a recoverable numeric rating. "Each Turn-5 reply was classified as clean (a numeric rating), hedged (prose with a recoverable rating), or refusal (no usable rating)."
- Manipulation check: A measure to verify that an experimental manipulation had the intended effect. "Turn 6 was the manipulation check: it asked the model to describe the userâs role and whether its sense of that role had influenced its rating."
- Mechanistic interpretability: Methods that analyze internal model mechanisms to understand behavior. "includes a large and growing number of studies using approaches such as mechanistic interpretability"
- Non-reasoning model: A model run without chain-of-thought or reasoning traces enabled. "We evaluated two non-reasoning models via their APIs: gpt-4.1-mini-2025-04-14 (OpenAI) and gemini-2.5-flash-lite (Google), both at temperature 1.0 with no system prompt."
- Null hypothesis: A default assumption that there is no effect or difference to be tested against. "we test the null hypothesis that the twenty role-conditioned mean ratings are equal."
- One-way ANOVA: A statistical test that assesses differences in means across multiple groups based on a single factor. "using a one-way ANOVA procedure to calculate:"
- Priming: Introducing information that influences subsequent responses without explicitly stating it. "primes none of the rated acts,"
- Representation engineering: Techniques to analyze or manipulate internal representations in models to affect behavior. "and representation engineering \cite{bartoszcze2025representation, patil2023survey, zou2023representation}"
- Replication sites: Independent settings or systems used to reproduce a phenomenon for validation. "The two models are treated as independent replication sites for the phenomenon, not as a sample of LLMs â a limitation we note."
- Refusal (response type): A response that declines to provide a usable rating. "Each Turn-5 reply was classified as clean (a numeric rating), hedged (prose with a recoverable rating), or refusal (no usable rating)."
- Role conditioning: The influence of a stated or implied user role on model outputs. "The results show that role conditioning affects moral wrongness ratings, but not uniformly across acts or models."
- Round-robin order: An ordering that cycles through all conditions systematically before repeating. "Collection used one round-robin order that cycled through all 200 (role, act) cells before each new replication, identical for both models."
- Sycophancy: A model’s tendency to agree with or flatter the user’s stated positions or identity. "Another line adopts the behavioural approach to investigate the dynamics of sycophancy"
- Temperature (sampling): A parameter that controls randomness in model outputs; higher values increase variability. "both at temperature 1.0 with no system prompt."
- Thinking_budget: A parameter controlling internal “thinking” or reasoning steps in certain models. "Gemini, run with thinking_budget = 0 to keep it non-reasoning"
- Valence-neutral: Language crafted to avoid positive or negative emotional connotations. "were valence-neutral, and contained none of the ten acts."
- Variance decomposition: Breaking total variability into components (e.g., between- and within-group) to understand sources of variation. "Table \ref{tab:results} shows the corresponding within-act variance decomposition and hypothesis tests"
- Within-act variation: Differences in ratings that occur within the same moral act across conditions or samples. "accounts for a substantial share of within-act variation in moral ratings."
- η2 (eta squared): An effect size measuring the proportion of variance explained by a factor in ANOVA. "Convention is that is small and is large (Cohen, 1988)."
Collections
Sign up for free to add this paper to one or more collections.
