Challenges of Evaluating LLM Safety for User Welfare (2512.10687v1)

Abstract: Safety evaluations of LLMs typically focus on universal risks like dangerous capabilities or undesirable propensities. However, millions use LLMs for personal advice on high-stakes topics like finance and health, where harms are context-dependent rather than universal. While frameworks like the OECD's AI classification recognize the need to assess individual risks, user-welfare safety evaluations remain underdeveloped. We argue that developing such evaluations is non-trivial due to fundamental questions about accounting for user context in evaluation design. In this exploratory study, we evaluated advice on finance and health from GPT-5, Claude Sonnet 4, and Gemini 2.5 Pro across user profiles of varying vulnerability. First, we demonstrate that evaluators must have access to rich user context: identical LLM responses were rated significantly safer by context-blind evaluators than by those aware of user circumstances, with safety scores for high-vulnerability users dropping from safe (5/7) to somewhat unsafe (3/7). One might assume this gap could be addressed by creating realistic user prompts containing key contextual information. However, our second study challenges this: we rerun the evaluation on prompts containing context users report they would disclose, finding no significant improvement. Our work establishes that effective user-welfare safety evaluation requires evaluators to assess responses against diverse user profiles, as realistic user context disclosure alone proves insufficient, particularly for vulnerable populations. By demonstrating a methodology for context-aware evaluation, this study provides both a starting point for such assessments and foundational evidence that evaluating individual welfare demands approaches distinct from existing universal-risk frameworks. We publish our code and dataset to aid future developments.

• The paper demonstrates that context-blind evaluations overestimate LLM advice safety, especially for high-vulnerability users.
• It introduces a robust two-part evaluation protocol using a 7-point scale across financial and health advice domains.
• Empirical results show that even enriched context disclosures do not fully close the safety gap, urging a shift to evaluator-side, vulnerability-aware frameworks.

Evaluating LLM Safety Through a User Welfare Lens: Methodological Advances and Open Challenges

Motivation and Problem Definition

Contemporary safety evaluations of LLMs typically target universal, adversarial, and catastrophic risks, focusing on generalized threats such as cyber-offense, manipulation, deception, or sycophancy. However, such frameworks are insufficient for measuring context-dependent harms that arise in practical, high-stakes use cases, notably financial and health advice. The same LLM output can have dramatically variable safety implications depending on a user's specific circumstances and vulnerabilities.

Figure 1: The response of an LLM to an advice seeking question can pose different risks for different users depending on their context and vulnerability.

The authors formalize User Welfare Safety as the degree to which LLM-generated advice is safe for individuals when acted upon, minimizing harms based on user-specific context. This reframes safety as fundamentally profile- and context-dependent, contrasting with dominant universality-oriented approaches. The insufficiency of current evaluation protocols for actual deployment scenarios—especially for users with enhanced vulnerability—serves as the central impetus for this work.

Experimental Framework: Context-Blind vs Context-Aware Safety Assessment

The core methodological contribution is a two-part, exploratory evaluation protocol, empirically operationalized on advice-giving scenarios spanning finance and health domains.

Evaluation proceeds along two axes:

• Context-Blind assessment: A judge rates the safety of an LLM response for a generic user, with no access to demographic or vulnerability information.
• Context-Aware assessment: The same LLM judge appraises the response explicitly for an annotated user profile with realistic vulnerability factors.

  Figure 2: Evaluation methodology comparing context-blind and context-aware safety assessment of LLM responses with real domain professional user profiles.

LLM outputs are scored along a 7-point scale that synthesizes harm likelihood, harm severity, and the adequacy of suggested safeguards. The authors' iterative evaluation pipeline leverages advanced LLMs (GPT-5, Claude Sonnet 4, Gemini 2.5 Pro) for both advice generation and judgment, ensuring measurement consistency via templated, chain-of-thought prompts.

Figure 3: Context-aware safety rating scale (1–7), each point reflecting safety judged relative to user vulnerability and circumstances.

Findings: Robust Safety Gap Moderated by User Vulnerability

Empirical results demonstrate a systematic, statistically significant safety gap: context-blind evaluations overestimate the safety of LLM advice as user vulnerability increases. For high-vulnerability users, the average safety score drops from "safe" (5/7) to "somewhat unsafe" (3/7) on a 7-point scale when evaluated with user context.

Figure 4: Mean context-blind and context-aware safety scores by LLM for Health Advice and Financial Advice, evaluated for Baseline and stratified by vulnerability (low/medium/high).

Key points:

• For low-vulnerability users, context-blind metrics provide a conservative risk estimate.
• For medium vulnerability (especially in health), context-aware scores drop by ~1 point below context-blind.
• For high-vulnerability users, context-aware safety scores consistently show a two-point drop relative to the context-blind baseline across all LLMs and both domains.
• These findings are robust to choice of LLM (advice or judge), domain, and prompt.

Analyzing the Limits of Realistic User Context Disclosure

The authors further examine whether designing user prompts with increasingly realistic context disclosures (i.e., simulating what users would actually reveal) closes the safety gap. Prompts are systematically enriched with user information—one, three, or five context factors—ranked by either professional relevance to safety or user-likelihood of disclosure.

Findings show only a modest narrowing of the safety gap for high-vulnerability users under maximal, professionally-prioritized context disclosure. The gap remains persistent and significant: even detailed, "realistic" prompts do not substitute for full evaluator-side user profiling.

Figure 5: Context-blind (dashed) and context-aware (solid) safety scores across increasing numbers of context factors in prompts for Claude Sonnet 4 (Health top, Finance bottom), stratified by user vulnerability level.

Additional results for Gemini 2.5 Pro and GPT-5 confirm this pattern.

Figure 6: Context-blind and context-aware safety scores for Gemini 2.5 Pro mirroring the qualitative gap pattern across increasing prompt context and vulnerability.

Figure 7: GPT-5 replicates the persistence of the context-aware penalty for high-vulnerability user prompts.

Notably, there is no substantive improvement in safety scores based on factor ranking scheme: user-likelihood and professional-relevance disclosures yield similar effects, highlighting the challenge posed by the intersection of user privacy, behavioral realism, and model safety.

Theoretical and Practical Implications

Evaluation epistemology: These results expose a fundamental methodological challenge. Universal, context-agnostic benchmarks systematically mask population-specific risks, particularly for marginalized or high-vulnerability groups. Stratification by vulnerability—anchored in holistic user profiles—is essential for any meaningful safety claim about LLM deployment in advice scenarios.

Scalability: The demonstrated approach is not directly scalable in its current form due to the combinatorial explosion of relevant user profiles and context variables. Crowd-sourcing expert-annotated user profiles and leveraging regulatory mandates for researcher access are proposed as axes of progress, with the framework well-aligned for compliance with upcoming user-centric regulatory structures (e.g., EU DSA, OECD, NIST).

Evaluation automation: The study validates LLM-as-judge pipelines for structured safety assessment, but also highlights their dependence on high-quality, user-contextualized evaluation protocols. Rigorous human expert validation remains indispensable, especially as algorithmic judges are susceptible to systematic model error or bias.

Multi-turn, sustained interactions: The work is single-turn, and recognizes as a next step the importance of longitudinal, conversational trajectories and memory accumulation, which can exhibit compounding safety impacts dependent on user context evolution.

Framework generalizability: The results, while not immediately generalizable to all populations or settings, underscore the necessity of vulnerability stratification and context-aware evaluation for both academic rigor and regulatory compliance.

Conclusion

This paper provides foundational evidence that context-blind safety evaluation of LLM outputs is systematically insufficient for assessing real-world advice risks, particularly for high-vulnerability users. The persistent gap between context-blind and context-aware safety scores—robust across domains, models, and realistic context disclosure strategies—compels a paradigm shift towards evaluator-side, vulnerability-aware frameworks. Ensuring LLM advice safety as these systems become ubiquitous requires both methodological innovation and alignment with emerging socio-technical regulatory standards.

See ["Challenges of Evaluating LLM Safety for User Welfare" (2512.10687)] for experimental details, extensive methodological discussion, and dataset/code availability.