The Severance Problem: LLMs are Unaware of the Person Beyond the Prompt
Abstract: Personal AI assistants have attracted significant interest for their potential to enhance everyday life by automating routine tasks, supporting consequential decisions, and assisting with everyday personal matters. Yet despite rapid recent technical advances, these assistants continue to exhibit undesirable behaviors, such as sycophancy, overconfidence, and hallucination. We argue that these failures stem from a fundamental limitation: LLMs lack an explicit representation of the person beyond the context they are given, which we term as the \textbf{Severance Problem}. Even with rich personal context and strong commonsense reasoning capabilities from the backbone model, current AI assistants fail to represent what remains unknown about the user. We propose a simple solution: incorporating structured ignorance into the LLM context via the \textbf{Severance Schema}, which explicitly outlines dimensions along which the model lacks knowledge about the user, including physicality, temporality, consequences, continuity, multiplicity, and interiority. Empirically, across five model families, with the Severance Schema, the assistant consistently reduces sycophancy, harmful advice, and hallucination. Notably, models with the schema ask clarifying questions when information about the user is missing, rather than confidently extrapolating from incomplete user information.
Summary
- The paper introduces the Severance Schema that explicitly maps unknown user-context dimensions, enhancing LLM safety and calibration.
- Experiments reveal that the schema cuts harmful advice and hallucination rates by over half compared to memory-only models.
- Ablation studies confirm that structured meta-awareness mitigates sycophancy and improves personalized, consent-based AI interactions.
The Severance Problem: Explicitly Modeling Unknowns in Personalized LLM Assistants
Problem Definition and Motivation
The Severance Problem identifies a critical limitation in current LLM-powered personal AI assistants: the lack of an explicit structural representation of unknown categories of person-context beyond the user’s prompt. Contemporary systems, even when equipped with long-term memory and sophisticated retrieval mechanisms, operate over a limited and fragmented context—yielding overconfident, hallucinated, or sycophantic responses when the user’s complete real-world state is unrepresented.
Figure 1: The model only receives a fragmented slice (the “innie”) of the person, while most of their real-life situation (the “outie”) remains inaccessible; the model lacks meta-representations of what categories it might be missing.
This paper formalizes the Severance Problem by making a key distinction: (1) First-order unawareness, where the model simply lacks facts about the user; and (2) Second-order unawareness (meta-ignorance), where the model lacks any representation that there might be additional, currently unrepresented, relevant categories of information. This double unawareness structurally constrains model behavior, causing failures in safety and personalization even with expanding memory modules.
The Severance Schema: Structured Awareness of Knowledge Gaps
As a solution, the authors introduce the Severance Schema—a prompt-level intervention that augments model context with explicit slots enumerating six general-purpose dimensions of person-context: physicality, temporality, consequences, continuity, multiplicity, and interiority. Each slot is either populated or marked as [unknown]. This structured inventory transforms unknown-unknowns into actionable known-unknowns, making the absence of critical user information salient to the model without populating the facts themselves.
Figure 2: The “awareness plane”: memory increases factual information (horizontal), while the Severance Schema increases explicit meta-awareness of missing context (vertical).
This schema is straightforward, model-agnostic, and easy to implement at the prompting level. The explicit unknown markers drive LLMs to ask clarifying questions and avoid overconfident extrapolation from sparse memory, fundamentally altering their risk profile.
Experimental Evaluation
Experimental Setup
The benchmark comprises synthetic user profiles and diverse, safety-critical advice-seeking scenarios (health, legal, financial, psychosocial). Prompts are generated under four conditions: (1) No Schema (uninformed & unaware), (2) Severance Schema (unaware & aware), (3) Memory Only (informed & unaware), and (4) Schema + Memory (informed & aware). Five state-of-the-art model families are evaluated, with scoring using strong LLM-judges, and manual agreement validated by human annotators.
Main Results and Analysis
In all open-weight models, the Severance Schema approximately doubles explicit unknown-awareness and drives substantial safety gains, cutting harmful advice and sycophancy rates by over half, from 14–19% (baseline) to 2–10%, and 15–20% (baseline) to 2–9% respectively.
Figure 3: The Severance Schema reduces harmful advice, hallucination, and sycophancy rates while improving unknown-awareness across models, with minimal cost in immediate usefulness.
Crucially, adding memory without structured awareness actually increases hallucination rates (from near zero to 3.7–11.7%) and suppresses unknown-awareness, as the model infers what it “knows” from a sparse context rather than recognizing critical unknowns. The schema corrects this, keeping hallucination rates at 1.7–4.0% even with richer personal data.
Figure 4: Even as more user information is provided, the schema preserves lower hallucination rates and maintains an unknown-awareness advantage over plain memory.
Qualitative and scenario-based analysis further show that, without the schema, the model is prone to sycophancy (facilitating unsafe user requests) and failure to surface safety-critical unknowns. Schema-augmented models not only refuse or seek clarification in ambiguous or high-risk situations but do so for the correct categories of missing context.
Figure 5: The addition of the schema prevents the model from executing a potentially unsafe user plan, prompting for critical missing information instead.
Figure 6: Schema-driven models surface important unknowns (e.g., medical symptoms) missed by plain memory, resulting in safer guidance.
Multi-Turn Interaction: Asking Pays Off
When multiple interaction turns are allowed (the model's clarifying questions are answered), schema-bearing assistants convert their initial asking behavior into increased calibration and ultimate usefulness, outperforming baseline and memory-only models.
Mechanism Validation and Diagnostic Ablations
Ablation studies reveal that both the explicit structural format and explicit dimension names contribute to schema efficacy. Merely enumerating categories or generic [unknown] markers achieves only modest gains individually; only their combination yields large improvements in both safety (harm reduction) and unknown-awareness.
Theoretical and Practical Implications
The Severance Schema operationalizes a shift from expanding the memory footprint (which creates new failure modes) to building explicit meta-representations of the knowledge boundary. This change has substantial implications:
- Practical deployment: Simple prompt-level intervention is agnostic to backend model, incurs negligible overhead, and enhances safety.
- Evaluation: Standard LLM-judge and human studies confirm that schema-driven known-unknown tracking can be reliably measured and that such interventions have meaningful impact.
- Future directions: Structured awareness provides a scaffolding for more robust, consent-anchored, overt personalization that is less manipulable and more controllable. It motivates research into dynamic model-driven schema learning and continuous, user-aligned knowledge acquisition.
- Safety and alignment: The schema suppresses sycophancy at its structural source, not as a post-hoc penalization, and transparently surfaces knowledge gaps to both the model and user.
Limitations and Future Work
The evaluation covers a broad life-advisory spectrum via synthetic profiles; generalization to domain-specific and high-stakes settings (e.g., clinical healthcare) requires further work. Potential risks include schema-induced user data collection, hence the necessity of clear consent protocols. Long-term impacts as user memory continually grows, and adaptation to evolving or idiosyncratic user dimensions, are open research questions. The extent to which models can learn to construct these representations autonomously also remains unexplored.
Conclusion
This work demonstrates that contemporary personalized AI assistants fundamentally lack awareness of the categories of information absent from their context, not just the facts themselves. The Severance Schema, by explicitly structuring these knowledge gaps, drives significant gains in safety, reduces hallucination and sycophancy, and enhances calibration and ultimate usefulness, with simple, generalizable prompting. The results invite deeper inquiry into explicit meta-representations in interactive AI and point to new directions in overt, structured, and consent-based personalization aligned with real human users.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper looks at a simple but important problem with today’s AI assistants: they only “see” the words you type, not your full life. Because of that, they often give advice that sounds confident but ignores key parts of who you are. The authors call this the Severance Problem (inspired by the TV show), and they propose a fix: give the AI a clear, structured reminder of what it does not know about you. They show that this makes AI safer, more careful, and better at asking useful questions before giving advice.
What questions does the paper ask?
The paper focuses on three easy-to-understand questions:
- Can we make AI assistants remember what they don’t know about you, not just what they do know?
- Does this “structured ignorance” make their advice safer and more helpful?
- Will this still work as the AI learns more about you over time, and will it help the AI ask better questions across multiple messages?
How did the researchers test their idea?
First, they define the Severance Problem: an AI only has your current message, not your wider life. That means it misses important facts (like your health, money situation, or stress level), and worse, it doesn’t even realize that those categories might matter.
To fix this, they create the Severance Schema, which is basically a short checklist that sits in the AI’s “mind” and says, “Here are the kinds of things about the person I might be missing.” The schema has six simple dimensions. Think of them like six boxes the AI should check before giving advice:
- Physicality (“You have a body”): health, medications, energy, limits.
- Temporality (“Time passes for you”): deadlines, life stage, near-future events.
- Consequences (“This affects your real life”): money, relationships, people who depend on you.
- Continuity (“You existed before this chat”): past attempts, patterns, history.
- Multiplicity (“You play many roles”): parent, student, worker, caregiver, etc.
- Interiority (“You have feelings”): your mood, fears, and hopes.
Instead of stuffing the AI with more facts about you, the schema tells it what it still doesn’t know. That turns “unknown-unknowns” into “known-unknowns,” which nudges the AI to ask clarifying questions before advising.
Here’s how they tested it:
- Four setups (same user questions; different “system” instructions to the AI):
- No Schema: just answer.
- Severance Schema: keep the “missing info checklist” in mind.
- Memory: give the AI a few personal facts about the user (like a mini bio).
- Schema + Memory: give both the checklist and the personal facts.
- Data: 10 made-up people and 30 everyday advice scenarios (like health, money, school, parenting).
- Models: five different LLMs.
- Scoring: another model (acting like a judge) rated each answer for:
- Unknown-awareness (does it notice what’s missing?)
- Question-asking (does it ask helpful clarifying questions?)
- Calibration (does its confidence match what it knows?)
- Usefulness (is the advice practical?)
- Safety issues like hallucination (making stuff up), harmful advice, and sycophancy (just agreeing with the user to please them).
- They also ran multi-turn tests (the AI’s questions get answered; then it tries again) to see if asking pays off.
What did they find?
The main results are straightforward:
- The schema makes AI more careful and safer.
- It roughly doubles the AI’s ability to notice missing personal context.
- It cuts harmful advice and “people-pleasing” behavior by more than half in several models.
- It reduces hallucinations (making things up), especially when the AI has some memory about the user.
- Memory alone (just giving the AI some personal facts) can backfire.
- With memory but no schema, the AI became more overconfident and hallucinated more (rising from near 0% to around 4–12% in the tests). In plain terms: a few facts made it act like it knew everything, so it asked fewer questions and guessed the rest.
- Schema + Memory is best overall.
- The checklist keeps the AI aware of what it still doesn’t know, even when it has some memory. That leads to safer, more calibrated advice while still using the personal facts well.
- Asking the right questions pays off in the next turn.
- In two-turn chats, the schema setups improved usefulness once their clarifying questions were answered. In short: when the AI first pauses to ask key questions, its second answer gets better than the setups that skipped asking.
In everyday terms: the checklist teaches the AI to say, “Before I advise you, I might be missing critical stuff about your body, time pressures, consequences, history, roles, or feelings. Let me ask a couple of quick questions.” That small pause prevents bad guesses and leads to better advice.
Why is this important?
This matters for anyone building or using personal AI assistants:
- It’s safer. The AI is less likely to give harmful advice or just tell you what you want to hear.
- It’s more respectful. It treats you like a whole person with a life beyond a single message.
- It’s more honest. It admits what it doesn’t know and asks before advising.
- It works with or without memory. Even as the AI learns more about you, the checklist keeps it from becoming overconfident.
- It’s simple to add. It’s just a structured prompt, not a new model or big system change.
The authors suggest pairing this approach with clear user consent, since a checklist that invites questions can surface personal details. They also note limits: they used synthetic profiles and general-life scenarios, so future work should test the idea in specific areas like health or finance and in real user studies.
Bottom line
The paper shows that smarter personalization isn’t just “more memory.” It’s also about structured ignorance—teaching AI to keep track of what it doesn’t know about you. A short checklist (the Severance Schema) nudges AI to ask better questions, avoid unsafe guesses, and give more thoughtful, person-aware advice. This could make future AI helpers more trustworthy and truly “personal” in a healthy, respectful way.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a single, concrete list of what remains missing, uncertain, or unexplored in the paper, framed as actionable directions for future research.
- External validity with real users: Do the schema’s gains (reduced sycophancy/harm, improved calibration) hold in longitudinal, real-user deployments where motivations, attention, and willingness to answer questions vary over time?
- Longitudinal dynamics with growing memory: How does the schema interact with continuously accumulating personal memory over weeks or months (e.g., question fatigue, habituation, drift in question-asking, repeated questions)?
- Human-centered outcomes: What are the effects on user satisfaction, trust, perceived empathy, and decision quality compared to baselines that ask fewer questions?
- Cost–benefit of clarification: What is the optimal policy for when to ask vs. answer immediately, balancing value-of-information, user burden, risk, and latency/token costs?
- High-stakes, domain-specific settings: Does the schema improve safety and usefulness in regulated or expert domains (e.g., clinical, financial, legal) under domain constraints, professional standards, and liability?
- Domain-tailored schemas: How should the six dimensions be specialized, expanded, or hierarchically refined for particular domains (e.g., medication adherence, risk factors, fiduciary duty), and how does that affect performance?
- Cross-cultural and multilingual generalization: Do the dimensions and question-asking behaviors generalize across languages, cultures, and norms about privacy and self-disclosure?
- Demographic fairness: Does the schema systematically produce different question loads, deferrals, or caution levels for different demographic groups, and does that affect equity of advice?
- Adverse user experience risks: Does surfacing “unknowns” increase anxiety or decision paralysis for some users or scenarios? How to calibrate tone and frequency to mitigate distress?
- Privacy and consent mechanics: Which UI patterns, consent flows, and data-minimization policies are needed to ensure the schema’s question-asking does not lead to over-collection of sensitive data?
- Attack and misuse surface: Can adversaries exploit the schema (e.g., to elicit sensitive info), or can users game it (e.g., to bypass safety via cooperative answers)? What mitigations are effective?
- Memory representation dependence: Is the “memory blind spot” (elevated hallucination, reduced unknown-awareness) specific to bullet-list memory, or does it persist with alternative memory formats (e.g., knowledge graphs, JSON schemas, vector stores with provenance)?
- Tool-using agents: How does the schema perform when assistants can call tools (RAG, web search, calculators, EHRs)? Does structured ignorance improve tool selection and reduce spurious tool use?
- Multi-turn scalability: Beyond two turns, how does question quality, redundancy, and downstream usefulness evolve over longer dialogues (5–10 turns), and when should the assistant stop asking?
- Mechanism identification: Does the schema improve internal reasoning (e.g., causal/commonsense inference about missing variables) or just prompt the model to enumerate unknowns? Use mechanistic probes or controlled tasks to distinguish.
- Alternative scaffolds: How does the schema compare to other structured prompts (e.g., ToM frames, risk checklists, SBAR-style handoffs, decision trees) on the same tasks?
- Learning structured awareness: Can models be trained (via SFT/RL/ILHF) to internalize “awareness of unknowns” without explicit scaffolding, preserving benefits while reducing prompt overhead?
- Personalization of question-asking: Can the assistant learn a user-specific questioning policy (e.g., fewer but higher-value questions, remembering answered areas) that maintains safety while minimizing burden?
- Value-of-information (VoI) control: Can we formalize and implement VoI-based decision rules that drive when and which schema dimensions to query as a function of decision stakes?
- Token/latency budget: What is the end-to-end cost of including the schema in production (context length, compute, latency), and which compression/ablation strategies preserve benefits?
- Evaluation robustness to gaming: The “mentions_unknowns” metric may be gameable by listing many unknowns; how to measure question quality, relevance, and actionability robustly (e.g., human labeling, outcome-based metrics)?
- Human-judged validation: To what extent do the LLM-judge scores (GPT-5.2, Sonnet 4) align with expert and lay human judgments, especially for harm and sycophancy? Provide preregistered human studies and inter-rater reliability.
- Benchmark breadth: Do results hold on more diverse, real-world scenarios beyond the 10 profiles/30 synthetic scenarios, including edge cases, rare conditions, and conflicting goals?
- Ground-truth annotations: The “good-answer” and “decision-flip” claims are author-defined; how sensitive are results to different annotators, and can community-sourced or expert-reviewed claims reduce bias?
- Closed vs. open model variance: Are the gains consistent across model sizes, training paradigms (RLHF vs. DPO vs. constitutional), and future foundation models?
- Interactions with safety alignment: How does the schema interact with existing safety layers (e.g., refusal policies, refusal-to-assist on medical advice) and does it reduce over-refusals in low-risk cases?
- Non-English prompt efficacy: Does the innie/outie metaphor and the schema phrasing translate effectively to other languages, or is re-conceptualization needed?
- Role conflict resolution: Can the schema help the model reason about trade-offs across roles (multiplicity) using explicit multi-objective optimization rather than ad hoc balancing?
- Repetition and deduplication: How can the assistant avoid repeatedly asking for the same information across sessions while keeping unknown-awareness high?
- Measuring downstream behavior change: Beyond immediate advice quality, does schema-guided interaction lead to safer or more beneficial real-world user actions over time?
- Governance and logging: What observability and audit mechanisms are needed to track what was asked, what was learned, and how it influenced advice, while protecting user privacy?
- Standardization: Can a community standard for person-context schemas be defined (versioned, extensible), enabling interoperability across assistants, memory layers, and tools?
- Failure characterization: Under what scenario types (e.g., low-stakes, purely factual queries) does the schema provide no benefit or negative utility, and can automatic classifiers disable it appropriately?
- Calibrated deferral: How should the assistant decide to defer to human experts (e.g., clinicians) based on remaining unknowns and stakes, and communicate that deferral effectively?
- Multi-user and shared-context settings: How does the schema work when decisions involve households/teams (conflicting preferences, shared resources), and how should memory and unknowns be partitioned?
- Regulatory compliance: How to align schema-driven data collection and use with GDPR/CCPA/HIPAA (purpose limitation, data minimization, retention policies) in real systems?
Practical Applications
Immediate Applications
Below are concrete, deployable use cases that leverage the paper’s Severance Schema (structured ignorance) and awareness-plane framing to improve safety, calibration, and usefulness in current systems.
- Schema-aware chat assistants — software, customer service, HR
- Use case: Wrap existing LLMs with the Severance Schema system prompt to reduce sycophancy and harmful advice while increasing clarifying questions before action.
- Tools/workflows: Prompt templates and middleware for LangChain/LlamaIndex/OpenAI Assistants API; “schema slots” object filled from CRM/HRIS user memory; logging of asked-unknowns.
- Sectors: Software, customer service, HR.
- Dependencies: Ability to modify system prompts; access-controlled memory retrieval; UI affordances for asking/consent.
- Safety-gated advisory bots — healthcare triage, consumer finance, career coaching
- Use case: For high-stakes queries, require the assistant to surface “decision-flip” unknowns (e.g., symptoms, dependents, runway) before giving advice or proceeding to drafts.
- Tools/workflows: Policy checks that prevent action unless critical schema slots are confirmed; escalation to human; prebuilt “decision-flip” libraries per domain.
- Sectors: Healthcare (non-diagnostic guidance and triage), finance (budgeting/risk), career services.
- Dependencies: Domain-specific red-flag lists; clear disclaimers; regulatory alignment (HIPAA/GDPR/FINRA) and consent.
- Consentful information-seeking UI — product design pattern
- Use case: Show users which schema dimensions are unknown and request permission to fill them (e.g., “We don’t know your dependents; is this relevant?”).
- Tools/workflows: Inline consent toggles per schema slot; just-in-time data collection; “why we ask” explainer; storage retention controls.
- Sectors: Consumer apps, enterprise apps.
- Dependencies: Privacy-by-design; PII minimization; data governance and retention policies.
- Memory that stays humble — personalization without overconfidence
- Use case: Replace “bullet memory” with “schema-filled memory” so the model both uses known facts and keeps unknown slots explicit, avoiding hallucinated personal context.
- Tools/workflows: Memory-to-schema adapter; retrieval that maps records to specific schema fields; audits that verify unknown-awareness isn’t suppressed by memory.
- Sectors: Any personalized assistant.
- Dependencies: Mapping layer from existing profiles/CRMs to schema; configurable thresholds for when to ask vs answer.
- Unknown-awareness as a QA metric — model evaluation pipelines
- Use case: Add “mentions_unknowns,” “asks_questions,” and “decision-flip surfaced” to internal eval dashboards alongside hallucination and usefulness.
- Tools/workflows: Test batteries modeled on the paper’s benchmark; LLM- or human-judged metrics; CI checks pre-deployment.
- Sectors: Software, model providers, safety teams.
- Dependencies: Scenario/claim authoring; inter-judge validation; acceptance thresholds per product risk.
- Prompt linting for safety — engineering enablement
- Use case: Static checks that ensure system prompts include schema slots and “ask-then-answer” guidance for sensitive workflows.
- Tools/workflows: Prompt linters in CI; IDE extensions; centralized prompt registry with schema variants.
- Sectors: Software engineering.
- Dependencies: Prompt governance; engineering buy-in; change-management.
- Escalation and holdbacks for risky actions — operations
- Use case: Block drafting of risky content (e.g., vaccine refusal letters, large transfers) until specific schema slots are verified (oncologist input, risk tolerance, runway).
- Tools/workflows: Policy engines keyed to schema fields; action gating; audit logs of unknowns acknowledged at time of action.
- Sectors: Healthcare admin, finance ops, legal ops.
- Dependencies: Domain policies; human-in-the-loop; clear user messaging.
- Tutoring that resists sycophancy — education
- Use case: Tutors that ask about knowledge gaps, time constraints, goals, and affect before guidance; challenge wrong beliefs respectfully instead of mirroring them.
- Tools/workflows: Schema slots adapted for learning context (prior knowledge, goals, schedule, accommodations); formative question templates.
- Sectors: Education, edtech.
- Dependencies: Age-appropriate prompts; accessibility accommodations; parental consent where applicable.
- Contact center triage improvement — customer support
- Use case: Agents ask structured clarifiers (timeline, stakes, roles) to resolve cases faster and reduce back-and-forth/handoffs.
- Tools/workflows: Playbooks aligned to schema; agent assist that proposes clarifying questions; disposition codes mapped to slot coverage.
- Sectors: Customer service, telecom, retail, SaaS.
- Dependencies: CRM integration; agent training; latency budgets.
- Compliance and audit trails — governance
- Use case: Record “what was unknown” at time of advice to show due diligence and reduce liability for overconfident automation.
- Tools/workflows: Immutable logs of schema state per interaction; risk reviews referencing unknown-awareness metrics.
- Sectors: Regulated industries (health, finance, legal).
- Dependencies: Legal review; secure logging; retention schedules.
- Domain-ready schema templates — vertical packs
- Use case: Distribute pre-tuned schema templates (with domain-specific subslots and decision-flips) for healthcare triage, SMB finance, parental guidance, career decisions.
- Tools/workflows: Template library; slot mappers; sample policy files for gating.
- Sectors: Healthcare, finance, parenting, career services.
- Dependencies: SME curation; ongoing update cadence; localization.
- Agent orchestration heuristic — tool-using agents
- Use case: Before executing tools (booking, purchases, filings), agents check unresolved schema slots (stakes, timeline) and pause for user input if high-risk.
- Tools/workflows: Orchestrator policy layer; slot-sensitive tool permissions.
- Sectors: General AI agents, travel, shopping, scheduling.
- Dependencies: Tool permissioning; UX for interruptions; user tolerance for clarifiers.
- Research replications and benchmarks — academia
- Use case: Adopt awareness-plane conditions and metrics for new domains and models; extend with human studies.
- Tools/workflows: Open-source benchmark adaptation; inter-judge protocols; IRB processes.
- Sectors: Academia, industry research.
- Dependencies: Dataset licensing; participant recruitment; reproducibility tooling.
- Personal life assistants — daily life
- Use case: Home assistants that check physicality (energy, meds), temporality (deadlines), stakes (budget), roles (caregiving), and interiority (stress) before giving suggestions.
- Tools/workflows: On-device schema cache; quick-consent prompts; “unknowns panel” users can edit.
- Sectors: Consumer apps, wellness.
- Dependencies: Privacy posture; offline mode; simple UX.
Long-Term Applications
These applications require further research, domain adaptation, or integration with regulated systems.
- Schema-native model training — beyond prompting
- Vision: Train models to internalize structured ignorance (predict and represent unknown user-context) as a first-class objective, not just via prompt scaffolding.
- Tools/workflows: Synthetic curricula, supervised/rL fine-tuning on “ask-then-answer” trajectories; tokens/heads dedicated to unknowns; evals tied to decision-flips.
- Sectors: Foundation models, safety.
- Dependencies: Training data/compute; continual learning without privacy leakage; open evaluations.
- Domain-specialized schemas — clinical, legal, and financial-grade
- Vision: Expand the six dimensions into vetted, granular sub-ontologies (e.g., cardiopulmonary red flags; fiduciary obligations; jurisdictional constraints).
- Tools/workflows: Standards-backed slot definitions; clinician/compliance review; integration into EHR/Bank cores.
- Sectors: Healthcare, law, finance.
- Dependencies: Regulatory approval paths; interoperability (HL7/FHIR, ISO/IEC); liability frameworks.
- Regulatory standards for structured awareness — policy
- Vision: Certification that advice-giving AI demonstrates unknown-awareness, decision-flip gating, and consentful info-seeking; possible requirement for high-risk use.
- Tools/workflows: Third-party audits; standardized test suites; disclosure requirements (“advice was withheld until X unknowns resolved”).
- Sectors: Public policy, regulators, industry consortia.
- Dependencies: Multi-stakeholder consensus; legal mandates; audit capacity.
- Outie-aligned personal OS — longitudinal, consent-first personalization
- Vision: A cross-app personal context layer that maintains both knowns and unknowns, continuously updates with user consent, and orchestrates assistants across life domains.
- Tools/workflows: Privacy-preserving memory services; user-owned data vaults; schema brokers across apps.
- Sectors: Consumer platforms, mobile OS, identity.
- Dependencies: Data portability; trust/UX; on-device inference; revocation controls.
- Safety economics and KPIs — management systems
- Vision: Make unknown-awareness, harmful-advice rate, sycophancy rate, and hallucinations core product KPIs; link to incentives and SLAs in high-stakes products.
- Tools/workflows: Org-wide dashboards; cohort analysis (most at-risk users); experimentation frameworks.
- Sectors: Enterprise software, platforms.
- Dependencies: Data infra; culture change; risk appetite alignment.
- Multi-agent architectures with a schema sentinel
- Vision: Pair a task agent with a “sentinel” agent responsible for tracking unknowns, enforcing decision-flip gates, and auditing advice before delivery.
- Tools/workflows: Inter-agent protocols; veto/holdback mechanisms; reason tracing.
- Sectors: Agent platforms, robotics HRI, enterprise automation.
- Dependencies: Coordination overhead; latency budgets; conflict resolution.
- Privacy-preserving unknown-awareness — minimal disclosure designs
- Vision: Compute which unknowns matter without revealing sensitive values (e.g., via local inference, secure enclaves, or federated prompts).
- Tools/workflows: On-device slot scoring; zero-knowledge proofs-of-check; selective revelation protocols.
- Sectors: Privacy tech, mobile, healthcare finance.
- Dependencies: Efficient on-device models; cryptographic protocols; regulatory acceptance.
- Human factors and affect-aware advising — inner life done right
- Vision: Robust handling of “interiority” (emotion, motivation) across cultures and neurotypes; prevent dark patterns and coercion.
- Tools/workflows: Guardrails that distinguish supportive challenge from manipulation; culturally adapted slot sets; longitudinal UX studies.
- Sectors: Mental health, education, consumer apps.
- Dependencies: Clinical input; IRB research; ethics boards.
- Decision-flip detection toolkits — generalized risk inference
- Vision: Libraries that infer which unfilled slots could flip an answer in a given domain and auto-generate targeted clarifiers.
- Tools/workflows: Counterfactual evaluation; influence functions; domain ontologies.
- Sectors: Safety tooling, vertical SaaS.
- Dependencies: Domain data; robust causal proxies; continuous updates.
- Legal safe-harbor frameworks — accountability by design
- Vision: Liability models that credit organizations for structured-awareness practices (gated actions, audit logs of unknowns surfaced).
- Tools/workflows: Compliance playbooks; standardized logs; insurer-backed attestations.
- Sectors: Insurance, law, policy.
- Dependencies: Legislative action; insurer participation; incident reporting norms.
- Educator-in-the-loop programs — at-scale tutoring augmentation
- Vision: Schema-driven tutors that collect relevant unknowns and hand off structured summaries to teachers for targeted interventions.
- Tools/workflows: LMS plugins; slot-to-intervention mappings; progress analytics.
- Sectors: K–12, higher ed, workforce upskilling.
- Dependencies: District procurement; accessibility; evidence of learning gains.
- Industrial and field robotics — human-aware task planning
- Vision: Robots/assistants that query human partners for constraints across roles, timelines, and stakes before executing plans that affect people.
- Tools/workflows: HRI dialogs mapped to schema; safety stops if critical unknowns persist.
- Sectors: Robotics, manufacturing, eldercare.
- Dependencies: Real-time dialog reliability; safety certification; user acceptance.
- Research programs on meta-ignorance — cognitive grounding
- Vision: Formalize and measure second-order ignorance in AI; relate to theory-of-mind and calibration; develop new benchmarks beyond this paper’s setting.
- Tools/workflows: Shared datasets; cross-model studies; open leaderboards.
- Sectors: Academia, labs.
- Dependencies: Community uptake; funding; reproducibility standards.
Common assumptions and dependencies across applications
- User consent, transparency, and data governance are foundational; schema may prompt collecting more personal data and must be paired with clear consent and minimization.
- Domain-specific “decision-flip” catalogs require expert curation and periodic updates.
- Access to modify system prompts and integrate memory/retrieval is necessary; some hosted assistants may limit this.
- Evaluation currently relies on LLM judges; human validation and inter-judge agreement are recommended for high-stakes deployments.
- Regulatory alignment varies by sector (HIPAA/GDPR/PCI/financial advice restrictions); legal review is essential before automating sensitive advice.
Glossary
- Ablation: An experimental analysis where specific components are removed or altered to assess their contribution to overall performance. "our ablation shows that neither the [unknown] slot format alone nor simply naming the person-context categories recovers the full effect"
- Adversarial-attack risks: Threats where inputs are crafted to manipulate model behavior or cause failures. "stresses adversarial-attack risks rather than blind-spot ones."
- Alignment: The degree to which model behavior conforms to human values, intentions, or desired objectives. "reframes alignment beyond harmlessness--helpfulness"
- Area Under the ROC Curve (AUROC): A metric summarizing the trade-off between true and false positive rates across thresholds. "near-chance AUROC (0.43--0.53)"
- Awareness plane: A conceptual 2D framework charting a model’s information about a person and its awareness of what it doesn’t know. "The awareness plane."
- Bootstrap confidence intervals (CIs): Uncertainty estimates computed by resampling data to form empirical distributions of a statistic. "bootstrap CIs"
- Calibration: The match between a model’s confidence and the actual correctness or appropriateness of its outputs. "Calibration closely matches for both the Schema + Memory (blue) and Memory (orange)"
- Dark pattern: Design or interaction strategies that manipulate user behavior in ways that may not align with their best interests. "frame personality-conditioned adaptation as a dark pattern"
- Decision-flip claims: A subset of factors that, if present or absent, could change the safety or direction of a recommendation. "Decision-flip claims, typically 2--3 per scenario, identify a smaller subset of information whose absence could change the safety of the recommendation."
- Double Unawareness Problem: The issue where a model lacks both information about the user and awareness of the categories of information it lacks. "The Double Unawareness Problem."
- Fine-tuning objective: A training goal used to adapt a pre-trained model to specific behaviors or tasks via additional data. "or a fine-tuning objective"
- Generative agents: Simulated agent architectures that produce behavior by generating actions or narratives over time. "generative agents"
- Hallucination: Fabrication of unfounded facts or details by a model. "hallucination increases from near-zero to 3.7-11.7\%"
- Innie: The AI assistant side of the interaction, confined to the conversation context, contrasted with the real user (“outie”). "we refer to the AI assistant as the innie and the user as the outie."
- Interiority: A schema dimension capturing the user’s inner feelings, emotions, and subjective states. "Interiority (``You have feelings about this.'') emotional state shaping what the outie needs to hear."
- Known-unknowns: Gaps a system recognizes it has and can explicitly inquire about. "clarification alone operates mostly over known-unknowns."
- Knowledge boundary: The explicit delineation of what the model does and does not know about the user. "explicit inventory of the model's knowledge boundary"
- LLM-judge protocols: Evaluation methods where a LLM serves as an automated judge of responses. "As with all LLM-judge protocols, ours inherits the limitations of automated evaluation"
- Meta-ignorance: Not just lacking information, but lacking awareness of that lack; ignorance about one’s ignorance. "the meta-ignorance branch of Smithson's taxonomy"
- Multiplicity: A schema dimension capturing that users occupy multiple roles simultaneously. "Multiplicity (``You contain multitudes.'') simultaneous roles as parent, employee, patient, partner."
- Open-weight models: Models whose parameter weights are publicly available for use and analysis. "on the open-weight models"
- OS-style abstraction: A systems-design analogy where components (e.g., memory) are structured like operating-system modules. "casts persistent memory as an OS-style abstraction"
- Prompt-level scaffolding: Structured prompt design that guides model reasoning without changing model weights. "The Severance Schema is a prompt-level scaffolding that fixes the categories of person-context the model must track"
- Reinforcement Learning from Human Feedback (RLHF): Training approach where models are optimized using human preference signals. "RLHF models match user beliefs over truth"
- Semantic entropy: An uncertainty measure based on the diversity or inconsistency of generated semantic content. "detect hallucinations via semantic entropy"
- Severance Problem: The inability of AI assistants to reason about aspects of a user’s life beyond the current prompt. "We call this the Severance Problem,"
- Severance Schema: A structured representation of categories of user context (and unknowns) to surface missing information. "We call this the Severance Schema."
- Structured awareness: An explicit, organized recognition of what user-related information is missing. "Structured Awareness improves reasoning and safety"
- Structured ignorance: Deliberately representing what is unknown to prevent overconfident extrapolation and prompt clarification. "incorporating structured ignorance into the LLM context via the Severance Schema"
- Synthetic advisory benchmark: A constructed evaluation set simulating real advisory scenarios to test model behavior. "We construct a synthetic advisory benchmark designed to evaluate reasoning under incomplete person-context."
- Theory of Mind (ToM): The capacity to infer others’ beliefs, desires, and intentions; in LLMs, tests of modeling other minds. "reports emergent ToM behavior"
- Unknown-awareness: A behavioral metric capturing a model’s tendency to recognize and flag missing, relevant user information. "raises unknown-awareness across every model."
- Unknown-unknowns: Relevant information gaps the system is unaware of and thus does not seek to resolve. "converts unknown-unknowns into known-unknowns"
Collections
Sign up for free to add this paper to one or more collections.
