Fix the Mind, Not the Move: Interpretable AI Assistance via Knowledge-Gap Localization

Abstract: AI assistants in human-AI collaboration often correct suboptimal human actions through behavioral feedback (e.g., alerts or steering-wheel nudges in assistive driving). Such interventions can mitigate immediate errors, but long-term improvement requires addressing the underlying misconceptions that cause repeated mistakes. We introduce SENSEI, a framework that infers user misconceptions from interaction behavior and provides targeted, minimal yet sufficient suggestions to correct them. Our approach departs from action- or trajectory-level interventions by operating over a structured knowledge representation to localize and correct the sources of erroneous behavior. Across three long-horizon tasks with diverse misconceptions and corresponding behaviors, SENSEI demonstrates zero-shot compositional generalization, disentangling multiple overlapping misconceptions despite training only on single-misconception cases. A user study further shows that our method identifies real human misconceptions and provides effective guidance that improves long-horizon task performance, successfully correcting $90\%$ of student misconceptions. Code and project page are available at https://misoshiruseijin.github.io/SENSEI/.

• The paper introduces SENSEI, a modular AI framework that diagnoses and corrects underlying knowledge gaps rather than merely addressing surface-level action errors.
• It employs a two-stage inference pipeline using CodeT5+ encoders and latent embedding edits to generate precise and interpretable textual corrections in symbolic planning tasks.
• Empirical evaluations demonstrate high alignment F1 scores, robust zero-shot compositional generalization, and significant improvements in user decision-making accuracy.

Interpretable AI Assistance via Knowledge-Gap Localization: An Expert Analysis of SENSEI

Introduction and Problem Formulation

The paper introduces SENSEI, a modular AI framework designed to enhance collaborative, long-horizon human–AI decision-making by diagnosing and correcting users' task-relevant knowledge gaps rather than merely correcting individual errors at the action or behavioral trajectory level (2606.05602). SENSEI formulates the assistance problem as structured knowledge alignment: observing a user's behavior, inferring divergences from expert knowledge at a symbolic, component level, and transmitting interpretable, minimal corrections to drive post-guidance behavior toward expert-level optimality, even for new task instances.

Existing behavior-correction systems typically intervene directly at the action level—through nudging, alerts, or trajectory editing—but fail to construct explicit, reusable diagnoses of underlying human misconceptions. In contrast, SENSEI shifts the focus to the causal knowledge components responsible for systematic planning errors. This paradigm supports diagnosis and guidance that generalize composably across tasks and scenarios, moving beyond reactive correction to proactive cognitive alignment.

Figure 1: SENSEI's conceptual pipeline: from behavior analysis and symbolic knowledge-gap identification to interpretable, knowledge-aware assistance.

Methodology

Knowledge Representation and Factorization

Task knowledge for both expert and student agents is modeled with PDDL, permitting explicit, structured decomposition into componentized symbolic variants (e.g., objects, actions, predicates). Each knowledge component, $\phi_i$, can take on multiple variants, $v_i$, representing specific beliefs or misconceptions (e.g., swapped item types, absent negative side-effects, omitted skills). The set of divergent component–variant pairs between the student and expert constitutes the knowledge gap set.

Two-Stage Inference Pipeline

SENSEI's architecture comprises:

1. Knowledge Gap Localization Module: Given expert and student behaviors, as well as the expert's symbolic task representation, this module utilizes frozen CodeT5+ encoders to produce embeddings. It predicts the minimal set of knowledge components where the student's understanding diverges from the expert.
2. Knowledge Edit Module: For each localized gap, this stage applies a latent embedding edit to the expert component, decodes the resulting latent vector into interpretable raw text (PDDL block), and synthesizes precise, student-specific corrections.

   Figure 2: SENSEI's architecture, combining knowledge gap localization with latent embedding edits and natural language correction synthesis.

Training directly supervises three outcomes—accurate gap localization, precise correction for true gaps, and avoidance of unnecessary edits. Notably, mixup-based regularization is deployed: synthetic multi-gap behaviors are compositionally generated at the embedding level from single-gap cases, driving zero-shot generalization to previously unseen combinations of misconceptions.

Experimental Evaluation

Data Generation and Task Domains

The evaluation spans three structured domains:

• Breakfast (Cooking): Multi-step household preparation, simulating hazards, skill gaps, and type confusions (e.g., salt-sugar swaps).
• Overcooked (Kitchen Navigation): Task planning with spatial constraints, tutorial gatekeeping, and action delicacy requirements.
• Rover (IPC Planning): Resource- and communication-constrained scientific sampling, with extended precondition and execution dependencies.

Behavioral traces and PDDL perturbations are synthesized to replicate various misconception types. Only single-misconception training trajectories are used; compositional multi-misconception scenarios are reserved for evaluation.

Figure 4: "Breakfast" domain - a representative, parameterized testbed for compositional, symbolic knowledge gap induction.

Baselines

Competitors include:

• Monolithic end-to-end transformer models (without separable gap localization).
• Prompted off-the-shelf LLMs (GPT-4o, GPT-5.2, Llama-3.3-70B).
• Random and heuristic localization bounds.
• Oracle localization upper bounds.

Metrics

Performance is measured on gap localization recall/precision, correction accuracy, end-to-end system-level recall/precision/F1, and false correction rates.

Results

SENSEI consistently achieves the highest alignment F1 across all domains, significantly outperforming LLM baselines and pure end-to-end models, due primarily to superior gap localization and minimal erroneous corrections. For example, in the Overcooked domain, SENSEI attains $\text{Sys}_\text{F1}=0.726$, compared to $0.033$ for the end-to-end baseline and $<0.05$ for LLMs.

Key empirical findings:

• Zero-shot compositional generalization: SENSEI, when trained solely on single-misconception data, accurately diagnoses and corrects previously unseen multi-misconception student behaviors.
• Ablation studies reveal that removing mixup augmentation leads to catastrophic drops in recall, validating the compositional inductive bias.
• Separation of localization and correction yields high correction accuracy ($\geq 97\%$ on true gaps) and avoids deleterious modifications to correct components.
• Off-the-shelf LLMs fail to generate consistent or semantically valid corrections in the symbolic domains used.

User Study

A 20-participant user study in the Overcooked domain demonstrates practical impact. After observing user behavior with induced knowledge gaps (via withheld instructions), SENSEI-generated guidance corrects $90\%$ of user misconceptions, improves plan–expert IoU alignment by $10.8\%$, and is rated $4.27/5$ for helpfulness, despite a moderate unhelpful-tip rate.

Figure 3: User study task and assistance loop—SENSEI localizes critical knowledge gaps and issues minimal, actionable guidance in the Overcooked-AI environment.

Figure 5: Interactive user environment supporting symbolic planning trace extraction and real-time, knowledge-aware assistance.

Implications and Future Research

SENSEI operationalizes a structured, interpretable approach to human–AI behavior alignment by targeting the underlying task knowledge rather than surface actions. This is a distinct contribution compared to existing shared-autonomy and trajectory-level correction paradigms. The demonstrated generalization and modularity suggest applicability to other domains characterized by explicit, componentized knowledge—such as programming education, scientific workflow automation, or advanced manufacturing.

The results suggest several research directions:

• Developing value-aware gap localization to prioritize corrections by downstream behavioral impact.
• Extending the framework beyond deterministic, logic-based PDDL to stochastic or continuous symbolic/task domains.
• Relaxing the expert-referenced model to accommodate environments with multiple optimal policies or agent–agent alignment scenarios.
• Integrating interaction strategies that dynamically balance recall and precision, mitigating user annoyance from false corrections.

Potential challenges include deployment in open-ended environments where "expert" optimality is subjective or multi-modal, as well as managing the normative risks of enforcing rigid expert-consistency in creative or non-standard tasks.

Conclusion

SENSEI establishes a principled, interpretable pipeline for knowledge-gap localization and correction in AI-assisted, long-horizon planning. By outperforming both monolithic neural baselines and large pretrained LLMs on compositional, symbol-structured diagnosis, SENSEI demonstrates that explicit knowledge alignment is a robust and generalizable foundation for cognitive AI assistance. The methodology and empirical evidence support future research aimed at scaling interpretable knowledge-aware systems to more complex and high-stakes collaborative domains.