- The paper introduces an innovative retrieval-augmented AI system that merges regulatory DEA documents with PubMed research to enhance substance use education.
- It details a multi-layer architecture employing dense vector embeddings and LangChain-based routing for context-aware, real-time responses.
- Expert evaluations show high performance in factual accuracy and citation quality, with ratings averaging from 'Good' to 'Excellent' across metrics.
Agentic Retrieval-Augmented AI for Substance Use Education: Integration of Regulatory and Scientific Knowledge
Problem Context and Motivation
Persistent deficiencies in substance use education—particularly with regard to scalability, personalization, and the provision of current, context-sensitive information—pose significant obstacles for both public health practitioners and affected populations. Prevailing reliance on static, manually curated resources is increasingly misaligned with evolving substance landscapes and modern information-seeking behaviors. Deployment of LLMs as health education agents has been constrained by well-documented limitations: fabrication (“medical hallucinations”), temporal staleness due to fixed training cutoffs, algorithmic propagation of stigma, lack of explicit evidence chains, and insufficient regulatory grounding. These factors pose substantiated risks, particularly for vulnerable cohorts requiring trauma-informed, context-aware educational support.
Architecture and System Design
The paper describes an agentic retrieval-augmented AI (RAG) web application architected for real-time, transparent substance use education. The system’s architecture comprises:
- Local Knowledge Base: 102 DEA-sourced documents processed with a robust pipeline incorporating PDF extraction (PyMuPDF4LLM/OCRmyPDF), structured text segmentation, and dense vector embeddings (Qwen3-Embedding-8B, 1024-dimensions). Stored in ChromaDB with HNSW indexing for high-throughput nearest-neighbor retrieval.
- External Real-Time Retrieval: Automated PubMed queries targeting systematic reviews and high-impact publications within the past five years, interfaced programmatically via the NCBI E-utilities API. External retrieval is orchestrated by the Model Context Protocol (MCP) to ensure modular, structured tool integration.
- Agentic Orchestration: LangChain-based agents apply explicit routing heuristics, weighting DEA sources for regulatory/policy questions and PubMed for clinical/neurobiological queries. Contextual rephrasing and conversational state tracking underpin coherent, multi-turn interactions.
- Generation: Qwen3-32B LLM produces responses with enforced source-grounded reasoning, transparent evidence attribution, and explicit acknowledgement of knowledge boundaries.
Overall, the architecture synthesizes static, high-authority regulatory data with dynamic, scientific literature to address the dual imperatives of legal accuracy and clinical/scientific currency. Modular design across five layers ensures extensibility and traceability.
Evaluation Methodology
Expert evaluation was executed with a five-member panel (mean domain experience: 14 years). The evaluation utilized:
- 30 Semantically-Deduplicated Questions: Grounded in professional curriculum, covering classifications, health effects, prevention, regulation, and treatment/recovery.
- 90 System Interactions: Each primary question paired with two context-dependent follow-ups to stress-test conversational history and contextual disambiguation.
- Four Evaluation Criteria: Factual accuracy, citation quality, contextual coherence, and regulatory appropriateness. Independent dual-rater assessment with five-point Likert scales.
- Reliability: Cohen’s kappa (K=0.78) indicating substantial inter-rater concordance.
The protocol adhered to medical informatics ethical standards, with exclusive reliance on publicly available, non-identifiable information.
Numerical Results
The system exhibited high aggregate performance—mean ratings spanned the “Good” (4) to “Excellent” (5) range across all metrics:
- Factual Accuracy: 4.31 (SD=0.68)
- Citation Quality: 4.18 (SD=0.74)
- Contextual Coherence: 4.26 (SD=0.71)
- Regulatory Appropriateness: 4.35 (SD=0.63)
No responses were jointly rated below “Adequate.” Performance stratification revealed the highest scores for regulatory domains reliant on DEA sources (classifications: 4.52, policy: 4.48), with modestly lower but still positive ratings for dual-source health effect (4.21) and prevention questions (4.19). Treatment and recovery (4.05) lagged, reflecting the paucity of treatment information in DEA resources and the limitations of abstract-only external retrieval.
The system’s conversational agent correctly preserved context across 93% of follow-ups, with failure concentrated in referent ambiguity cases (7%). PubMed retrieval yielded relevant abstracts for 73% of queries, with DEA retrieval coverage at 87%. Experts did not observe harmful misinformation or catastrophic factual errors.
Technical and Practical Implications
The demonstrated integration of regulatory and scientific RAG pipelines resolves several domain-unique requirements for health education:
- Authoritative, Verifiable Responses: The dual-source system eliminates the black-box opacity and hallucination risk typifying base LLM deployments.
- Context-Preserving Dialogue: Long-range conversational memory enables natural, pedagogically aligned questioning, supporting knowledge scaffolding in learners.
- Transparent Source Attribution: Fine-grained citation detail, including local match percentages and full article references, allows for independent verification and trust calibration.
- Epistemic Humility: The agent frequently enacts hedging when source coverage is ambiguous or incomplete, aligning with safe deployment principles in health applications.
However, response utility was occasionally diminished by technical density or citation clutter, especially when translating scientific abstracts for lay user contexts. Design improvements involving adaptive verbosity and citation summarization are needed.
The architecture is directly transferable to other domains where regulatory oversight and scientific evidence require fine integration—environmental health, occupational safety, and emergent public health threats, among others.
Theoretical Implications and Future Directions
The results substantiate the premise that retrieval-augmented, agentic AI architectures can meet the stringent demands of source verifiability, dynamic updating, and regulatory consistency critical for high-stakes education domains. By exposing internal reasoning and retrieval paths, the system provides a model for trustworthy, explainable AI in health communication.
The absence of a controlled baseline (e.g., DEA-only, PubMed-only, parametric-only LLMs) precludes direct causal attribution of dual-retrieval superiority. Furthermore, the system’s US-centric regulatory corpus limits global generalization—extension to multi-jurisdictional knowledge frameworks is essential for international relevance.
Critical next steps involve:
- Longitudinal User Studies: To evaluate educational impact, retention (Kirkpatrick Level 3), and behavior change in target learners.
- Controlled Comparative Benchmarks: Ablation testing to quantify performance deltas and failure modes.
- Adaptive Language and Interaction Design: Development of techniques for user literacy assessment, query clarification, and context disambiguation.
Conclusion
Agentic retrieval-augmented AI integrating regulatory and scientific knowledge demonstrates high performance in factual, regulatory, and contextual dimensions for substance use education. The dual-source approach offers a blueprint for transparent, evidence-based AI systems in public health communication, with implementation and evaluation strategies generalizable to parallel high-stakes domains. While robust empirical validation through user-centric outcome studies remains outstanding, the research provides a rigorous architecture and protocol for future extensible, trustworthy AI deployment in health education.
Reference: "Agentic AI for Substance Use Education: Integrating Regulatory and Scientific Knowledge Sources" (2605.00383)
