Health System Scale Semantic Search Across Unstructured Clinical Notes

Abstract: Introduction: Semantic search, which retrieves documents based on conceptual similarity rather than keyword matching, offers substantial advantages for retrieval of clinical information. However, deploying semantic search across entire health systems, comprising hundreds of millions of clinical notes, presents formidable engineering, cost, and governance challenges that have prevented adoption. Methods: We deployed a semantic search system at a large children's hospital indexing 166 million clinical notes (484 million vectors) from 1.68 million patients. The system uses instruction-tuned qwen3-embedding-0.6B embeddings, stores vectors in a managed database with storage-optimized indexing, maintains full-text metadata in a low-latency key-value store, and operates within a HIPAA-compliant governance framework. We evaluated the system through three experiments: optimization of embedding model and chunking strategy using a physician-authored benchmark dataset, characterization of full-scale performance (cost, latency, retrieval quality), and clinical utility assessment via comparison of chart abstraction efficiency across three tasks. Results: The system delivers sub-second query latency (median 237 ms single-user, 451 ms 20-user concurrency) with monthly costs of approximately USD 4,000. Qwen3 embeddings with 300-token chunk size achieved 94.6% accuracy on a clinical question-answering benchmark. In clinical utility evaluation across three abstraction tasks, semantic search reduced time-to-completion by 24 to 89% compared to clinician-performed chart review while maintaining comparable inter-rater agreement. Conclusion: Health-system-scale semantic search is both technically and operationally feasible. The system provides infrastructure supporting interactive search, cohort generation, and downstream LLM-powered clinical applications without requiring specialized informatics expertise.

• The paper introduces a scalable semantic search infrastructure processing 166 million clinical notes with dense vector retrieval.
• The approach uses 300-token chunking and instruction-tuned embeddings to achieve sub-second query latency while reducing operational costs.
• The system delivers significant efficiency gains in clinical abstraction through metadata filtering and robust evaluation of embedding models.

Health System Scale Semantic Search for Clinical Notes: Institutional Infrastructure and Clinical Utility

System Architecture and Design Choices

The paper "Health System Scale Semantic Search Across Unstructured Clinical Notes" (2604.25605) describes the successful deployment of semantic retrieval infrastructure over 166 million clinical notes from 1.68 million patients within a large pediatric healthcare system. The architecture is composed of four primary modules: text chunking and embedding, a managed vector database for approximate nearest-neighbor search, a cost-efficient key-value store for full-text and metadata, and a web-based querying interface. Key design considerations include chunking clinical notes into semantically coherent 300-token segments with overlap, leveraging the instruction-tuned qwen3-embedding-0.6B for vectorization, and partitioning storage to minimize cloud costs without sacrificing retrieval speed.

The separation of dense vector storage from full-text content was critical for economic scalability: storing the full corpus (484 million vectors) in a storage-optimized index reduced operational costs to approximately $$4,000/month, which is an order of magnitude lower than in-memory alternatives. The vector search utilizes ScaNN and SOAR algorithms to achieve sub-second query latency, and incremental indexing allows seamless updating as new clinical notes are authored.</p> <h2 class='paper-heading' id='embedding-model-and-retrieval-optimization'>Embedding Model and Retrieval Optimization</h2> <p>The evaluation of embedding models—qwen3-embedding-0.6B, nomic embed, E5 large, and BioClinical BERT—demonstrated the superiority of instruction-tuned architectures (qwen3-embedding-0.6B) in representing clinical concepts. On a physician-crafted clinical question-answering benchmark (CHOP_MCQA_v0.5), the selected 300-token chunk setup reached 94.6 <p>Error analysis revealed temporal misalignment and document-type confounds as primary sources of retrieval error, underscoring the necessity for metadata-based filters (patient, date, note type, specialty, author, etc.) to improve precision for information-centric queries. The chunking mechanism, although effective, points to future directions involving semantically-aware segmentation to enhance concept isolation further.</p> <h2 class='paper-heading' id='clinical-workflow-and-efficiency-outcomes'>Clinical Workflow and Efficiency Outcomes</h2> <p>The practical utility of the semantic system was evaluated through controlled abstraction tasks reflecting real research workflows—documenting genetic diagnoses, seizure onset age, and discovering ballet-related foot injury cohorts. In each scenario, semantic search decreased abstraction time by substantial margins (24–89$$K = 0.945$, Krippendorff’s$\alpha=0.950$) confirmed equivalence in data fidelity, indicating that efficiency gains did not come at the cost of accuracy.

The infrastructure empowers researchers to interactively query the full patient population, supporting rapid cohort identification and iterative chart review that scale efficiently across clinical specialties. Furthermore, semantic search reliably surfaced important information from neglected note types (e.g., nurse triage and telephone encounters), highlighting the inadequacy of keyword-centric approaches for comprehensive clinical abstraction.

Governance, Security, and Institutional Integration

Operating within a HIPAA-compliant Google Cloud environment (Arcus), the system employs project-level containerization, granular allowlists, query logging, and standardized IRB protocols to satisfy security and privacy mandates. Specially protected health information is programmatically excluded, and all accesses are auditable, enabling scalable institutional adoption. The framework’s modularity and public reference implementation facilitate porting to other health systems, EHR vendors, and cloud providers, pending adaptation of configuration parameters and governance rules.

Implications, Limitations, and Future Directions

The technical demonstration establishes semantic search as an infrastructural solution rather than a bespoke application. The authors provide strong empirical evidence that dense retrieval, when architected for cost and usability, is viable for health-system-scale deployment—contradicting assumptions regarding prohibitive latency and expense.

Practically, this infrastructure enables interactive search, cohort generation, and downstream LLM-powered reasoning (RAG) across massive unstructured clinical corpora, extending access beyond technical informaticians to non-expert clinical researchers. Theoretically, it lays groundwork for integrating real-time, multimodal retrieval systems and enhancing factual grounding for LLMs in clinical applications. As healthcare institutions increasingly digitize and automate workflows, such infrastructure positions semantic search as a foundational component for both clinical research and operational intelligence.

Limitations include the single-institution deployment, the exclusion of multimodal data (images, tabular), and reliance on fixed-size chunking with heuristics rather than content-aware segmentation. Extending the system to handle multimodal retrieval and concept-based chunking, as well as validating performance across diverse documentation practices and EHR environments, are necessary future steps.

Conclusion

This study rigorously demonstrates the feasibility of institution-scale semantic search across unstructured clinical notes, achieving sub-second retrieval latency and operational cost efficiency. Dense retrieval infrastructure, when combined with high-performance embedding models and governance-aware metadata filtering, delivers substantial gains in chart abstraction efficiency with maintained accuracy. The results inform scalable architectures for clinical search and RAG in health systems, setting the stage for future expansion into multimodal indexing, automated concept segmentation, and broader institutional integration.