- The paper introduces FastOMOP, an open-source architecture that separates governance, observability, and orchestration so agent teams can generate OMOP-based real-world evidence with deterministic safety controls.
- The NL2SQL proof of concept achieved overall R₀ scores of 0.94 on Synthea, 0.865 on IDRIL, and 0.84 on MIMIC-IV, while performance declined for queries involving three or more concepts and certain drug mappings.
- FastOMOP correctly flagged all 20 out-of-scope questions and blocked all 20 adversarial queries through process-boundary MCP governance, but broader red-team testing is needed to validate its safety claims.
Motivation and problem statement
The OMOP Common Data Model (CDM) has become the de facto standard for harmonising electronic health record (EHR) data, with OHDSI networks spanning nearly one billion patients across 83 countries. Yet converting these repositories into real-world evidence (RWE) remains a bespoke, manual process requiring a rare combination of clinical, epidemiological and technical expertise. LLMs have shown promise for discrete subtasks such as natural-language cohort definition and text-to-SQL translation, but the EHRSQL 2024 shared task demonstrated that even state-of-the-art LLMs failed to achieve positive reliability scores under clinical safety constraints. The authors argue that this failure reflects an architectural rather than a capability gap: monolithic model calls cannot simultaneously provide flexible clinical reasoning and the deterministic safety guarantees required for production deployment, while multi-agent systems introduce emergent behaviours, coordination failures and cascading errors that existing governance approaches do not address.
The FastOMOP architecture
FastOMOP is an open-source foundational multi-agent architecture built on four design principles: separation of concerns, safety by design, least privilege, and complete observability. Its central claim is that these principles should be realised as independent infrastructure layers from which specialised agent-teams inherit guarantees automatically:
- Governance layer: deterministic, rule-based validation enforced at the process boundary between agents and external systems, implemented within Model Context Protocol (MCP) servers. It combines operation whitelisting, dangerous-operation blacklisting, and institutional policy enforcement, with all rules defined in external configuration files for institutional customisation.
- Observability layer: automatic capture of full execution traces — reasoning steps, tool invocations, data transformations — with prompt versioning to support reproducibility and A/B testing. Traces also enable programmatic benchmarking through trace ingestion.
- Orchestration layer: an autonomous planning agent that dynamically decomposes tasks and delegates to sub-agents, constrained by structural guardrails (e.g., semantic concept resolution must precede SQL generation). Context is preserved across hand-offs via structured context objects, so semantic interpretations such as resolving "diabetic" to Type 2 Diabetes Mellitus are not re-derived downstream.
Agent-teams are pluggable components satisfying interface contracts; any future team (phenotyping via ATLAS, study design via HADES, data quality via ACHILLES) inherits the same safety boundaries without reimplementing trust infrastructure.
Proof-of-concept NL2SQL agent team
To validate the architecture, the authors implemented a three-agent NL2SQL team: a Planning Agent for decomposition and synthesis, a Semantic Agent mapping clinical language to OMOP concept IDs (SNOMED CT, RxNorm, LOINC) via concept_ancestor navigation with ranked candidates for ambiguous mappings, and a SQL Agent operating exclusively through a sandboxed MCP server exposing read-only metadata retrieval and query execution against whitelisted tables. The implementation uses Python 3.13, the Agno orchestration framework, Pydantic-validated inter-agent communication, ibis-framework with sqlglot for query transpilation, and a provider abstraction supporting OpenAI, Azure, Anthropic and Ollama endpoints, including air-gapped deployments. Default configuration used locally hosted gpt-oss:120b at temperature 0.1 for all agents.
Evaluation
The system was evaluated on 2,047 natural-language–SQL–result triplets generated by the FOEM framework from 56 NOSTOS templates instantiated per dataset, spanning single-concept, multi-concept, temporal, aggregation and complex clinical categories. Three OMOP CDM v5.4 datasets were used: Synthea synthetic data (n=680), MIMIC-IV in OMOP format (n=1,101), and IDRIL, a real-world NHS dataset from Lancashire Teaching Hospitals covering over two million patients (n=266). Two model configurations were tested after a calibration phase on one query per template, with prompts fixed thereafter.
| Dataset |
n |
Overall R₀ |
| Synthea |
680 |
0.94 |
| IDRIL |
266 |
0.865 |
| MIMIC-IV |
1,101 |
0.84 |
Performance was strongest on temporal queries (R₀ = 0.897–0.971 across datasets) and complex/other queries (up to 1.000 on MIMIC-IV and IDRIL), while drug-related conjunctions were weaker (drug AND drug: 0.589 on MIMIC-IV; single drug: 0.500 on IDRIL). Accuracy dropped sharply to approximately 0.4 for queries involving three or more concepts, attributed to context-window exhaustion from repeated tool calls. On reliability, all 20 out-of-scope questions were correctly flagged (OBR = 1.0), and the MCP-layer governance blocked all 20 adversarial queries — including DDL/DML attacks, PHI exfiltration attempts, privilege escalation and prompt-injection patterns — yielding ABR = 1.0.
These results support the paper's central claim: process-boundary governance delivers safety guarantees independent of model choice, prompt configuration or task complexity, because validation executes in a separate process and is architecturally unreachable from agent reasoning. A fully compromised or hallucinating agent still cannot execute an unsafe query. This directly addresses the abstention failures documented in EHRSQL 2024 and contrasts with in-agent guardrails (prompt instructions, output filters, self-consistency checks), which remain dependent on the model's reasoning integrity.
Limitations and open questions
The paper is explicit about several constraints. First, the validated workflow is reactive rather than proactive: the system handles closed, well-defined questions but cannot support open-ended exploratory analysis ("explore why treatment Z is failing"), which would require evolving the orchestration layer from a static supervisor to a dynamic planner capable of generating novel workflows. Second, the multi-concept accuracy degradation stems from the Semantic Agent's reliance on SQL LIKE fuzzy matching for concept retrieval; the proposed remedy — a semantic knowledge base with embeddings and hierarchical relationships in a vector or hybrid vector-graph store — is identified but not implemented. Third, and most significantly for the headline safety claim, the adversarial evaluation comprised only 20 hand-crafted queries. This demonstrates the governance mechanism but does not characterise robustness under systematic attack; larger-scale red-teaming with diverse vectors including prompt injection and multi-step circumvention is required before confidence bounds on the safety guarantee can be established. Finally, R₀ of 0.84–0.94, while strong relative to prior benchmarks, still leaves a non-trivial error rate on answerable queries, and per-category weaknesses (notably drug-related mappings on real-world data) remain unresolved.
Conclusion
FastOMOP contributes a foundational, open-source architecture that separates governance, observability and orchestration as infrastructure layers beneath pluggable agent teams, validated on an NL2SQL proof-of-concept across synthetic, US critical-care and NHS real-world OMOP CDM data. With R₀ scores of 0.84–0.94 and perfect adversarial and out-of-scope block rates, the results indicate that the reliability gap in agentic RWE generation is architectural rather than a matter of model capability, and that deterministic process-boundary validation provides model-independent safety guarantees. The immediate roadmap is integrating HADES, ACHILLES and DARWIN EU tooling as governed tools within the same boundaries; the principal open question is whether the governed foundation can be extended to proactive cognitive workflows without diluting the determinism that underpins its safety properties.
