Infherno: End-to-End FHIR Synthesis

• Infherno is an end-to-end, agent-based system that converts free-form clinical notes into structured HL7 FHIR resources using LLM-driven orchestration.
• It integrates real-time Python code execution with external terminology lookup to enforce strict FHIR schema validation and ensure data accuracy.
• Evaluations demonstrate high fidelity to human annotations, low hallucination rates, and potential for scalable clinical data integration.

Infherno is an end-to-end, agent-based system for synthesizing structured HL7 FHIR (Fast Healthcare Interoperability Resources) resources from free-form clinical notes, designed to address critical challenges of clinical data integration and healthcare interoperability. Utilizing a LLM agent framework, real-time code execution, and external healthcare terminology lookup, Infherno guarantees conformance to the FHIR schema and demonstrates high fidelity against human-annotated references in predicting FHIR resources from unstructured health texts (Frei et al., 16 Jul 2025).

1. System Architecture and LLM Agent Coordination

Infherno is architected as a single multi-step LLM “agent” based on the Smolagents library. Its operation closely follows the ReAct (Reason–Act) paradigm, where the agent interleaves natural language “thoughts,” external tool invocations, and direct Python code generation to iteratively build and validate FHIR resource objects in memory.

The agent interacts with three principal classes of tools:

• Terminology Lookup (“Code Search”): HTTP-based querying against a FHIR terminology server (e.g., SNOMED CT) to retrieve concept codes from specified ValueSets.
• In-Sandbox Code Execution: Dynamic execution of Python leveraging the fhir.resources library (version 8.0.0), enabling real-time construction and schema validation of FHIR resources.
• Final Serialization: Collection and transformation of all in-memory resources into a valid FHIR R4 Bundle JSON.

At each agent cycle, state management is handled via two main context sources:

• The system prompt, which incorporates the patient’s note, target resource types (commonly Patient, Condition, MedicationStatement), permitted ValueSets, and sample code snippets.
• The full dialog history of “Thoughts,” “Tool Calls,” and “Observations,” supporting iterative, context-aware reasoning.

The full coordination loop is driven by the following pseudocode structure (LaTeX style):

Initialize prompt_context with: – clinical_note, target_types, ValueSets, example_code history ← [] while True do agent_output ← LLM(prompt_context, history) if agent_output.call == "CodeSearch" then result ← CodeSearchTool(agent_output.query, agent_output.valueset) history.append((agent_output, result)) elseif agent_output.contains_python then try exec(agent_output.python_code) # uses fhir.resources obs ← "Code executed successfully" except Exception as e: obs ← str(e) history.append((agent_output, obs)) elseif agent_output.final_answer then bundle_json ← collect_all_resources().as_json() return bundle_json else history.append((agent_output, "No-op")) end if end while

All information extraction, terminology mapping, and schema validation are embodied within this single agent process.

2. Schema Validation and FHIR Conformance

Infherno achieves strict FHIR conformance by delegating resource construction to the fhir.resources library, which implements all HL7 FHIR R4 constraints at the class and field level:

• Required attributes, value types, enumerations, and cardinalities are enforced during Python object instantiation.
• Any schema violation (e.g., omitted required field, wrong type) triggers a Python exception with a diagnostic message.
• The agent, upon receiving a schema exception as an “Observation,” re-enters the reasoning loop to iteratively correct the error, effectively implementing a lazy constraint-solving process.

As a result, every serialized FHIR Bundle produced by Infherno is guaranteed syntactically and semantically valid prior to output.

3. Implementation: Models, Tooling, and Execution Environment

The system’s core LLM agent experiments utilized Gemini-2.5-Pro via the Gemini API, although integration with OpenAI GPT-4 and local Hugging Face models is supported.

• Prompt design embeds the overall task definition, permitted FHIR types/ValueSets, and example fhir.resources code, with explicit instructions to always invoke CodeSearch before populating new codeable concepts.
• Code execution runs inside an isolated Smolagents Python sandbox, preloaded with fhir.resources and a thin CodeSearch client. Execution is time-limited and OS-isolated to mitigate security risk.
• An example agent dialogue illustrates the pipeline:
  • Thought: "I need a SNOMED code for ‘erosive gastritis’.”
  • Action: CodeSearch(query="erosive gastritis", valueset="SNOMED_CT:Condition")
  • Observation: [{'code':'235719002','display':'Erosive gastritis'}]
  • Thought: "Now instantiate Condition with that code and link it to the patient."
  • Action (Python): code creating a new Condition resource with the returned SNOMED code.

All previous steps and generated code, including errors, are part of the evolving dialog state.

4. Evaluation Data and Performance Metrics

Infherno’s evaluation was performed using ten synthetic German discharge letters, manually post-edited for realism. Human-annotated FHIR resources (Patient, Condition, MedicationStatement) serve as the gold standard.

Comparison between model output (“Predicted,” PD) and the human baseline (HB) is performed at the atomic “item” level (individual fields or codings), with a nuanced tagging system for matches, semantic equivalence, differences, missingness, and hallucinations.

Key results:

• Total items: 446.
  • Neutral (exact or semantically equal): 314
  • Better than HB: 86
  • Worse than HB: 46
  • Hallucinations (X-tagged): 10/446 (2.3%)
• Syntactic conformance: 100% of Bundles parse as valid FHIR R4 due to library enforcement.
• Precision/Recall/F1 (approximate calculations, not reported directly):
  • Precision $\approx 0.76$
  • Recall $\approx 0.94$
  • F1 $\approx 0.84$
• Relative performance: Infherno produced strictly more correct items than the human annotator in 86 cases, fewer mistakes in 46 cases, and hallucinated only 2.3% of items.

5. Failure Modes and Limitations

Observed error sources include:

• Ambiguous or insufficiently informative clinical text, causing omission of non-critical FHIR fields.
• Forced or suboptimal resource encodings due to limitation to Patient, Condition, and MedicationStatement types (e.g., encoding measurements or findings that may be better represented by Observation).
• Occasional semantic hallucination (incorrect code assignment), frequently linked to imprecise CodeSearch queries.

Identified limitations and deployment considerations:

• Dataset comprises only ten synthetic German discharge notes; broader generalization to varied clinical note sources remains untested.
• Absence of inter-annotator agreement, with a single annotator serving as the human baseline.
• Dependence on proprietary LLMs (Gemini-2.5-Pro) for current performance benchmarks; the impact of smaller, open models remains to be established.
• SNOMED CT licensing/hosting constraints could impact applicability in some institutional settings.
• Secure sandboxing and ValueSet curation are required for robust, institution-scale deployment.

Mitigation proposals include extending supported resource types, applying ensemble CodeSearch strategies, agent confirmation loops for critical codes, and fine-tuning open-weights LLMs on text-to-FHIR datasets.

6. Comparative Advantages and Integration Context

Infherno occupies a distinct position versus previous approaches, combining:

• Agentic LLM design with explicit, modular tool calls for terminology lookup and schema validation.
• On-the-fly code-based FHIR resource assembly, intercepting errors prior to Bundle finalization.
• Explicit enforcement of schema constraints through reliable third-party Python libraries (fhir.resources), a property lacking in pure text-generation systems.

The architecture supports deployment as an interchangeable backbone for automated FHIR generation, promoting institution-to-institution interoperability and facilitating real-world clinical data integration. By attaining lower hallucination rates and improved recall compared to baseline annotators—and guaranteeing schema conformance algorithmically—Infherno addresses longstanding reliability and generalizability challenges in free-text-to-FHIR translation (Frei et al., 16 Jul 2025).