MedPAO: Protocol-Driven Clinical Report Structuring

• MedPAO is a protocol-driven framework that structures free-text chest X-ray reports using the ABCDEF schema for precise clinical categorization.
• It integrates specialized tools and a Plan-Act-Observe loop with an LLM engine to extract, map, and verify clinical concepts from radiology reports.
• Evaluations demonstrate high macro F1 scores and improved transparency over traditional opaque LLM methods in clinical report structuring.

MedPAO is a protocol-driven, agentic framework for turning free-text medical reports into structured, protocol-compliant data through a Plan-Act-Observe (PAO) loop, specialized tools, and an LLM engine. In the reported instantiation, it focuses on chest X-ray reports and the ABCDEF protocol, with the explicit objective of producing inspectable, verifiable reasoning rather than a single black-box LLM output. The framework is designed to address hallucinations, weak protocol adherence, and the opacity of monolithic LLM systems in clinical report structuring (Vaidya et al., 6 Oct 2025).

1. Problem Setting and Rationale

Radiology reports are typically written in free text, contain rich clinical observations, vary across radiologists in style, ordering, and terminology, and rarely preserve the systematic structure radiologists use internally. This makes it difficult to aggregate data for research or quality improvement, build predictive models, and audit reports or enforce adherence to clinical protocols. MedPAO addresses this problem in the specific setting of chest X-ray report structuring under the ABCDEF protocol (Vaidya et al., 6 Oct 2025).

The framework is motivated by limitations of standard LLM approaches such as direct prompting, JSON-style extraction prompts, and fine-tuning. In this setting, such approaches may hallucinate findings not present in the report, misclassify findings into incorrect protocol sections, omit important protocol categories, and produce outputs that appear structured while violating clinical workflow logic. They also provide no verifiable reasoning in the sense of an explicit, stepwise mapping from raw text to extracted concepts, protocol categories, and final structured output (Vaidya et al., 6 Oct 2025).

A central contrast in the MedPAO formulation is between protocol-grounded modular processing and “opaque, monolithic LLM systems.” The latter are described as opaque, hard to audit, not modular, and not protocol-grounded, because there is no explicit programmatic enforcement that every concept must be categorized under ABCDEF or that a category remains empty only if no concepts of that type exist. This suggests that MedPAO is positioned not merely as an extraction pipeline, but as a clinically constrained reasoning architecture (Vaidya et al., 6 Oct 2025).

2. Protocol Grounding and the ABCDEF Schema

MedPAO treats the clinical protocol as the central reasoning scaffold. In the reported study, the protocol is ABCDEF for chest X-ray analysis: A for Airways, B for Lung fields and pleura, C for Cardiome­diastinal contours and great vessels, D for Diaphragm and subdiaphragmatic structures, E for External anatomy, and F for Foreign bodies and devices (Vaidya et al., 6 Oct 2025).

The protocol serves both as a target schema for the output and as a reasoning guide for concept categorization. The exact textual protocol description from Jones et al. (2025) is provided as context to the LLM during concept categorization and report generation. Protocols are represented as textual descriptions and as an implicit schema in which each extracted concept must be assigned a label in $\{A, B, C, D, E, F\}$, and the final report is organized as sections for A–F populated by categorized concepts (Vaidya et al., 6 Oct 2025).

This protocol grounding constrains reasoning at multiple stages. During concept categorization, the model sees the ABCDEF protocol description, the concept string, and its SNOMED/RADLEX hierarchies, and must assign exactly one ABCDEF category. During report generation, the model is instructed to generate findings organized under A–F headings, to use the original source sentences for phrasing to avoid hallucinations, and to populate categories only with concepts that were actually extracted. A plausible implication is that the protocol acts as both a semantic prior and an output validator, narrowing the space of admissible outputs (Vaidya et al., 6 Oct 2025).

3. Agent Architecture and the Plan-Act-Observe Loop

MedPAO is implemented as a protocol-driven agent built around three core elements: an LLM engine, a Model Context Protocol (MCP) client/server framework, and a Plan-Act-Observe loop. The reported LLM engine is DeepSeek-R1-70B, used for planning, observing, ontology filtering, concept categorization, and report generation. The MCP client implements PAO logic and maintains context about executed tools and outputs, while the MCP server hosts and executes the tools (Vaidya et al., 6 Oct 2025).

The PAO loop is iterative. In the Plan phase, the MCP client prompts the LLM with the user’s request, the available tools, and any previously executed steps, and asks what should be done next and which tool should be called with what parameters. In the Act phase, the MCP server executes the requested tool and returns structured outputs. In the Observe phase, the LLM receives the original query, accumulated tool outputs, and the protocol description, and decides whether the user’s request has been satisfied or whether further iterations are necessary (Vaidya et al., 6 Oct 2025).

This architecture is intended to make the reasoning process explicit and inspectable. The paper describes the framework as producing a “transparent process managed by a Plan-Act-Observe (PAO) loop and specialized tools,” and as providing a “verifiable alternative to opaque, monolithic models.” That verifiability is operationalized through modular intermediate artifacts rather than through an end-to-end latent representation alone (Vaidya et al., 6 Oct 2025).

4. Tooling, Workflow, and Intermediate Representations

The MedPAO pipeline comprises six specialized tools: get concept, Ontology Mapping, Ontology Filtering, Concept Categorization, Report Generation, and check cache. The first tool extracts clinical concepts and source sentences from the free-text report. Ontology Mapping uses the NCBO BioPortal Annotator API with SNOMED-CT and RADLEX, with lemmatization applied before querying because BioPortal uses rule-based term matching. Ontology Filtering uses DeepSeek-R1-70B with few-shot prompting to distinguish a primary ontology from secondary ontologies. Concept Categorization maps concepts to ABCDEF categories, and Report Generation produces protocol-structured report text. The check cache tool reuses previously computed results to avoid recomputation (Vaidya et al., 6 Oct 2025).

The concept extraction model is a fine-tuned MedLlama-8B (MedLlama-3 variant), using LoRA. Its training data comprised 40 real annotated chest X-ray reports from Eye Gaze MIMIC-CXR, manually annotated by a radiologist/clinician, and 160 synthetic report-concept pairs generated using GPT-4, Gemini 2.5-Flash, Claude 3.5, and DeepSeek-R1 with few-shot prompts and constrained vocabularies from SNOMEDCT and RADLEX. The final dataset contained 200 samples and 311 unique medical concepts (Vaidya et al., 6 Oct 2025).

The task can be expressed as a staged mapping from input text to structured report. The input is freestyle chest X-ray report text $T$. The first intermediate output is a set of medical concepts $C = \{c_1, \dots, c_n\}$. The second is ontology mappings for each concept. The third is a categorization function $g: C \to \mathcal{K}$ where $\mathcal{K} = \{A,B,C,D,E,F\}$. The final output is a structured representation aligned with ABCDEF, in textual or JSON form. The paper describes the pipeline as:

1. extraction,
2. ontology mapping,
3. categorization,
4. report generation (Vaidya et al., 6 Oct 2025).

A representative example in the study maps “right pleural effusion” and “basilar atelectasis” to B, “cardiomegaly / enlarged cardiac silhouette” to C, and “right internal jugular central venous catheter” to F, then generates an ABCDEF-structured report in which each finding appears under the corresponding section. This suggests that MedPAO is designed not only for concept normalization but also for clinically legible report re-expression (Vaidya et al., 6 Oct 2025).

5. Evaluation and Reported Performance

Because publicly available ABCDEF-aligned structured reports were lacking, the authors created a ground-truth dataset by extracting findings from MIMIC-CXR reports and annotating each report with clinically significant concepts and an ABCDEF category label for each concept. A custom annotation tool was used by radiologists and clinicians, and the final evaluation set contained 200 ground-truth samples (Vaidya et al., 6 Oct 2025).

The study evaluated three components: concept extraction, concept categorization, and report generation quality. For concept extraction, baselines included DeepSeek-R1-Distill-Qwen-7B, Qwen2.5-7B, and DeepSeek-R1-70B, with metrics reported at 80% and 90% fuzzy matching thresholds. At 80% similarity, the get concept tool achieved macro F1 of 0.86, weighted F1 of 0.91, subset accuracy of 0.28, and Hamming loss of 0.0023. At 90% similarity, it achieved macro F1 of 0.80, weighted F1 of 0.84, and subset accuracy of 0.20. McNemar’s tests showed statistically significant improvements over smaller baselines (Vaidya et al., 6 Oct 2025).

For concept categorization, MedPAO was compared with DeepSeek-R1-Distill-Llama-70B and Qwen2.5-72B-Instruct used in prompt-based fashion without agentic scaffolding. The reported performance for MedPAO was macro precision 0.95, weighted precision 0.97, macro recall 0.94, weighted recall 0.96, macro F1 0.94, and weighted F1 0.96, with Jaccard similarity of 0.92 macro and 0.94 weighted. The F1-score highlighted in the abstract corresponds to the weighted average F1 for concept categorization. Confusion matrices showed more frequent cross-category misclassifications for baseline models and a more diagonal confusion matrix for MedPAO (Vaidya et al., 6 Oct 2025).

Human evaluation was performed on 50 generated reports by one clinician and a panel of two radiologists, using 1–5 scales for accuracy and structure. The clinician’s average accuracy was 4.64 and average structure 4.59. The radiologist panel’s average accuracy was 4.52 and average structure 4.48. The abstract’s headline score of 4.52/5 refers to the radiologists’ average accuracy rating. The paper states that these scores surpass baseline approaches relying solely on LLM foundation models, although detailed baseline human ratings are not tabulated (Vaidya et al., 6 Oct 2025).

6. Interpretability, Limitations, and Position in the Literature

A defining property of MedPAO is its emphasis on transparency and verifiability. Each processing stage—extraction, ontology mapping, categorization, and report generation—is separate and inspectable. Concepts are linked to source sentences in the original report, categorization is constrained by the ABCDEF protocol, and the final report is derivable from intermediate artifacts. The paper characterizes this as “verifiable reasoning,” meaning that the reasoning is decomposed into modular tools whose inputs and outputs can be audited against protocol and ontology rules (Vaidya et al., 6 Oct 2025).

The framework also has explicit limitations. It depends on well-defined protocols such as ABCDEF, so domains without standardized protocols would require protocol design before deployment. It required expert annotation effort for evaluation and concept extraction fine-tuning. Although presented as modality-agnostic in principle, each new domain would require a clear protocol and possibly a new concept extraction fine-tuning dataset. The system also incurs engineering overhead from ontology integration, caching, and MCP integration, and its performance depends on the availability and stability of large models such as DeepSeek-R1-70B. Latency can be significant without caching, reaching up to approximately 280 seconds per end-to-end task on Gaudi2 (Vaidya et al., 6 Oct 2025).

Within the broader literature, MedPAO is positioned against earlier rule-based or supervised clinical text structuring systems, single-shot LLM extraction methods, and more generic multi-agent architectures. Its stated novelty lies in treating a clinical protocol as the central reasoning backbone of an LLM-driven agent, combining fine-tuned concept extraction, ontology-based grounding with SNOMED-CT and RADLEX, and protocol-driven categorization and generation. The paper presents it as a step toward safe, transparent, protocol-aligned AI agents for medical report structuring, rather than as a generic prompt-engineering strategy (Vaidya et al., 6 Oct 2025).