MasonNLP Submission: Clinical Order Extraction

• The paper introduces a prompt-engineered, few-shot pipeline using LLaMA-4 17B to extract clinical orders from lengthy multi-turn conversations.
• It defines a rigorous schema with controlled vocabularies for order type, description, reason, and provenance, ensuring clear mapping of conversational evidence.
• Evaluation shows strengths in scalability and schema flexibility, while challenges remain in implicit reason extraction and category ambiguity.

Medical order extraction refers to the automated identification of clinical directives (e.g., prescriptions, laboratory tests, imaging, or follow-up instructions) from unstructured sources, such as electronic health records (EHRs), discharge summaries, or extended doctor–patient dialogues. The goal is to convert these orders into actionable, structured data by determining the type of order, its description, clinical rationale, and supporting provenance (evidence from the conversation). This capability is foundational for streamlining documentation, powering clinical decision support, and enabling downstream automation in healthcare systems.

1. Task Definition and Schema

Structured medical order extraction requires transforming long, multi-turn conversational transcripts into a formalized representation. In the MEDIQA-OE 2025 shared task, the schema for each order consists of:

• order type: a controlled vocabulary (medication, laboratory, imaging, follow-up)
• description: detailed textual content for the order
• reason: free-text clinical justification, often implicit or context-dependent
• provenance: a set of supporting turn IDs from the dialogue evidencing the order

Transcripts typically span 95–100 turns, each turn annotated with both turn ID and speaker role. The extraction process must capture distributed evidence and express both explicit orders and implicit reasoning, a challenge compounded by conversational structure, speaker dynamics, and clinical language variation.

2. Model Architecture and Prompt Engineering

The MasonNLP submission utilizes a general-purpose, instruction-tuned LLM—LLaMA-4 17B—in a few-shot setting. No domain-specific fine-tuning was performed. The architecture exploits:

• Long contextual window (>8,000 tokens): necessary for handling extended multi-turn dialogues and distributed evidence.
• Instruction-following capability: allows prompt programming to enforce output structure, field types, and dialogue context retention.

Prompt engineering centers on a single, carefully selected in-context training example, directly incorporated before the transcript. This example specifies the schema, provides concrete output formatting (comma-separated fields for order type, description, reason, and provenance), and demonstrates how to handle missing fields using the “null” value. The prompt instructs the model to only use allowed order types, to cite provenance using explicit turn IDs, and to output a consistent structure amenable to downstream normalization.

A key design choice was to represent the transcript as line-by-line utterances annotated with speaker and turn ID (e.g., [turn_id] Speaker: Utterance). This preserves crucial dialogue structure and facilitates order provenance mapping. Free-form outputs, such as unformatted natural language, are systematically normalized to JSON objects in a post-processing pipeline.

3. Evaluation Metrics and Performance

System outputs were evaluated using standard metrics applied separately to each field:

• Rouge-1 F1: for description and reason, measuring overlap against reference text
• Strict F1: for order type, penalizing incorrect order category assignments
• MultiLabel F1: for provenance, measuring correspondence of predicted turn sets with ground truth

On held-out test data, the system achieved:

• Average F1: 37.76 (aggregate across fields)
• Description F1: 39.05
• Order type F1: 50.91
• Reason F1: 19.78
• Provenance F1: 41.32

Notably, the addition of a single in-context example markedly improved reason and provenance accuracy compared to a zero-shot configuration, underscoring the sensitivity of LLMs to minimal supervision for output grounding.

4. Analysis of Strengths, Limitations, and Error Modes

Significant strengths of the approach include:

• Scalability: The use of a general-purpose, instruction-tuned LLM with no domain-specific fine-tuning offers easy portability and baseline competitiveness for specialized clinical tasks without reliance on large annotated corpora.
• Flexible Schema Grounding: Prompt engineering allows for fast adaptation to variations in dialogue structure and labeling schema.

Key limitations identified:

• Implicit Reason Extraction: Many clinical justifications are diffuse, require multi-turn reasoning, or are only implicit in the dialogue—leading to low F1 in the “reason” field.
• Hallucination and Incomplete Outputs: The model sometimes produced orders with missing fields, fabricated content not present in the transcript, or cited non-permitted order categories; these required post-hoc normalization.
• Ambiguity in Category Assignment: Model outputs sometimes reflected semantic ambiguity, such as predicting “referral” or “surgery” where only medication, laboratory, imaging, or follow-up are allowed. This suggests further refinement of instructions and possibly domain adaptation.

5. Comparative Context and Significance

The MasonNLP submission ranked 5th out of 17 entries with 105 total submissions in the MEDIQA-OE 2025 shared task, placing among systems that employed specialized fine-tuning or retrieval-augmented generation (RAG) techniques. While methods such as WangLab and silver_shaw achieved higher overall F1 (circa 60), MasonNLP’s purely prompt-engineered, in-context learning setup offers a competitive baseline absent domain-specific retraining. This suggests that effective prompt design with modern LLMs can mitigate, but not fully replace, tailored adaptation for field-specific IE problems.

The transition from zero-shot to few-shot (as evidenced by gains in provenance and reason F1) further underscores the rapid incremental value of minimal supervised guidance.

6. Future Directions

A plausible implication is that integrating retrieval-augmented generation could enhance output grounding, improve implicit reason extraction, and reduce hallucinated outputs. Refining prompts—by including multiple exemplars, richer reasoning templates, or more rigorous schema enforcement—may yield additional improvements. Domain adaptation, such as fine-tuning on small clinical corpora or leveraging clinical knowledge bases, could close the performance gap especially for complex justification or nuanced order types. Further research is warranted in precision for temporal information (timing, dosage) and subtype granularity (laboratory test details).

7. Implications for Clinical NLP and Workflow Automation

The demonstrated methodology points to the feasibility of deploying non-domain-specific, instruction-tuned LLMs for structured clinical IE tasks at scale, provided that prompt structures are optimized and minimal supervision is available. This is particularly relevant for rapid prototyping and deployment in healthcare environments where large-scale annotated data and computational resources for fine-tuning may be limited. The explicit inclusion of provenance through cited turn IDs can support more transparent clinical documentation and traceable decision-support pipelines. However, for critical downstream applications, further efforts in improving grounding, accuracy, and interpretability remain essential.

In conclusion, structured medical order extraction via prompt-engineered, few-shot LLMs is a viable and scalable baseline that advances the intersection of conversational AI, clinical NLP, and medical workflow automation. Ongoing refinements in retrieval augmentation and domain-specific adaptation will likely define the next phase of research in this area (Karim et al., 12 Oct 2025).