CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents

Abstract: Clinical reasoning agents based on LLMs aim to automate tasks such as intensive care unit (ICU) monitoring and patient state tracking from electronic health records (EHRs). Existing systems typically rely on manually curated clinical tools or skills for concepts such as sepsis detection and organ failure assessment. However, maintaining these tool libraries requires substantial expert effort, while zero-shot querying or code generation often produces inefficient and unreliable reasoning chains, especially under institution-specific clinical policies. We introduce CodeClinic, a benchmark built on MIMIC-IV for evaluating whether LLM agents can synthesize and compose reusable clinical skills instead of relying on fixed toolboxes. The benchmark contains two complementary tasks: longitudinal ICU surveillance and compositional information seeking. The longitudinal setting simulates monitoring patient trajectories with structured decisions every four hours across 25 findings and eight clinical families, while the compositional setting spans 63k instances across 259 tasks in nine domains and is stratified by compositional dependency depth to evaluate increasingly complex multi-step reasoning. We further propose an offline autoformalization pipeline that converts natural-language clinical guidelines into reusable and verified Python skill libraries through iterative LLM refinement. Compared with zero-shot code generation, the resulting libraries improve consistency while reducing per-query token usage by up to 40%.

• The paper demonstrates that autoformalization of clinical guidelines into reusable code libraries significantly boosts accuracy and efficiency in LLM-based clinical reasoning.
• The benchmark employs longitudinal ICU surveillance and compositional queries over 63,000 clinical tasks to evaluate agent performance on temporal and multi-hop reasoning.
• The study reveals that agents using pre-verified function libraries outperform zeroshot and RL-finetuned models, achieving higher trajectory accuracy and improved token efficiency.

CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents

Introduction and Motivation

The CodeClinic benchmark addresses a key challenge in the automation of clinical reasoning with LLM agents: the synthesis and composition of modular, reusable clinical functions that can robustly operate over EHR data, instead of relying on static, expert-crafted toolboxes or inefficient zeroshot code generation. Clinical tasks such as ICU surveillance and dynamic information extraction from EHR systems demand stringent temporal reasoning, multi-step clinical concept composition, and adaptability to varying institutional policies. Existing approaches—either manually curated function libraries or static schemas—are not scalable and are often insufficient for handling the complexity and diversity of real-world ICU workflows.

Benchmark Design

CodeClinic is constructed atop the MIMIC-IV database, introducing two complementary evaluation axes:

1. Longitudinal ICU Surveillance: Agents perform structured patient monitoring every four hours throughout entire ICU stays. At each checkpoint, trajectory-aware decisions—across 25 clinical findings and eight organ/system families—are required, under strict causality and visibility constraints that prohibit future data leakage. The rolling history of prior decisions is continuously summarized and used for stateful reasoning.

   Figure 1: The longitudinal ICU surveillance loop, with the agent assessing patient state at 4-hour intervals, updating its memory, and outputting structured surveillance decisions.

2. Compositional Information Seeking: Encompasses 63,000 queries covering 63 clinical concepts in nine domains, each stratified by dependency depth to systematically probe the agent’s multi-hop compositional reasoning over clinical concepts. The dependency structure forms an explicit DAG, capturing levels from direct feature extraction to full clinical syndrome computation.

   Figure 2: State dynamics for key medical concepts; the model must track and update clinical states across evolving patient trajectories.

The compositional query specification integrates natural-language guidelines (synthesized from PubMed literature) and programmatic functions, enforcing both semantic fidelity and reproducibility. Representative queries include complex syndrome definitions (e.g., Sepsis-3) requiring dynamic recomputation of intermediate clinical states.

Figure 3: Dependency tree associated with a prototypical sepsis question (left) and the broader taxonomy of MIMIC-derived clinical concepts (right).

Clinical Autoformalization Pipeline

To address the dual need for automation and consistency in clinical code generation, CodeClinic introduces an offline "autoformalization" pipeline. This pipeline uses an LLM-driven ReACT loop for tool synthesis: guidelines and database access are provided as search functions to the agent, which then iteratively constructs, debugs, and verifies Python functions implementing each clinical concept. Code validation is performed on a small, labeled train split, with iterative error correction until a performance threshold is met. The best code artifacts are consolidated into a static library and used for efficient inference during testing.

Figure 4: Autoformalization pipeline overview—offline iterative code synthesis and verification produce a robust function library, deployed for efficient agent inference.

This separation of library construction from inference yields substantial improvements in reproducibility and token efficiency, as all agents can call into compact, pre-verified concept functions, amortizing both latency and cost.

Experimental Evaluation

Longitudinal ICU Surveillance Task

Agent performance on the longitudinal task is assessed using trajectory accuracy (correct sequential decision sequences across the entire 13-step, 48-hour horizon), per-checkpoint accuracy, and set-based F1 over suspected conditions and alert states. Results demonstrate that frontier LLMs (Claude-Sonnet-4.6, Gemini-3.1-Pro, GPT-5.4) achieve moderate per-checkpoint accuracy (66–76%) but much lower full-trajectory accuracy (just 25–34%), reflecting persistent errors in tracking time-evolving patient state and in propagating clinical syndromes across checkpoints.

Figure 5: Left—trajectory accuracy declines with increasing monitoring window; right—longitudinal metric summary (zeroshot agents), indicating sharp performance drop with longer horizons.

Autoformalization consistently improves models' ability to maintain accurate state, as the temporal semantics and recency-expiry rules are encoded in reusable functions, not re-derived at each turn.

Figure 6: Step-level action accuracy comparison—models equipped with autoformalization-derived tools outperform zeroshot code generation in longitudinal state tracking across checkpoints.

Compositional Information Seeking Task

Zero-shot agent baselines across major model families reveal that while simple, feature-level queries (Level 1) are solved with up to ~58% accuracy by the best models, multi-step composite queries (Level 3+) yield a pronounced performance decline (~37–41%). This trend is consistent across closed and open models, underscoring the current limits of LLM compositional clinical reasoning on realistic EHR data.

The autoformalization baseline demonstrates absolute accuracy improvements (5–7 points) over both zeroshot code generation and RL-finetuned agents, with significant reductions in per-query token usage (up to 40%). Notably, the advantage of code library reuse over parameter finetuning is magnified on deep compositional queries, indicating the importance of explicit, verified procedural abstraction for complex clinical logic.

Figure 7: Across model families, the autoformalization approach universally yields higher compositional query accuracy and improved token efficiency.

Policy Implications and Contradictory Results

A key claim validated in the paper is that RL-based finetuning (GRPO)—while improving on shallow database patterns—fails to scale to deep clinical reasoning, due to the combinatorial explosion of dependency paths and the limited expressive power for synthesizing modular code. In contrast, agents leveraging autoformalized libraries attain higher accuracy at all stratification levels, with lower cost and consistent temporal logic, challenging prevailing assumptions about parameter tuning as the dominant approach for clinical code automation.

Practical and Theoretical Implications

Practically, CodeClinic demonstrates that autoformalization of natural-language guidelines into robust function libraries is feasible at scale given only limited supervision. This directly reduces the need for ongoing expert-tool curation, facilitating rapid, policy-adaptive deployment of reasoning agents in clinical contexts. The approach inherently improves reproducibility and safety, as all outputs are derived from traceable, verifiable computation traces, which is a notable requirement in clinical auditing and regulation.

Theoretically, this work exposes the structural constraints of current LLMs—explicit, tool-based modularization is necessary to overcome the limitations of direct end-to-end code generation for tasks with deep compositionality or temporal state dependence. It also suggests that the next advances in clinical agent design will require hybrid architectures, where LLMs efficiently orchestrate modular, validated code assets in agentic loops, rather than relying on implicit memorization or shallow pattern matching.

Future Directions

Potential extensions include:

• Incorporation of multimodal EHR data (e.g., notes, imaging) into the autoformalization pipeline.
• Development of domain-adaptive or institution-specific function libraries, with continual verification against site-specific practices.
• Deployment and prospective evaluation of autoformalized agents as co-pilots in real ICU environments, with comprehensive monitoring of rare failure modes and generalization across health systems.
• Investigation of methods to learn not just atomic skills, but generalized concept dependency graphs and transformation templates, enabling even more robust code reuse.

Conclusion

CodeClinic provides a rigorous, agent-oriented benchmark for evaluating clinical reasoning automation with LLMs, and demonstrates that autoformalization pipelines—converting guidelines to verified, reusable function libraries—yield significant accuracy, consistency, and efficiency gains over traditional zeroshot or RL-finetuned approaches. This paradigm offers a scalable path toward flexible, institutionally-grounded, and auditable clinical reasoning agents, with broad implications for the design and deployment of medical AI systems (2605.09675).