Automatic Generation of Executable BPMN Models from Medical Guidelines

Abstract: We present an end-to-end pipeline that converts healthcare policy documents into executable, data-aware Business Process Model and Notation (BPMN) models using LLMs for simulation-based policy evaluation. We address the main challenges of automated policy digitization with four contributions: data-grounded BPMN generation with syntax auto-correction, executable augmentation, KPI instrumentation, and entropy-based uncertainty detection. We evaluate the pipeline on diabetic nephropathy prevention guidelines from three Japanese municipalities, generating 100 models per backend across three LLMs and executing each against 1,000 synthetic patients. On well-structured policies, the pipeline achieves a 100% ground-truth match with perfect per-patient decision agreement. Across all conditions, raw per-patient decision agreement exceeds 92%, and entropy scores increase monotonically with document complexity, confirming that the detector reliably separates unambiguous policies from those requiring targeted human clarification.

• The paper presents an end-to-end pipeline that automatically generates executable BPMN models from medical guidelines, integrating LLM-driven document processing and iterative repair loops.
• It validates the approach with rigorous experiments using KPI instrumentation and entropy-based ambiguity detection, achieving 100% ground truth match on well-structured policies.
• The study demonstrates practical applications in scalable healthcare policy digitization while addressing limitations in BPMN construct diversity and ambiguity resolution.

Automated Generation of Executable BPMN Models from Medical Guidelines

Introduction and Motivation

The paper "Automatic Generation of Executable BPMN Models from Medical Guidelines" (2604.07817) proposes an end-to-end pipeline that automates the transformation of medical guideline documents into executable, data-aware BPMN models suitable for simulation-based policy evaluation. Healthcare organizations rely on complex written policies for operational coordination and compliance, but manual translation of these documents into process models is inefficient and error-prone. The approach leverages LLMs to address the bottleneck of automating policy digitization and introducing execution semantics, KPI instrumentation, and uncertainty quantification via entropy-based ambiguity detection.

System Architecture

The pipeline consists of six sequential stages, beginning with PDF document processing, translation, and formal narrative generation grounded in database schema. Subsequent stages synthesize structurally validated BPMN models, augment them for direct execution in SpiffWorkflow, and handle KPI instrumentation to enable simulation-based evaluation.

Figure 1: System Architecture for the automated generation and simulation of executable BPMN models from medical guidelines.

Document processing utilizes PyMuPDF4LLM to extract structured text and tables, including robust artifact repair. The translation step preserves logical constructs and domain-specific terminology, ensuring semantic fidelity. In narrative generation, the pipeline employs multi-phase prompting to translate policy prose into Boolean expressions over database columns—resolving criterion tokenization, operator precedence, and schema mapping with ranking heuristics to reduce semantically spurious column binding.

The BPMN synthesis module enforces strict compliance with eight structural validation rules, applying an iterative repair loop to correct split/merge gateway anomalies, default flow assignments, XML entity encoding, and task/gateway cardinality. Executable augmentation translates BPMN user tasks to script/service tasks in Python, ensures gateway condition syntax compatibility, and binds variables directly to the pandas-backed patient data context for runtime access.

KPI instrumentation employs LLM-driven mapping (majority vote across stochastic responses) to associate five KPIs with relevant model tasks, supporting simulations over synthetic patient populations. The uncertainty detector calculates normalized Shannon entropy over KPI outcome distributions across multiple generated models, flagging documents where ambiguous policy language yields multi-modal simulation results.

Experimental Evaluation

The evaluation uses diabetic nephropathy guidance policies from three Japanese municipalities of varying structural complexity. Three LLM backends (Gemini 2.5 Pro, Gemini 2.5 Flash, GPT-5.1) generate 100 models per city, each executed against 1,000 synthetic patient records spanning the relevant decision thresholds.

For well-structured policies (City 1), Gemini Pro models achieve 100% agreement with ground truth KPIs, with zero entropy across all models. Flash and GPT-5.1 exhibit ground-truth match rates of 86-87%, with distinct failure modes—Flash presents generation failures due to unrepairable BPMN violations, while GPT-5.1 shows variable-binding errors.

Compound Boolean eligibility logic (City 2) introduces increased variability, as evidenced by entropy rising to 51–69% and KPI fragmentation driven by compound gateway interpretation. For implicit temporal dependencies (City 3), all backends show high entropy and near-uniform disagreement, confirming that ambiguity is intrinsic to document structure rather than model shortcomings.

The per-patient decision agreement rates further corroborate functional equivalence: City 1 yields Cohen’s $\kappa \geq 0.918$ (perfect for Gemini Pro); City 2 moderates at $\kappa = 0.850$–$0.931$; City 3 drops to $\kappa \approx 0$ despite high raw agreement, exposing the base-rate paradox where rare eligibility classes mask discriminative failure.

Strong Numerical Results and Analytical Insights

• On well-structured policies, the pipeline achieves 100% ground-truth match and perfect per-patient decision agreement for Gemini 2.5 Pro.
• Across all conditions, raw per-patient agreement exceeds 92%, but Cohen’s $\kappa$ drops sharply in the presence of document ambiguity.
• Normalized entropy scores increase monotonically with policy complexity: 0%–43.5% (City 1), 51.4%–69.1% (City 2), 64.8%–99.6% (City 3), robustly separating unambiguous from ambiguous documents.
• Generation failures and variable-binding errors are reproducible across LLMs, enabling cross-LLM consensus voting to mitigate stochastic noise and semantic mismatches.

Practical and Theoretical Implications

Practically, the pipeline enables large-scale, simulation-based policy analysis for healthcare process optimization. It allows municipalities to digitize and evaluate numerous guidelines with minimal manual intervention, supporting rapid scenario modeling and resource allocation.

Theoretically, the approach establishes a functional equivalence paradigm in the evaluation of generated BPMN models—moving beyond prior works focused on structural similarity or subjective user assessment. The entropy-based ambiguity detector provides a rigorous mechanism for quantifying uncertainty arising from policy language, offering targeted human-in-the-loop correction and future improvements via semi-automated clarification.

The system’s modular LLM-driven architecture and programmatic repair loops set a precedent for integrating generative AI in domain-constrained model synthesis, combining prompt engineering with explicit validation rules and majority-vote schemes to enforce reliability.

Limitations and Future Prospects

Current limitations include reliance on a restricted BPMN construct palette—excluding parallel gateways, timers, and sub-processes—and evaluation against only three test cases. The approach is domain-specific but amenable to extension; future developments aim to incorporate more complex BPMN semantics, human-in-the-loop ambiguity resolution, and generalization to non-healthcare domains. The entropy-centric evaluation framework could serve as a foundation for broader AI-based program synthesis tasks requiring ambiguity quantification and consensus refinement.

Conclusion

The proposed pipeline automates the extraction, synthesis, and execution of BPMN models from medical guidelines, achieving high fidelity in functional decision logic for unambiguous policies and robust entropy-driven detection of ambiguity. KPI instrumentation and simulation enable quantitative policy evaluation at scale, moving the field closer to fully automated, trustworthy process modeling in healthcare and beyond. The cross-LLM majority-vote strategy and entropy analysis underpin practical model selection and human oversight, and set the stage for future advances in data-aware executable process generation.