Ambiguity Detection and Elimination in Automated Executable Process Modeling

Abstract: Automated generation of executable Business Process Model and Notation (BPMN) models from natural-language specifications is increasingly enabled by LLMs. However, ambiguous or underspecified text can yield structurally valid models with different simulated behavior. Our goal is not to prove that one generated BPMN model is semantically correct, but to detect when a natural-language specification fails to support a stable executable interpretation under repeated generation and simulation. We present a diagnosis-driven framework that detects behavioral inconsistency from the empirical distribution of key performance indicators (KPIs), localizes divergence to gateway logic using model-based diagnosis, maps that logic back to verbatim narrative segments, and repairs the source text through evidence-based refinement. Experiments on diabetic nephropathy health-guidance policies show that the method reduces variability in regenerated model behavior. The result is a closed-loop approach for validating and repairing executable process specifications in the absence of ground-truth BPMN models.

• The paper introduces a simulation-driven framework that quantifies ambiguity using normalized Shannon entropy and divergent KPI simulation outputs.
• It employs model-based diagnosis to localize ambiguous decision gateways by identifying minimal diagnosis sets that correlate with behavioral differences.
• Empirical evaluations on healthcare policies demonstrate over 90% model consistency post-repair, validating the targeted ambiguity elimination approach.

Diagnosis-Driven Behavioral Ambiguity Detection and Repair for LLM-Generated Executable Process Models

Problem Definition and Motivation

Automated translation of natural-language policy documents and clinical guidelines into executable BPMN models via LLMs is increasingly central for domains such as healthcare, where simulation and quantitative analysis of interventions are critical. However, free-form language specifications readily admit ambiguity and underspecification, leading LLMs to generate multiple structurally valid but semantically divergent BPMN instantiations from the same input. Structural soundness guarantees are insufficient: consistent and reliable downstream simulation requires that the source text admits a unique (or at least behaviorally stable) executable interpretation. This work formalizes ambiguity from the perspective of simulation output distributions—defining actionable ambiguity as source text that, under repeated LLM-based BPMN generation, yields models with significantly divergent KPI outcomes. The framework prioritizes behavioral evidence as the ambiguity trigger, rather than solely linguistic features.

Simulation-Driven Ambiguity Detection and Entropy as Consistency Proxy

The core metric for ambiguity is the normalized Shannon entropy of the empirical KPI output distribution resulting from simulating $N$ BPMN models generated from the same source text and instantiated on identical synthetic input populations. Given sparse, high-dimensional output vectors, the approach aggregates outcomes and defines normalized entropy $H_{\text{norm}}$ as a scale-independent measure of generation consistency. Empirically, high entropy correlates with unstable model interpretation. The proposed method categorizes output stability into four qualitative consistency bands using this entropy, enabling efficient flagging of ambiguous specifications.

Figure 1: The empirical KPI distribution for City 1 under 100 distinct BPMN generations reveals the presence of behavioral ambiguity in the original process description.

Gateway-Level Diagnostic Localization via Model-Based Diagnosis

Ambiguity localization proceeds via behavioral comparison and model-based diagnosis (MBD). The method selects representative models from dominant KPI classes (clusters) in the empirical distribution and pinpoints decision nodes (gateways) responsible for output divergence. The diagnosis is strictly behavioral: for each pair of divergent traces, the workflow identifies the earliest divergence in KPI outputs, localizes the corresponding tasks, and computes conflict sets comprising intermediary gateways. Minimal hitting sets over these conflicts yield minimal diagnosis sets (MDSs)—the minimal sets of gateways whose logical difference is causally responsible for simulation divergence.

Figure 2: The target BPMN model for City 1 with the minimal diagnosis gateways highlighted, indicating the specific decision points under evaluation.

Figure 3: The minimum diagnosis set for City 1, summarizing the causal inference mapping divergent simulation traces to process model components.

A further refinement step prunes the diagnosis by cross-checking for input-conditioned equivalence: gateways with logical conditions identical in reference and target models for relevant input subpopulations are excluded from the MDS. This pathway-based refinement leverages AST-level condition comparison.

Narrative Mapping and Targeted, Evidence-Grounded Textual Repair

Once the MDS points to responsible gateways, the framework maps these components back to their verbatim narrative origins using LLM-assisted codebook alignment. The ambiguous textual spans are identified and paired with structured ambiguity reports detailing both competing interpretations and their behavioral consequences.

Repairs are then executed through a four-step loop: (1) precise localization of implicated text, (2) explicit selection of interpretation informed by authoritative external evidence (e.g., regulatory or programmatic documentation), (3) minimal but explicit rewrite—disambiguating logical operators, dependencies, and qualifying clauses, and (4) narrative reconstruction with traceability metadata linking revisions to evidence. Regeneration and re-simulation on the repaired text act as behavioral validation of repair efficacy.

Figure 4: Post-repair, the KPI distribution for City 1 collapses to a highly concentrated pattern, validating the successful removal of executable ambiguity.

Experimental Evaluation

City 1 Policy

Analysis of the City 1 guideline revealed high entropy in simulation outputs; the top five KPI combinations each accounted for a substantial portion of model behavior, indicating significant ambiguity. Diagnosis isolated two gateways, corresponding to clinical eligibility and health guidance acceptance, as behavioral divergence sources. Textual repair clarified AND/OR logic in selection criteria and explicitly defined participation acceptance, yielding a post-repair reduction in entropy and 90%+ model generation consistency.

Figure 5: The City 2 original policy generates a multimodal KPI distribution, mirroring the findings for City 1 and confirming behavioral ambiguity.

City 2 Policy

Analogously, City 2's more operational policy also produced a multimodal KPI distribution. Diagnosis revealed four primary ambiguous gateways, all of which mapped to textually underspecified eligibility/pathway clauses. Repairs made explicit the logic for albumin testing eligibility, workflow termination on non-eligibility, physician override, and notification subpopulation. The result was high generation consistency post-repair, with approximately 70% of models yielding identical KPIs, and substantial entropy suppression.

Figure 6: Minimum diagnosis set in City 2 identifies key gateways that drive behavioral divergence across model instances.

Figure 7: Repairing the process specification in City 2 sharply increases KPI output concentration, evidencing ambiguity elimination.

Methodological and Practical Implications

A distinguishing methodological innovation is the closure of the behavioral validation loop grounded in simulation evidence. By connecting natural language, executable process model, simulated behavior, and structured ambiguity repair, the framework addresses limitations of purely textual ambiguity detection and LLM-based repair (2604.10884). Unlike text-centric methods [berry2003ambiguity, bashir2025requirementsambiguityllm], ambiguity here is actionable only if it has downstream behavioral effect, as revealed by discrete model simulation divergence. This mechanism is particularly relevant for policy, clinical decision support, and compliance applications, where end-to-end reproducibility and traceability from specification to quantitative evaluation are required.

The framework's dependencies—such as MBD efficiency, the adequacy of the monitored KPIs, and the representativity of the input population—define its detection and repair horizon. Unobservable or non-impactful ambiguities will not be captured. The repair quality further depends on LLM prompt design and the availability/quality of authoritative supporting evidence.

Connections to Prior Work

The approach dovetails with and extends contemporary techniques in ambiguity detection, such as controlled natural language and tool-supported requirements engineering [lami2005quars, fuchs1999attempto, YADAV202185], but inverts the focus: the behavioral artifact, not the source language, validates (dis)ambiguation. It complements BPMN extraction pipelines [kourani2024processmodelingllm, vanDerAa2019text2process] by providing a downstream evaluation and correction loop for ambiguous source materials, enhancing model reliability in policy automation.

Theoretical and Practical Outlook

The closed-loop diagnosis-driven repair paradigm enables scalable ambiguity management in domains where exhaustive ground-truth BPMN annotation is infeasible and where behaviorally stable simulation is required. As LLM-driven process modeling and policy automation proliferate, frameworks such as this will be critical for operationalizing model validation, traceability, and regulatory compliance. Open challenges remain: extending the observable ambiguity set via richer KPI monitoring, integrating human-in-the-loop expert review, and formalizing repair sufficiency criteria using underlying domain ontologies.

Conclusion

This work systematically formulates and addresses the problem of executable ambiguity in LLM-generated BPMN from natural-language specifications by leveraging simulation output divergence as its operational definition. Through behavioral diagnosis, gateway-level localization, and evidence-grounded repair, the framework yields more reliable process models without requiring annotation or semantic supervision. The empirical results on health-policy automation confirm that entropy-driven ambiguity detection, MBD-based localization, and targeted revision can substantially stabilize executable BPMN generation under LLM pipelines. This positions diagnosis-driven repair as a foundational tool for robust automated process modeling in the absence of ground-truth artifacts.