How well can a large language model explain business processes as perceived by users?

Abstract: LLMs are trained on a vast amount of text to interpret and generate human-like textual content. They are becoming a vital vehicle in realizing the vision of the autonomous enterprise, with organizations today actively adopting LLMs to automate many aspects of their operations. LLMs are likely to play a prominent role in future AI-augmented business process management systems, catering functionalities across all system lifecycle stages. One such system's functionality is Situation-Aware eXplainability (SAX), which relates to generating causally sound and human-interpretable explanations. In this paper, we present the SAX4BPM framework developed to generate SAX explanations. The SAX4BPM suite consists of a set of services and a central knowledge repository. The functionality of these services is to elicit the various knowledge ingredients that underlie SAX explanations. A key innovative component among these ingredients is the causal process execution view. In this work, we integrate the framework with an LLM to leverage its power to synthesize the various input ingredients for the sake of improved SAX explanations. Since the use of LLMs for SAX is also accompanied by a certain degree of doubt related to its capacity to adequately fulfill SAX along with its tendency for hallucination and lack of inherent capacity to reason, we pursued a methodological evaluation of the perceived quality of the generated explanations. We developed a designated scale and conducted a rigorous user study. Our findings show that the input presented to the LLMs aided with the guard-railing of its performance, yielding SAX explanations having better-perceived fidelity. This improvement is moderated by the perception of trust and curiosity. More so, this improvement comes at the cost of the perceived interpretability of the explanation.

• The paper presents SAX4BPM, a framework that leverages LLMs and knowledge graphs to automate context-aware explanations in business process management.
• The framework integrates services such as process mining, causal discovery, and feature importance analysis to derive actionable insights from event logs.
• Evaluation across multiple domains shows that incorporating process knowledge enhances fidelity but may compromise interpretability, indicating a nuanced trade-off.

Explainability of Business Processes Using LLMs: An Analysis

In this paper, the authors present a framework called SAX4BPM to automate the generation of Situation-Aware eXplanations (SAX) for business processes using LLMs. This research aims to enhance the explainability of AI-augmented Business Process Management Systems (ABPMS) by integrating LLMs to create human-interpretable explanations based on process context.

SAX4BPM Framework

The SAX4BPM suite is composed of services and a central knowledge repository designed to produce explanations by synthesizing various knowledge constituents of a business process.

• Knowledge Graphs: At the heart of the SAX4BPM is a knowledge graph (KG), implemented as a Labeled Property Graph (LPG) in a Neo4j database. This graph serves as the backbone for storing and querying business process data.

  Figure 1: Knowledge graph schema shows data storage and relationships in SAX4BPM.

• Services Invocation: SAX4BPM provides several services, including Mining4Process for process model discovery, Causal4Process for causal relationship elicitation, ContextEnrichment for adding contextual information, and X4Process for feature importance analysis. These services process event logs to produce JSON inputs for the LLM.

  Figure 2: SAX4BPM services invocation illustrates the integration of SAX components.

Illustrative Example: Parking Fines

An example scenario showcasing the SAX4BPM application involves the management of parking fines. The framework utilizes data generated on processes where fines are issued based on parking violations.

• Mining4Process: Extracts process models showing sequences of tasks, such as verifying handicap permits and determining hazardous parking.

  Figure 3: Process model discovery exposes the procedural workflow for handling parking violations.

• Causal Discovery: Identifies causal links between activities, signifying dependencies that can clarify the sequence of actions influencing fines.

  Figure 4: Causal graph discovery for identifying execution dependencies in the parking fines scenario.

• XAI Feature Importance: Determines the role of various features affecting the decision-making process, e.g., the impact of choosing different towing companies or location regions.

  Figure 5: XAI feature importance highlights factors influencing the processing of parking fines.

Hypotheses and Evaluation

The paper hypothesizes that equipping LLMs with additional knowledge about business processes, such as process and causal execution dependencies, enhances the perceived quality of explanations regarding business process activities.

• Research Methodology: A mixed experimental design was applied across domains (pizza delivery, parking fines, and loan approval), using an online survey to measure perceived explanation quality against several constructs, including fidelity and interpretability.

  Figure 6: Experiment model shows the relationship between input knowledge types and perceived explanation quality.

• Findings: Results indicated that while additional knowledge enhances fidelity, it may compromise interpretability. This trade-off was domain-specific and moderated by factors like user trust and curiosity.

Conclusion

The integration of LLMs in creating explainable business process insights presents both opportunities and challenges. While the introduction of SAX explanations showed promising enhancement in user perception of fidelity, the balance between comprehensibility and informativity remains delicate. This research invites further exploration of knowledge synthesis methods and their impact on automated explainability in business systems. Future avenues suggest further automation of input narratives using dynamic templating and fine-tuning of LLMs for more contextualized outputs.