StepFly: Automated TSG Execution

• StepFly is a three-stage framework that formalizes and automates Troubleshooting Guide (TSG) execution by converting unstructured guides into precise DAGs.
• It integrates LLM-driven quality assurance with structured, offline preprocessing to achieve high accuracy in TSG remediation and execution.
• The system leverages workflow-aware, parallel executor scheduling to reduce incident resolution time and enhance operational efficiency.

StepFly is a three-stage, agentic framework for automating Troubleshooting Guide (TSG) execution in large-scale IT incident management. Motivated by empirical findings on the shortcomings of manual and previous LLM-based automation approaches—specifically regarding TSG quality deficiencies, complex control-flow handling, data-intensive operations, and underutilized parallelism—StepFly integrates LLM-driven quality assurance, structured transformation of unstructured guides to formalized representations, and workflow-aware distributed execution. The system achieves a reported success rate of approximately 94% in real-world incident contexts, outperforms established baselines in efficiency metrics, and enables significant wall-clock reductions for parallelizable TSGs (Mao et al., 11 Oct 2025).

1. System Architecture and Workflow

StepFly is architected as an end-to-end automated TSG execution pipeline structured into three sequential stages:

1. TSG Quality Improvement (Offline)

Utilizing an empirical study of 92 real-world TSGs, the authors identified prevalent issues across clarity, control/data flow, database instructions, and document structure. A set of writing guidelines and templates was developed, enforcing explicit step titling, instruction specificity, and explicit branching. The core of this stage is the TSG Mentor tool, an LLM-based assistant that ingests raw TSGs, reformats them to template standards, annotates detected issues inline, and suggests corrections. The Mentor implementation employs few-shot prompting with dynamically selected exemplars via GPT-4.1-mini and demonstrated an F1 of 0.81 on inline-issue detection (recall 0.78, precision 0.85).

2. Offline Preprocessing

Unstructured TSGs are parsed and converted into directed acyclic graphs (DAGs) $G=(V,E)$, where each node $v \in V$ represents an atomic troubleshooting step and each edge $e \in E$ encodes execution dependencies with explicit conditional labels. Enabledness for any node $v$ is formally expressed as:

$$\mathrm{enabled}(v) = \left( \forall e \in \mathrm{in}(v): \mathrm{state}(e) \neq \mathrm{unknown} \right) \;\wedge\; \left( \exists e \in \mathrm{in}(v): \mathrm{state}(e) = \mathrm{enabled} \right)$$

Extraction accuracy over 15 held-out TSGs yielded node-level F1 of $99.42\%$, edge-level F1 of $93.47\%$, and overall F1 of $94.89\%$. Query Preparation Plugins (QPPs) are then synthesized for each log/metric query template, mediating parametrized Kusto (KQL) or similar query compilations. QPPs achieved a 97.3% extraction success—failures were limited to escape character handling.

3. Online Execution

StepFly orchestrates execution via a DAG-guided Scheduler, a pool of identical Executors, structured key–value memory (backed by MongoDB), and a plugin suite including QPPs, log/metric retrieval, DevOps, and code execution. Executors operate in an iterative ReAct-style loop, utilizing chain-of-thought reasoning until plugin calls succeed or configured limits are reached, subsequently writing results (as structured objects) to persistent memory. The Scheduler dynamically enables nodes as dependencies resolve, dispatching executors in parallel whenever $|$\texttt{ReadyQueue}$|>1$ (no extra synchronization; parallelism is determined by DAG width).

2. Formalization of Guide Structure and Execution

StepFly represents each TSG as a formal DAG to enforce strict adherence to prescribed workflows and enable concurrent execution of independent steps. Each node $v \in V$0 corresponds to an atomic or compound TSG instruction, and directed edges $v \in V$1 optionally encode outcome-based conditions (e.g., “Y”/“N”).

The system initializes node and edge states to “unknown,” except for a special Start node (marked enabled). Execution proceeds as an event-driven loop:

1. While the ReadyQueue is nonempty, the Scheduler dispatches executors for enabled nodes.
2. Each executor executes its assigned step with access to full incident context, plugins, and memory.
3. Upon completion, the executor updates outcome-labeled edge states and enables subsequent nodes where all incoming edges are resolved and at least one is enabled:

$v \in V$2

1. Execution completes when the End node is reached or no enabled nodes remain.

Parallelization is opportunistic: the Scheduler will allocate as many concurrent Executors as permitted by the width of the DAG at the point of scheduling, maximizing parallel processing of subgraphs.

3. Technical Contributions and Innovations

StepFly yields several innovations in agentic automation for TSG execution:

• LLM-Guided Quality Remediation: The TSG Mentor automates quality assurance and repair, surpassing prior ad-hoc manual checks. Inline detection and suggestion allow for standardized and precise TSGs, improving downstream automation success.
• Formal Control Flow Extraction: By transforming TSGs into explicit DAGs, StepFly strictly prevents incorrect step reordering or skipping, a limitation in previous CoT and ReAct agentic methods.
• Data-Intensive Query Optimization: QPPs obviate inefficient, error-prone on-the-fly query synthesis. Pre-extracted templates reduce token usage and prevent runtime parsing errors, achieving >97% extraction accuracy.
• Structured, Scalable Memory: All executor outputs are stored as structured MongoDB objects accessible via memory handles, avoiding prompt bloating and enhancing result management for data-intensive troubleshooting.
• Multi-Executor Parallelism: Fine-grained parallel execution is achieved by explicit DAG analysis, with practical speedups up to 70.6%, limited only by the inherent parallelizability of the input DAGs.

4. Empirical Evaluation and Results

StepFly was evaluated on 15 production-critical TSGs (3–28 steps, mean 9) and 80 real-world incidents from large-scale service operations. Multiple LLMs were benchmarked: GPT-4.1, GPT-4.1-mini, GPT-4o, and Grok-3. Comparisons included ReAct (single-agent CoT) and TaskWeaver (multi-agent code-first).

System Performance Metrics

Method	GPT-4.1	GPT-4.1-mini	GPT-4o	Grok-3
ReAct	71.25%	61.25%	66.88%	72.50%
TaskWeaver	80.00%	68.12%	72.50%	80.62%
StepFly–QPP	91.88%	73.75%	81.88%	85.62%
StepFly	94.38%	84.38%	92.50%	88.75%

Median execution time for StepFly (GPT-4.1) was 88% that of ReAct and 38% that of TaskWeaver. Token consumption was 71% of ReAct’s and 61% of TaskWeaver’s.

Parallelization

Of the 15 TSGs, 7 were parallelizable. With 5 executors, mean wall-clock time reduction was 51.31%, with maxima at 70.6% (TSG6: 243.36s → 71.57s) and 70.4% (TSG3: 207.93s → 61.59s). The minimum observed was 32.9% (TSG7). These results indicate substantial efficiency gains in settings with inherent TSG parallelism.

5. Significance, Limitations, and Future Directions

StepFly demonstrates that LLM-based, multi-stage agentic frameworks can systematically address the core limitations of unstructured, manual, or naïve LLM-driven TSG automation. The explicit formalization and modular workflow analysis enable correctness, reliability, and scalability in real-world automated incident response.

A current limitation is the reliance on templates and condition formalization in TSGs, necessitating high-quality initial inputs and accurate DAG extraction. Failures in QPP extraction were confined primarily to character encoding issues. A plausible implication is that further refinements in natural language understanding and template generalization could extend applicability to domains with noisier or less-structured guides.

Potential future research may investigate real-time dynamic TSG adaptation, integration with multimodal monitoring systems, or cross-domain generalization to complex engineering playbooks.

6. Context and Relationship to Prior Work

StepFly’s hybrid agentic approach contrasts with monolithic CoT or ReAct-style single-agent routines, as well as code-centric multi-agent systems like TaskWeaver. The explicit structuring and orchestration of guide steps via DAGs represent an advancement in the reliable automation of procedural IT operations. The empirical results position StepFly as an effective platform in both performance and efficiency dimensions within the automated incident management domain (Mao et al., 11 Oct 2025).