Pumbaa: FM-Based Workflow Repair Tool
- Pumbaa is a repair tool designed to fix nine specific defect types in FM-generated DSL workflows, using structured feedback from Timon.
- It functions as a post-synthesis reliability layer within a natural language-to-workflow pipeline, applying defect-specific prompts for iterative repair.
- Empirical evaluation shows that integrating Timon’s static analysis improves repair rates by over 6× compared to FM-only self-repair approaches.
Searching arXiv for the cited paper and closely related workflow-repair work to ground the article. {"queries":[{"query":"(Masoumzadeh et al., 29 Sep 2025) Towards Reliable Generation of Executable Workflows by Foundation Models"},{"query":"foundation model generated DSL workflows static analysis repair arXiv workflow synthesis defects"}]} I found the target paper on arXiv and verified the surrounding context is consistent with the provided data. I’ll use the supplied details as the primary factual basis and cite the paper directly as (Masoumzadeh et al., 29 Sep 2025). Pumbaa is an FM-based repair tool designed to automatically fix defects in DSL workflows that are generated by foundation models (FMs). It operates inside a pipeline that goes from natural-language requirements to executable workflows, and it is tightly coupled with Timon, a bespoke static analyzer that detects nine categories of defects without executing the workflow. In the framework presented in "Towards Reliable Generation of Executable Workflows by Foundation Models" (Masoumzadeh et al., 29 Sep 2025), Pumbaa addresses a practical reliability gap: despite careful prompting and FM-friendly grammar, synthesized WorkFlow DSL programs often contain syntactic and semantic defects, and Pumbaa uses Timon’s structured defect reports to patch those defects and return an updated workflow for re-analysis.
1. Position within the natural-language-to-workflow pipeline
The pipeline in which Pumbaa operates has two main stages. The first is Requirements Refinement, using AlignMind to obtain step-by-step instructions from vague user scenarios. The second is DSL Workflow Synthesis, using a Converter Agent and a Skill Chooser to produce a JSON workflow in the WorkFlow DSL. Pumbaa is not the synthesis component itself; it is the repair engine that intervenes after synthesis when the resulting workflow is defective (Masoumzadeh et al., 29 Sep 2025).
Its placement is operationally specific. Natural-language inputs are transformed into FM-generated DSL workflows, Timon statically analyzes those workflows, Pumbaa repairs the reported defects, and Timon then re-checks the revised output. The resulting loop is: natural-language inputs → FM-generated DSL workflow → Timon detection → Pumbaa repair → Timon re-check → executable workflow. This organization makes Pumbaa a post-synthesis reliability layer rather than a general workflow generator.
A plausible implication is that the design separates generation from validation and repair in order to reduce coupling between prompt-based synthesis and workflow correctness. The paper’s formulation suggests that this separation is central to making FM-produced workflows executable in practice.
2. Defect model and repair scope
The paper presents the first taxonomy of defects observed in FM-generated DSL workflows, categorizing them into 18 distinct types grouped into three categories: data errors, logical errors, and mismatch between instructions and workflows (Masoumzadeh et al., 29 Sep 2025). Within data errors, the taxonomy includes Incorrect Inputs in Workflows, Incorrect Outputs in Workflows, and Violation of Type Consistency Constraints. Within logical errors, it includes Violation of Reachability Constraints, Omission of Logic, Inclusion of Incorrect Logic, and Violation of Concurrency Constraints. Within mismatch between instructions and workflows, it includes Incorrect Control Flow Construct, Over-engineering, Defective Structure, and Incorrect Skill Usage, with four subtypes: incorrect action, suboptimal action, non-existing action, and defective parameters.
Pumbaa does not target all 18 categories directly. Its repair scope is aligned to the nine defect types that Timon can detect through static analysis. Those nine are Unparsable JSON, Invalid DSL, Unreachable variables, Unused variables, Incorrect data type propagation, Hallucinated skills, Skills with defective parameters, Malformed expressions, and Incorrect outputs. Timon detects these categories, and Pumbaa is designed to repair them by leveraging Timon’s feedback.
The defect prevalence reported in the study is high: 87.27% of the studied instances contained at least one defect. The paper also identifies “skills with defective parameters” as the most frequent defect category, with 516 incidences. This suggests that interface conformance between tasks and skill schemas is a dominant failure mode in FM-generated DSL workflows.
| Detectable by Timon and targeted by Pumbaa | Description from the framework |
|---|---|
| Unparsable JSON | JSON cannot be parsed |
| Invalid DSL | Grammar violations checked via Pydantic type checks |
| Unreachable variables | Use before def across control-flow |
| Unused variables | Def never used downstream |
| Incorrect data type propagation | Mismatched primitive types |
| Hallucinated skills | Task action not in skills database |
| Skills with defective parameters | Wrong/missing input/output fields relative to skill schema |
| Malformed expressions | Conditions/cases that fail to parse to an AST |
| Incorrect outputs | Violations of the requested output formatting rule |
The repair scope is therefore narrower than the full taxonomy. Omission of Logic, Inclusion of Incorrect Logic, and Over-engineering, for example, are part of the taxonomy but are not listed among the nine statically detectable categories. A plausible implication is that these defects remain harder to validate without stronger semantic grounding in the original instructions or complementary dynamic analysis.
3. Architecture, interfaces, and repair loop
Pumbaa takes as input a defective workflow together with Timon’s defect report, and it outputs an updated workflow JSON that aims to fix all reported incidences of one defect type at a time (Masoumzadeh et al., 29 Sep 2025). Timon, in turn, takes an FM-generated workflow as a JSON string and outputs a structured defect report, with locations, or stops early if the JSON or DSL is invalid. The interaction is therefore explicitly report-driven: Timon identifies and localizes defects, and Pumbaa patches them under defect-type-specific prompting.
The internal repair method is based on tailored prompt templates crafted for each defect category. For Invalid DSL, the prompt includes the WorkFlow TypeScript type definition and grammar. For Incorrect outputs, it includes the precise output-field format rule supplied at synthesis time. For Hallucinated skills and Skills with defective parameters, it includes a list of available skills, with input/output schemas and descriptions, as in-context references. The authors state that prompts were iteratively refined using grammar/spell checking, few-shot exemplars, and best-practice guidance from OpenAI and Anthropic.
Pumbaa uses a test-time iterative strategy. For each defect type, it attempts up to repairs in a pass@k setting. After each repair, Timon re-analyzes the updated workflow to verify whether defect instances have been resolved and to update the report. Pumbaa treats each defect type independently to avoid confounding effects and reuses the original defective input for each category. This procedural choice is important because it constrains cross-category interference: a repair attempt for one category is not applied cumulatively as the starting point for another category.
The framework also specifies how DSL semantics and constraints are handled. Syntax and grammar adherence are checked by Pydantic-based DSL instantiation. Control-flow and data-flow constraints are enforced via CFG, use-def, and def-use analyses. Skill interfaces are validated by matching task actions to the skills database and checking parameter bindings. Execution preconditions are reflected in expression parsing and variable reachability checks.
4. Formal basis and illustrative repair behavior
Timon constructs a control-flow graph of the workflow and then builds use-def and def-use chains. Let denote the CFG nodes and edges extracted from the WorkFlow JSON; Timon’s dataflow checks are performed over this graph to find unreachables and unuseds and to validate type propagation. Expressions in conditions and switch branches are parsed into ASTs, and variable references found therein are integrated into use-def and def-use chains (Masoumzadeh et al., 29 Sep 2025).
The paper provides a simplified WorkFlow DSL example involving possibly unreachable variables. In the defective version, one conditional defines either or , while a subsequent conditional may use in one branch and in the other under a different predicate. Timon flags these cases because the relevant variables may be undefined on certain paths. The paper’s repaired version aligns the second conditional with the first branch predicate so that each branch uses the variable defined on the corresponding first-branch path. The stated effect is that variable reachability becomes consistent in both branches.
That example is significant because it clarifies the kind of repair Pumbaa can induce. The prompt includes Timon’s report identifying which variables are undefined on specific paths and instructs the model to rewire branch conditions or hoist definitions to ensure reachability. This is not merely syntactic correction; it is a constrained control-flow repair informed by static dataflow analysis.
A second example concerns skills with defective parameters. A defective task invoking Email.Send provides only recipient, while the skill schema requires recipient, subject, and body. Timon flags the defect as “skills with defective parameters,” and Pumbaa’s prompt includes the relevant skills database entry so that the FM can complete the required parameters correctly. This example illustrates that Pumbaa’s operation depends not only on DSL grammar but also on externally supplied skill schemas and descriptions.
5. Empirical performance
The experimental dataset consists of 55 open-coded FM-generated WorkFlow DSL programs derived from 200 synthetic user scenarios refined via AlignMind. For detection evaluation, 40 workflows are randomly selected. For repair evaluation, Pumbaa runs with three FMs—GPT‑4o, LLaMA, and Qwen—each with temperature 0.7 and standard context limits, and per-defect pass@10 is used to measure the number of defect incidences fixed within at most 10 repair attempts (Masoumzadeh et al., 29 Sep 2025).
Timon’s reported detection accuracy varies by defect type. Precision and recall are 100% / 100% for Unparsable JSON, Invalid DSL, Hallucinated skills, Skills with defective parameters, Malformed expressions, and Incorrect outputs. For Unreachable variables the reported figures are 50% / 71%; for Unused variables, 75% / 100%; and for Incorrect data type propagation, 37% / 72%. Overall, Timon achieves 56% precision and 80% recall.
Pumbaa’s repair effectiveness is reported as pass@10 counts across all nine defect types. GPT‑4o repaired 286/990 incidences (28.89%), LLaMA 71/990 (7.17%), and Qwen 41/990 (4.14%). The per-defect highlights are uneven. Incorrect outputs were repaired perfectly, at 31/31 (100%), by all three FMs. Hallucinated skills were also repaired perfectly, at 3/3 (100%), by all three. Skills with defective parameters were repaired at 160/516 (31.01%) by GPT‑4o and 0 by LLaMA and Qwen. Unused variables were repaired at 60/128 (46.88%) by GPT‑4o, 16/128 (12.50%) by LLaMA, and 0 by Qwen. Unreachable variables were repaired at 20/254 (7.87%) by GPT‑4o, 21/254 (8.27%) by LLaMA, and 7/254 (2.76%) by Qwen. Incorrect data type propagation was repaired at 12/55 (21.82%) by GPT‑4o and 0 by LLaMA and Qwen. Invalid DSL had 0/3 repaired across models; those cases were marked incomprehensible if parsing or instantiation remained impossible.
The comparative result against an FM-only baseline is one of the paper’s central empirical findings. A baseline self-repair setting without Timon feedback fixed 45/990 defects (4.55%). With Timon feedback, GPT‑4o improved to 28.89%, which the paper describes as over 6× more fixes. The iterative pass@k analysis further shows that GPT‑4o with feedback saturates at : pass@1 is 27.07%, pass@2 is 28.08%, pass@3 is 28.59%, pass@4 is 28.59%, pass@5 is 28.69%, and pass@6 is 28.89%. About 10% of workflows in the evaluation set became fully repaired.
6. Limitations, operational guidance, and relation to prior repair systems
The paper states clear limitations for both detection and repair (Masoumzadeh et al., 29 Sep 2025). For detection, some reachability and type-propagation false positives and false negatives arise because feasibility depends on runtime values and complex control flow, including nested branches and loops. Solely static analysis cannot guarantee path feasibility. The paper recommends complementing Timon with dynamic analyses or SMT solving, for example Z3, to reason about contradictory conditions and bound satisfiability.
For repair, effectiveness varies strongly by defect type and model. Open-source dense models, specifically LLaMA and Qwen, were substantially worse than GPT‑4o for several defects, especially parameter repairs and type propagation. Even with GPT‑4o, not all defects are fixed; when Pumbaa fails, the system should fall back to human review with a precise defect report from Timon. The paper also notes prompt sensitivity and hallucinations: automated prompt optimizers had mixed impacts, ranging from −38.33% to +55% in specific categories. Repair effectiveness further depends on the availability and quality of the skills database and the explicit DSL grammar, and low-resource or novel DSLs remain challenging for FMs.
The practical guidance is correspondingly procedural. Inputs should be gathered as user scenarios and refined with a requirement-refiner such as AlignMind to produce step-wise instructions. Workflow synthesis should use the Converter Agent with Skill Chooser, preferably semantic similarity mode, to generate a WorkFlow JSON. Timon should then be run to detect defects, with JSON and grammar validity repaired first if necessary. Pumbaa should be invoked per defect category with Timon’s report and the original defective workflow, with pass@k up to 10 and re-analysis after each attempt. The paper advises increasing while improvements are observed, noting that saturation often occurs by , and falling back to human correction if defects remain. It also specifies best practices: provide DSL grammar during syntax and structure repairs, supply the skills catalog with schemas and natural-language descriptions during skill repairs, keep temperature moderate at 0.7, include few-shot examples per defect category, treat defect categories independently, and constrain context, for example with semantic similarity, to reduce irrelevant distractions during repair.
In relation to prior LLM-based repair systems, the paper distinguishes Pumbaa from Automated Program Repair approaches that focus on general-purpose languages and often rely on test suites, bug localization, or repository-level repair. Pumbaa instead targets FM-generated DSL workflows and leverages static analysis feedback tailored to DSL syntax and semantics, CFG-based dataflow, and skills schemas. Its repair loop is defect-type-specific and grounded in Timon’s precise, actionable reports rather than generic unit tests. The paper’s stated contributions are the first taxonomy of 18 defect types in FM-generated DSL workflows, Timon as a static analyzer purpose-built for nine of those types, Pumbaa as an FM-based repair tool that improves substantially over FM-only self-repair, and an end-to-end, replicable pipeline from natural-language requirements to executable workflows. This suggests that Pumbaa is best understood not as an isolated repair heuristic but as the repair component of a broader reliability architecture for FM-powered workflow generation.
