Chat2Workflow: Benchmark for Workflow Synthesis

• Chat2Workflow is a benchmark for generating executable visual workflows from natural language, shifting from textual synthesis to platform-constrained graph construction.
• It leverages real-world workflows and multi-turn dialogue revisions to test whether LLMs can produce structurally valid, variable-consistent, and runnable workflow artifacts.
• The evaluation protocol uses execution-aware metrics that reveal gaps between syntactic correctness and actual task success, highlighting challenges in industrial deployment.

Searching arXiv for Chat2Workflow and closely related workflow-grounded dialogue/workflow extraction papers. arXiv search results identified the benchmark paper "Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language" (Zhong et al., 21 Apr 2026) and related workflow-grounded dialogue and workflow-extraction papers including (Min et al., 2023, Ramakrishnan et al., 2023, Burdisso et al., 2024), and (Choubey et al., 24 Feb 2025). Chat2Workflow is a benchmark and problem setting for generating executable visual workflows directly from natural language, with an explicit emphasis on workflows that can be transformed and deployed on practical platforms such as Dify and Coze (Zhong et al., 21 Apr 2026). It studies whether LLMs can automate a development process that, in current industrial practice, is still largely manual: developers must design workflow graphs, write prompts for individual nodes, connect control flow, and repeatedly revise the logic as requirements change. The benchmark therefore targets not merely plan description or code-like synthesis, but the production of graph-structured, variable-consistent, runnable workflow artifacts under multi-round instruction updates (Zhong et al., 21 Apr 2026).

1. Definition and scope

Chat2Workflow is introduced in the context of real-world industrial deployments where executable visual workflows are treated as a mainstream paradigm because they offer reliability and controllability (Zhong et al., 21 Apr 2026). The motivating claim is that current workflow development is costly, time-consuming, and error-prone because it depends on manual engineering. The benchmark asks whether an LLM can replace part of that process by taking natural-language requirements and producing a workflow that is not only plausible at the level of intent, but also structurally valid and executable on a target platform.

A central feature of the topic is its distinction between workflow-aware dialogue and workflow synthesis. In adjacent task-oriented dialogue research, the workflow is typically already given and the problem is to generate responses that comply with it, or to predict future actions within it (Min et al., 2023, Ramakrishnan et al., 2023). Chat2Workflow instead makes the workflow itself the output artifact. This suggests a shift from policy-conditioned language generation toward platform-constrained graph construction.

The topic also differs from dialog-workflow extraction work that reconstructs service procedures from historical conversations. Those methods infer workflows from existing interactions; Chat2Workflow evaluates the forward direction, in which a model must generate a new executable workflow from requirements expressed in natural language (Choubey et al., 24 Feb 2025, Burdisso et al., 2024).

2. Corpus construction and benchmark design

Chat2Workflow is built from real-world production workflows collected from official sources and GitHub, specifically from Dify and Coze configurations (Zhong et al., 21 Apr 2026). The authors reverse-engineered those workflows into task instructions, clustered workflows from the same task context, and rewrote them into multi-turn dialogue rounds in which later rounds modify, add, or refine earlier requirements. The benchmark is therefore designed to test not only one-shot workflow generation but also long-horizon revision under evolving constraints.

Benchmark element	Value
Tasks	27
Multi-turn instructions	79
Test instances	237
Test cases per instruction	3
Domains	Research, Document, Enterprise, Developer, Education, AIGC
Curated node types	20 most frequently used node types

The benchmark’s deployability depends on an explicit curation of the workflow substrate. The authors selected the 20 most frequently used node types, simplified node input/output interfaces so that the model only needs to fill essential variables, and built node knowledge bases from official documentation (Zhong et al., 21 Apr 2026). The generated workflow is emitted as JSON and then converted through a rule-based pipeline into platform-native YAML for import. Because Dify workflows are natively YAML, this transformation is a necessary bridge from language-model output to executable artifact (Zhong et al., 21 Apr 2026).

This construction gives Chat2Workflow an unusually operational character relative to many NLP benchmarks. The instances are not synthetic toy graphs; they are derived from workflows already used in practice, and each instruction is paired with executable test cases (Zhong et al., 21 Apr 2026).

3. Formal task and representation

The task is formalized as a multi-turn workflow generation problem. Given a current-round instruction $q$, historical dialogue $H$, and a LLM $\mathcal{M}_\theta$, the model must output a graph-structured workflow

$$\mathcal{G}(\mathcal{V},\mathcal{E}) \leftarrow \mathcal{M}_\theta(q, H),$$

where $\mathcal{G}$ is a DAG, $\mathcal{V}=\{v_1,\dots,v_{|\mathcal{V}|}\}$ is the node set, and $\mathcal{E}=\{(v_i,p,v_j)\}$ are directed edges connecting nodes through output branch port $p\in \mathbb{Z}_{\ge 0}$ (Zhong et al., 21 Apr 2026). This definition makes the output object explicitly graph-theoretic rather than textual or action-sequential.

Because direct generation of valid executable workflows is difficult, the paper adopts a structured CoT-style output with three tagged components: <node_selection>, <design_principle>, and <workflow> (Zhong et al., 21 Apr 2026). The model first identifies selected nodes, then provides a planning rationale, and finally emits workflow JSON. The workflow JSON is the executable core; it is subsequently transformed into YAML and imported into the target platform.

This representation imposes several simultaneous constraints. The output must be structurally valid as a graph, internally consistent with the declared node set, variable-consistent with node interfaces, and executable after platform conversion (Zhong et al., 21 Apr 2026). In contrast, workflow-guided response generation in task-oriented dialogue optimizes whether an utterance or multi-turn block complies with a specified workflow action, not whether a graph artifact can be imported and run (Min et al., 2023). Likewise, multi-step workflow action prediction models future action sequences or action graphs, but not deployable visual workflows with node-level platform semantics (Ramakrishnan et al., 2023).

4. Evaluation protocol and empirical performance

Chat2Workflow uses an execution-aware two-stage evaluation protocol (Zhong et al., 21 Apr 2026). The first stage checks whether the workflow representation is legally formed and convertible to platform format; the second checks whether the imported workflow actually runs and satisfies the instruction semantics on test cases. These stages correspond to Pass Rate and Resolve Rate. Pass Rate evaluates whether the generated JSON is consistent with the CoT sequence, matches the reference variables, and can be converted into valid YAML and imported. Resolve Rate evaluates whether the executed workflow’s output fulfills the instruction requirements (Zhong et al., 21 Apr 2026).

The implementation uses DeepSeek-V3 for evaluation and reports 100% agreement with human judgment for the pass stage and 98.83% agreement for the resolve stage, based on independent sampling of 500 pass-stage samples and 1,282 resolve-stage samples (Zhong et al., 21 Apr 2026). This matters because the benchmark relies on automated execution-based judgment rather than purely string-level comparison.

The empirical study covers 15 models, comprising 4 closed-source and 11 open-source systems (Zhong et al., 21 Apr 2026). Among closed models, Gemini-3-Pro-Preview achieves the best average performance with 80.17% Pass Rate and 71.59% Resolve Rate. Among open models, GLM-4.7 is strongest on average with 69.20% Pass Rate and 55.98% Resolve Rate (Zhong et al., 21 Apr 2026).

A defining empirical pattern is the persistent gap between syntactic or representational correctness and actual task success. Resolve Rate is always below Pass Rate, with the largest average discrepancy reaching 20.96% for GLM-4.6, and a task-specific discrepancy of 43.44% in the Education scenario for the same model (Zhong et al., 21 Apr 2026). The benchmark therefore operationalizes a distinction that is often blurred in generation tasks: a workflow may parse, convert, and import successfully while still failing to execute correctly or satisfy the requirement semantics.

5. Error modes, multi-round degradation, and robustness strategies

The paper identifies three concrete failure modes through a case study on the StudyPlanner task (Zhong et al., 21 Apr 2026). First, models can produce invalid edge connections, such as links that violate the internal structure of special nodes like iteration blocks. Second, they can produce logical inconsistency, for example by mentioning nodes in the workflow that were not declared in <node_selection>. Third, they can generate workflows with executable but incorrect behavior: the artifact imports successfully, yet still fails to solve the task.

Performance also degrades across dialogue rounds. Each task contains 2–4 rounds, and later rounds modify earlier requirements; for most models, both Pass Rate and Resolve Rate decline steadily over the first three rounds (Zhong et al., 21 Apr 2026). This indicates difficulty in preserving coherent graph semantics under iterative revision, rather than only difficulty in initial graph generation.

To mitigate recurrent execution errors, the paper proposes an error-driven agentic baseline built with the OpenCode framework (Zhong et al., 21 Apr 2026). This baseline moves from zero-shot prompting to a SKILL-based paradigm, explicitly encoding task-critical guidelines, multi-turn rules, and variable references. It also extracts Variable Summaries from prior iterations to reduce context decay in later turns. The execution loop permits up to 5 retry attempts and applies targeted auto-repair for the most frequent structural and semantic failures: malformed code fences, JSON decoding failures, topological sorting violations, and node selection inconsistencies (Zhong et al., 21 Apr 2026).

The improvement is measurable but limited. GPT-5.2 improves from 54.71% zero-shot Resolve Rate to 60.05% agentic Resolve Rate, an absolute gain of 5.34%. GPT-5.1 improves from 39.38% to 44.31%, a gain of 4.93% (Zhong et al., 21 Apr 2026). This suggests that execution feedback and targeted repair are useful, but they do not close the remaining gap to industrial-grade reliability.

6. Relation to adjacent research traditions

Chat2Workflow occupies one part of a broader research space in which conversation and workflow structure are coupled but not in the same way. In workflow-guided response generation, the workflow is a known policy and the objective is to generate responses that comply with it. The COMPLIANCEOPT framework combines a learned ComplianceScorer with RL optimization and reports strong compliance gains on ABCD and MultiWOZ 2.2, including 0.8670 compliance on ABCD for the oracle variant, close to the ground-truth score of 0.8676 (Min et al., 2023). In multi-step dialogue workflow action prediction, the output is a predicted future action sequence or action graph rather than executable workflow JSON; that setting reports 20% more automated workflow steps than 1-step prediction (Ramakrishnan et al., 2023). These lines of work treat workflow as latent or given policy state for response planning, not as a platform-ready visual artifact.

A second adjacent tradition focuses on extracting workflows from conversations. The framework in "Turning Conversations into Workflows" uses procedural-element retrieval and QA-CoT prompting to recover service workflows from historical interactions, reporting 12.16% average macro accuracy improvement over the baseline and an end-to-end evaluation aligned with human assessments (Choubey et al., 24 Feb 2025). "Dialog2Flow" learns action-driven sentence embeddings from a unified corpus of 20 task-oriented dialog datasets, 3.4 million utterances, and 52 domains, then clusters utterances into workflow states; its induced graphs reach an average node-count difference of 6.86% from the gold workflow graph for the strongest configuration (Burdisso et al., 2024). These works infer workflows from existing dialogs, whereas Chat2Workflow tests direct generation of new executable workflows from requirements.

A third tradition consists of operational conversational workflow systems. TaskFlow models task-oriented dialogue as a tree with dialogue actions as nodes and transitions as edges, and in real after-sale service scenarios reduces estimated human effort from 21 to 9 person-days while producing high rates of correctly handled dialogues (Xi et al., 2022). Low-code architectures connect conversational frontends to execution backends, such as the Rasa + Node-RED design in which Node-RED acts as a low-code fulfillment middleware implementing the action server HTTP API (Weber, 2024). DMN-based decision-support chatbots semi-automatically derive conversational decision workflows from existing decision tables and are designed to ask only necessary questions (Estrada-Torres et al., 2021). GUI-inspired prompt designs make submit-like acknowledgment and reset-like context switching explicit state-transition tasks in multi-turn chatbots (Anbalagan et al., 2 Jul 2025). Domain-specific chat-driven workflow agents, such as ChatCFD for OpenFOAM pipelines and Chat-of-Thought for FMEA generation, further show that chat can serve as a control layer for structured engineering processes rather than only as a response interface (Fan et al., 28 May 2025, Constantinides et al., 11 Jun 2025).

Taken together, these neighboring lines of work imply a broader taxonomy. One regime optimizes workflow compliance in dialogue, another performs workflow extraction from historical interactions, and Chat2Workflow targets workflow synthesis: the direct production of executable, visual, platform-constrained workflow graphs from natural language requirements (Zhong et al., 21 Apr 2026). Within that taxonomy, Chat2Workflow is notable for making execution, importability, multi-round revision, and industrial deployability first-class evaluation targets rather than secondary implementation details.