---
title: 'Chat2Workflow: Benchmark for Workflow Synthesis'
url: https://www.emergentmind.com/topics/chat2workflow
type: topic
---

# Chat2Workflow: Benchmark for Workflow Synthesis

Searching arXiv for Chat2Workflow and closely related workflow-grounded dialogue/workflow extraction papers.
arXiv search results identified the benchmark paper "Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language" [2604.19667] and related workflow-grounded dialogue and workflow-extraction papers including [2311.08300], [2311.09593], [2410.18481], and [2502.17321].
Chat2Workflow is a benchmark and problem setting for generating **executable visual workflows** directly from **natural language**, with an explicit emphasis on workflows that can be transformed and deployed on practical platforms such as **Dify** and **Coze** [2604.19667]. It studies whether large language models can automate a development process that, in current industrial practice, is still largely manual: developers must design workflow graphs, write prompts for individual nodes, connect control flow, and repeatedly revise the logic as requirements change. The benchmark therefore targets not merely plan description or code-like synthesis, but the production of graph-structured, variable-consistent, runnable workflow artifacts under multi-round instruction updates [2604.19667].

## 1. Definition and scope

Chat2Workflow is introduced in the context of **real-world industrial deployments** where executable visual workflows are treated as a mainstream paradigm because they offer **reliability** and **controllability** [2604.19667]. The motivating claim is that current workflow development is costly, time-consuming, and error-prone because it depends on manual engineering. The benchmark asks whether an LLM can replace part of that process by taking natural-language requirements and producing a workflow that is not only plausible at the level of intent, but also structurally valid and executable on a target platform.

A central feature of the topic is its distinction between **workflow-aware dialogue** and **workflow synthesis**. In adjacent task-oriented dialogue research, the workflow is typically already given and the problem is to generate responses that comply with it, or to predict future actions within it [2311.08300] [2311.09593]. Chat2Workflow instead makes the workflow itself the output artifact. This suggests a shift from policy-conditioned language generation toward **platform-constrained graph construction**.

The topic also differs from dialog-workflow extraction work that reconstructs service procedures from historical conversations. Those methods infer workflows from existing interactions; Chat2Workflow evaluates the forward direction, in which a model must generate a new executable workflow from requirements expressed in natural language [2502.17321] [2410.18481].

## 2. Corpus construction and benchmark design

Chat2Workflow is built from **real-world production workflows** collected from official sources and GitHub, specifically from **Dify** and **Coze** configurations [2604.19667]. The authors reverse-engineered those workflows into task instructions, clustered workflows from the same task context, and rewrote them into **multi-turn dialogue rounds** in which later rounds modify, add, or refine earlier requirements. The benchmark is therefore designed to test not only one-shot workflow generation but also long-horizon revision under evolving constraints.

| Benchmark element | Value |
|---|---|
| Tasks | 27 |
| Multi-turn instructions | 79 |
| Test instances | 237 |
| Test cases per instruction | 3 |
| Domains | Research, Document, Enterprise, Developer, Education, AIGC |
| Curated node types | 20 most frequently used node types |

The benchmark’s deployability depends on an explicit curation of the workflow substrate. The authors selected the **20 most frequently used node types**, simplified node input/output interfaces so that the model only needs to fill essential variables, and built node knowledge bases from official documentation [2604.19667]. The generated workflow is emitted as JSON and then converted through a rule-based pipeline into platform-native YAML for import. Because Dify workflows are natively YAML, this transformation is a necessary bridge from language-model output to executable artifact [2604.19667].

This construction gives Chat2Workflow an unusually operational character relative to many NLP benchmarks. The instances are not synthetic toy graphs; they are derived from workflows already used in practice, and each instruction is paired with executable test cases [2604.19667].

## 3. Formal task and representation

The task is formalized as a **multi-turn workflow generation problem**. Given a current-round instruction $q$, historical dialogue $H$, and a language model $\mathcal{M}_\theta$, the model must output a graph-structured workflow
$$
\mathcal{G}(\mathcal{V},\mathcal{E}) \leftarrow \mathcal{M}_\theta(q, H),
$$
where $\mathcal{G}$ is a **DAG**, $\mathcal{V}=\{v_1,\dots,v_{|\mathcal{V}|}\}$ is the node set, and $\mathcal{E}=\{(v_i,p,v_j)\}$ are directed edges connecting nodes through output branch port $p\in \mathbb{Z}_{\ge 0}$ [2604.19667]. This definition makes the output object explicitly graph-theoretic rather than textual or action-sequential.

Because direct generation of valid executable workflows is difficult, the paper adopts a structured CoT-style output with three tagged components: `<node_selection>`, `<design_principle>`, and `<workflow>` [2604.19667]. The model first identifies selected nodes, then provides a planning rationale, and finally emits workflow JSON. The workflow JSON is the executable core; it is subsequently transformed into YAML and imported into the target platform.

This representation imposes several simultaneous constraints. The output must be **structurally valid** as a graph, **internally consistent** with the declared node set, **variable-consistent** with node interfaces, and **executable** after platform conversion [2604.19667]. In contrast, workflow-guided response generation in task-oriented dialogue optimizes whether an utterance or multi-turn block complies with a specified workflow action, not whether a graph artifact can be imported and run [2311.08300]. Likewise, multi-step workflow action prediction models future action sequences or action graphs, but not deployable visual workflows with node-level platform semantics [2311.09593].

## 4. Evaluation protocol and empirical performance

Chat2Workflow uses an **execution-aware two-stage evaluation protocol** [2604.19667]. The first stage checks whether the workflow representation is legally formed and convertible to platform format; the second checks whether the imported workflow actually runs and satisfies the instruction semantics on test cases. These stages correspond to **Pass Rate** and **Resolve Rate**. Pass Rate evaluates whether the generated JSON is consistent with the CoT sequence, matches the reference variables, and can be converted into valid YAML and imported. Resolve Rate evaluates whether the executed workflow’s output fulfills the instruction requirements [2604.19667].

The implementation uses **DeepSeek-V3** for evaluation and reports **100%** agreement with human judgment for the pass stage and **98.83%** agreement for the resolve stage, based on independent sampling of **500** pass-stage samples and **1,282** resolve-stage samples [2604.19667]. This matters because the benchmark relies on automated execution-based judgment rather than purely string-level comparison.

The empirical study covers **15 models**, comprising **4 closed-source** and **11 open-source** systems [2604.19667]. Among closed models, **Gemini-3-Pro-Preview** achieves the best average performance with **80.17% Pass Rate** and **71.59% Resolve Rate**. Among open models, **GLM-4.7** is strongest on average with **69.20% Pass Rate** and **55.98% Resolve Rate** [2604.19667].

A defining empirical pattern is the persistent gap between syntactic or representational correctness and actual task success. Resolve Rate is always below Pass Rate, with the largest average discrepancy reaching **20.96%** for **GLM-4.6**, and a task-specific discrepancy of **43.44%** in the **Education** scenario for the same model [2604.19667]. The benchmark therefore operationalizes a distinction that is often blurred in generation tasks: a workflow may parse, convert, and import successfully while still failing to execute correctly or satisfy the requirement semantics.

## 5. Error modes, multi-round degradation, and robustness strategies

The paper identifies three concrete failure modes through a case study on the **StudyPlanner** task [2604.19667]. First, models can produce **invalid edge connections**, such as links that violate the internal structure of special nodes like iteration blocks. Second, they can produce **logical inconsistency**, for example by mentioning nodes in the workflow that were not declared in `<node_selection>`. Third, they can generate workflows with **executable but incorrect behavior**: the artifact imports successfully, yet still fails to solve the task.

Performance also degrades across dialogue rounds. Each task contains **2–4 rounds**, and later rounds modify earlier requirements; for most models, both Pass Rate and Resolve Rate decline steadily over the first three rounds [2604.19667]. This indicates difficulty in preserving coherent graph semantics under iterative revision, rather than only difficulty in initial graph generation.

To mitigate recurrent execution errors, the paper proposes an **error-driven agentic baseline** built with the **OpenCode** framework [2604.19667]. This baseline moves from zero-shot prompting to a **SKILL-based paradigm**, explicitly encoding task-critical guidelines, multi-turn rules, and variable references. It also extracts **Variable Summaries** from prior iterations to reduce context decay in later turns. The execution loop permits **up to 5 retry attempts** and applies targeted auto-repair for the most frequent structural and semantic failures: malformed code fences, JSON decoding failures, topological sorting violations, and node selection inconsistencies [2604.19667].

The improvement is measurable but limited. **GPT-5.2** improves from **54.71%** zero-shot Resolve Rate to **60.05%** agentic Resolve Rate, an absolute gain of **5.34%**. **GPT-5.1** improves from **39.38%** to **44.31%**, a gain of **4.93%** [2604.19667]. This suggests that execution feedback and targeted repair are useful, but they do not close the remaining gap to industrial-grade reliability.

## 6. Relation to adjacent research traditions

Chat2Workflow occupies one part of a broader research space in which conversation and workflow structure are coupled but not in the same way. In **workflow-guided response generation**, the workflow is a known policy and the objective is to generate responses that comply with it. The COMPLIANCEOPT framework combines a learned **ComplianceScorer** with RL optimization and reports strong compliance gains on **ABCD** and **MultiWOZ 2.2**, including **0.8670** compliance on ABCD for the oracle variant, close to the ground-truth score of **0.8676** [2311.08300]. In **multi-step dialogue workflow action prediction**, the output is a predicted future action sequence or action graph rather than executable workflow JSON; that setting reports **20%** more automated workflow steps than 1-step prediction [2311.09593]. These lines of work treat workflow as latent or given policy state for response planning, not as a platform-ready visual artifact.

A second adjacent tradition focuses on **extracting workflows from conversations**. The framework in "Turning Conversations into Workflows" uses procedural-element retrieval and **QA-CoT** prompting to recover service workflows from historical interactions, reporting **12.16%** average macro accuracy improvement over the baseline and an end-to-end evaluation aligned with human assessments [2502.17321]. "Dialog2Flow" learns action-driven sentence embeddings from a unified corpus of **20 task-oriented dialog datasets**, **3.4 million utterances**, and **52 domains**, then clusters utterances into workflow states; its induced graphs reach an average node-count difference of **6.86%** from the gold workflow graph for the strongest configuration [2410.18481]. These works infer workflows from existing dialogs, whereas Chat2Workflow tests direct generation of new executable workflows from requirements.

A third tradition consists of **operational conversational workflow systems**. **TaskFlow** models task-oriented dialogue as a tree with dialogue actions as nodes and transitions as edges, and in real after-sale service scenarios reduces estimated human effort from **21** to **9 person-days** while producing high rates of correctly handled dialogues [2205.06436]. Low-code architectures connect conversational frontends to execution backends, such as the **Rasa + Node-RED** design in which Node-RED acts as a low-code fulfillment middleware implementing the action server HTTP API [2410.00006]. DMN-based decision-support chatbots semi-automatically derive conversational decision workflows from existing decision tables and are designed to ask only necessary questions [2109.00822]. GUI-inspired prompt designs make submit-like acknowledgment and reset-like context switching explicit state-transition tasks in multi-turn chatbots [2507.01862]. Domain-specific chat-driven workflow agents, such as **ChatCFD** for OpenFOAM pipelines and **Chat-of-Thought** for FMEA generation, further show that chat can serve as a control layer for structured engineering processes rather than only as a response interface [2506.02019] [2506.10086].

Taken together, these neighboring lines of work imply a broader taxonomy. One regime optimizes **workflow compliance** in dialogue, another performs **workflow extraction** from historical interactions, and Chat2Workflow targets **workflow synthesis**: the direct production of executable, visual, platform-constrained workflow graphs from natural language requirements [2604.19667]. Within that taxonomy, Chat2Workflow is notable for making execution, importability, multi-round revision, and industrial deployability first-class evaluation targets rather than secondary implementation details.

Source: https://www.emergentmind.com/topics/chat2workflow