--- title: 'ComfyGPT: Optimizing ComfyUI Workflows' url: https://www.emergentmind.com/topics/comfygpt type: topic --- # ComfyGPT: Optimizing ComfyUI Workflows Searching arXiv for recent papers on ComfyGPT and related ComfyUI workflow-generation systems. ComfyGPT is a self-optimizing multi-agent system for automatic generation of executable ComfyUI workflows from natural-language task descriptions. It was introduced as a system that targets the steep learning curve created by ComfyUI’s node-based architecture, where workflows are represented as directed computational graphs with typed inputs and outputs, and where JSON workflow specifications are often long, redundant, and brittle under direct large language model generation. Its defining features are a link-level generation paradigm that models workflows as sets of node-to-node connections rather than full JSON graphs, and a specialized FlowAgent trained with supervised fine-tuning followed by reinforcement learning using GRPO to reduce hallucinations and improve execution fidelity [2503.17671]. ## 1. Problem formulation and scope ComfyUI exposes image-generation and related visual pipelines as node graphs in which each functional component is a node with typed inputs and outputs. In this setting, a workflow is a directed graph in which outputs $O$ of one node feed into inputs $I$ of another. ComfyGPT addresses the problem of mapping open-ended task descriptions into such executable workflows across diverse categories including text-to-image, image editing, style transfer, 3D generation, video generation/editing, and others [2503.17671]. The motivation arises from three specific difficulties. First, ComfyUI workflows contain intricate connections: nodes often have multiple inputs and outputs, and workflows exhibit many-to-many link patterns. Second, the ecosystem contains diverse modules, including thousands of node types with evolving names and I/O signatures, which makes schema validation and execution brittle. Third, full JSON graph representations can exceed typical LLM context limits and can accumulate errors in open-loop multi-step prompting. ComfyGPT therefore aims to improve robustness and accuracy by decomposing workflow generation to link-level primitives, penalizing invalid nodes during optimization, and reconciling outdated node names through retrieval against a continuously updated node database [2503.17671]. This positioning is important in the broader line of ComfyUI workflow-generation systems. The paper explicitly contrasts ComfyGPT with ComfyGen, which is restricted to text-to-image workflows; HuggingGPT, which delegates tasks through external APIs in an open-loop manner; and ComfyBench-style multi-agent prompting systems that remain constrained by context and open-loop prompting. A plausible implication is that ComfyGPT should be understood as an early attempt to shift workflow generation from prompt engineering toward structured graph synthesis with execution-aware optimization [2503.17671]. ## 2. Multi-agent architecture and closed-loop optimization ComfyGPT comprises four coordinated agents: ReformatAgent, FlowAgent, RefineAgent, and ExecuteAgent. These agents form a closed optimization loop in which generation, correction, and execution are coupled rather than treated as isolated steps [2503.17671]. | Agent | Role | Input/Output | |---|---|---| | ReformatAgent | Converts JSON workflows into a concise link-level diagram format | Input: workflow JSON or description; Output: link diagram $D=\{l_i\}$ | | FlowAgent | Primary generator from description to link diagram | Input: natural-language description; Output: generated diagram $d$ | | RefineAgent | Validator-refiner using a node database $K$ with 6,362 nodes | Input: $d$, description, incorrect node, TopK candidates; Output: corrected diagram $d'$ | | ExecuteAgent | Reconstructs JSON and runs the result on the ComfyUI server | Input: $d'$; Output: JSON workflow, execution status, runtime errors | ReformatAgent reduces context load by converting lengthy JSON into a diagram of directed link tuples. Each tuple is written as $[n_{out}, O_{out}^j, n_{in}, I_{in}^k]$, emphasizing connectivity rather than node internals. This provides compact training targets and reduces redundancy. FlowAgent then maps a description directly to such link diagrams. RefineAgent is a secondary validator that integrates an LLM with a knowledge-retrieval node database and replaces incorrect or obsolete node names with semantically appropriate candidates. ExecuteAgent converts the refined diagram back into ComfyUI JSON and executes it on the ComfyUI server [2503.17671]. The training and inference loops are explicitly coupled. During training, ReformatAgent produces diagram targets, FlowAgent undergoes SFT and RL, RefineAgent updates references to the latest node names, and ExecuteAgent provides pass/fail signals during benchmark evaluation. During inference, FlowAgent generates $d$, RefineAgent corrects it to $d'$, and ExecuteAgent runs the result. The paper describes feedback as manifesting through RL improvements to FlowAgent, RefineAgent updates that prevent execution errors from outdated names, and ablations that isolate the contribution of each agent [2503.17671]. ## 3. Link-level workflow generation The central technical idea in ComfyGPT is to generate individual node links rather than entire workflows. Formally, the node set is $N=\{n_k\}$, each node $n_k$ has inputs $I^k=\{I_m^k\}$ and outputs $O^k=\{O_m^k\}$, and the workflow is represented as a link set $$ D=\{l_i\}, \qquad l_i=[n_{out}, O_{out}^j, n_{in}, I_{in}^k]. $$ This representation decomposes graph synthesis into typed connections and thereby narrows the error surface relative to direct JSON emission [2503.17671]. The baseline graph-level approach is described as direct LLM emission of the entire JSON graph, followed by schema validation and execution, with failures often arising from node-name hallucinations or mismatched I/O types. In ComfyGPT’s link-level approach, ReformatAgent transforms workflows into link lists, FlowAgent predicts a sequence $L=[l_1 \ldots l_T]$, canonicalization enforces deterministic node identifiers such as `CLIP Text Encoder_0` or `KSampler_1`, RefineAgent resolves invalid nodes using retrieval from $K$, and ExecuteAgent reconstructs JSON for execution [2503.17671]. The paper gives the following inference sketch: 1. $d \leftarrow \text{FlowAgent}(\text{desc})$ 2. if `invalid_nodes(d)`: $d \leftarrow \text{RefineAgent}(d,\text{desc},K)$ 3. `json ← ConvertDiagramToJSON(d)` 4. `status ← ExecuteOnComfyUI(json)` 5. return `json, status` The reason this helps is stated in three parts. Precision improves because each link must respect node I/O names and invalid edges are easier to detect. The error surface is reduced because the model no longer emits full JSON objects with sprawling metadata. Context usage also improves because link lists are more compact than raw workflow JSON. This suggests that the contribution is not merely a new serialization format, but a change in the granularity at which the generation problem is posed [2503.17671]. The paper’s basic text-to-image example illustrates this representation with nodes such as `CLIP Text Encoder_0`, `KSampler_0`, `VAELoader_0`, `VAEDecode_0`, and `SaveImage_0`, linked by tuples such as `[CLIP Text Encoder_0, CONDITIONING, KSampler_0, positive]` and `[KSampler_0, output, VAEDecode_0, samples]`, with parameters including `sampler="Euler"` and `steps=30` [2503.17671]. ## 4. Training regime, datasets, and evaluation metrics ComfyGPT is trained and evaluated with FlowDataset and FlowBench. FlowDataset contains 13,571 workflow-description pairs collected from OpenArt, LibLib, ComfyWorkflows, and Civital. Its six core categories are Text-to-Image Generation, Image Editing, Style Transfer, 3D Generation, Video Editing/Generation, and Others; Image Editing is further subdivided into HD Upscaling/Image Restoration, Redrawing, Outpainting, Character-Based Guidance, Face Swap, and Background Change/Remove. Preprocessing includes JSON schema validation, removal or expansion of special nodes, deletion of redundant nodes such as `Note` and `Reroute`, graph-connectivity enforcement, preliminary execution validation, semantic enhancement of descriptions by ChatGPT-4.0 mini, and LLM-assisted categorization with manual consolidation. The training split contains 12,571 samples, and 1,000 reserved samples form FlowBench [2503.17671]. FlowBench is a 1,000-sample benchmark drawn from FlowDataset with proportional sampling by category and workflow length and with the additional construction condition that at least 70% of the data in each category is executable. The evaluation protocol requires models to generate workflows from descriptions and assesses them with four metrics: Format Validation (FV), Pass Accuracy (PA), Pass Instruct Alignment (PIA), and Pass Node Diversity (PND) [2503.17671]. The metrics are tailored to workflow generation rather than free-form text generation. FV checks node existence and I/O consistency and is defined as $$ FV = \frac{1}{N}\sum_{i=1}^{N} V(w_i), $$ where $V(w_i)\in\{0,1\}$ indicates whether all nodes exist and I/O specifications are consistent. PA measures execution success on the ComfyUI server: $$ PA = \frac{1}{N}\sum_{i=1}^{N} E(j_i). $$ PIA measures semantic alignment of successfully executed workflows, using ChatGPT-4.0 mini to judge whether the node-name list can fulfill the description: $$ PIA = \frac{1}{\sum_i E(j_i)} \sum_{i=1}^{N} E(j_i)\cdot A(desc_i,w_i). $$ PND measures diversity of node types across passed workflows and is defined as the size of the union of node-type sets over all passed workflows [2503.17671]. A notable design choice is that these metrics are used for evaluation, but not directly as RL reward components in the original ComfyGPT paper. The RL reward is strictly node-existence based. This is explicitly stated, and it distinguishes the system from later work such as ComfyUI-R1, which uses a fine-grained rule–metric hybrid reward, and ComfySearch, which uses hierarchical terminal rewards with validator-mediated trajectory supervision [2506.09790] [2601.04060]. ## 5. FlowAgent optimization and empirical performance FlowAgent is the primary generator in ComfyGPT. The final version uses Qwen2.5-14B as the backbone and is trained first with supervised fine-tuning and then with reinforcement learning using GRPO. For SFT, the model is trained on the 12,571-sample FlowDataset training split for 3 epochs with learning rate $5\times10^{-5}$, batch size 1, and 4×80GB NVIDIA A100 GPUs. The objective is $$ J_{\text{SFT}}(\theta)=\mathbb{E}_{desc,d\sim P(FD)}\left[\sum_{t=1}^{T}\log \pi_\theta(d_t \mid desc, d_{