ComfyGPT: Optimizing ComfyUI Workflows

• ComfyGPT is a self-optimizing multi-agent system that converts natural language into executable ComfyUI workflows using a compact link-level generation paradigm.
• It employs a coordinated four-agent design—ReformatAgent, FlowAgent, RefineAgent, and ExecuteAgent—to enhance precision, validate node connections, and reduce hallucinations.
• Evaluation on FlowDataset/FlowBench shows improved format validity, reduced node errors, and robust execution compared to traditional full JSON generation methods.

Searching arXiv for papers on ComfyGPT and related ComfyUI workflow-generation systems. ComfyGPT is a self-optimizing multi-agent system for automatic generation of executable ComfyUI workflows from natural-language task descriptions. It was introduced as a system that targets the steep learning curve created by ComfyUI’s node-based architecture, where workflows are represented as directed computational graphs with typed inputs and outputs, and where JSON workflow specifications are often long, redundant, and brittle under direct LLM generation. Its defining features are a link-level generation paradigm that models workflows as sets of node-to-node connections rather than full JSON graphs, and a specialized FlowAgent trained with supervised fine-tuning followed by reinforcement learning using GRPO to reduce hallucinations and improve execution fidelity (Huang et al., 22 Mar 2025).

1. Problem formulation and scope

ComfyUI exposes image-generation and related visual pipelines as node graphs in which each functional component is a node with typed inputs and outputs. In this setting, a workflow is a directed graph in which outputs $O$ of one node feed into inputs $I$ of another. ComfyGPT addresses the problem of mapping open-ended task descriptions into such executable workflows across diverse categories including text-to-image, image editing, style transfer, 3D generation, video generation/editing, and others (Huang et al., 22 Mar 2025).

The motivation arises from three specific difficulties. First, ComfyUI workflows contain intricate connections: nodes often have multiple inputs and outputs, and workflows exhibit many-to-many link patterns. Second, the ecosystem contains diverse modules, including thousands of node types with evolving names and I/O signatures, which makes schema validation and execution brittle. Third, full JSON graph representations can exceed typical LLM context limits and can accumulate errors in open-loop multi-step prompting. ComfyGPT therefore aims to improve robustness and accuracy by decomposing workflow generation to link-level primitives, penalizing invalid nodes during optimization, and reconciling outdated node names through retrieval against a continuously updated node database (Huang et al., 22 Mar 2025).

This positioning is important in the broader line of ComfyUI workflow-generation systems. The paper explicitly contrasts ComfyGPT with ComfyGen, which is restricted to text-to-image workflows; HuggingGPT, which delegates tasks through external APIs in an open-loop manner; and ComfyBench-style multi-agent prompting systems that remain constrained by context and open-loop prompting. A plausible implication is that ComfyGPT should be understood as an early attempt to shift workflow generation from prompt engineering toward structured graph synthesis with execution-aware optimization (Huang et al., 22 Mar 2025).

2. Multi-agent architecture and closed-loop optimization

ComfyGPT comprises four coordinated agents: ReformatAgent, FlowAgent, RefineAgent, and ExecuteAgent. These agents form a closed optimization loop in which generation, correction, and execution are coupled rather than treated as isolated steps (Huang et al., 22 Mar 2025).

Agent	Role	Input/Output
ReformatAgent	Converts JSON workflows into a concise link-level diagram format	Input: workflow JSON or description; Output: link diagram $D=\{l_i\}$
FlowAgent	Primary generator from description to link diagram	Input: natural-language description; Output: generated diagram $d$
RefineAgent	Validator-refiner using a node database $K$ with 6,362 nodes	Input: $d$, description, incorrect node, TopK candidates; Output: corrected diagram $d'$
ExecuteAgent	Reconstructs JSON and runs the result on the ComfyUI server	Input: $d'$; Output: JSON workflow, execution status, runtime errors

ReformatAgent reduces context load by converting lengthy JSON into a diagram of directed link tuples. Each tuple is written as $[n_{out}, O_{out}^j, n_{in}, I_{in}^k]$, emphasizing connectivity rather than node internals. This provides compact training targets and reduces redundancy. FlowAgent then maps a description directly to such link diagrams. RefineAgent is a secondary validator that integrates an LLM with a knowledge-retrieval node database and replaces incorrect or obsolete node names with semantically appropriate candidates. ExecuteAgent converts the refined diagram back into ComfyUI JSON and executes it on the ComfyUI server (Huang et al., 22 Mar 2025).

The training and inference loops are explicitly coupled. During training, ReformatAgent produces diagram targets, FlowAgent undergoes SFT and RL, RefineAgent updates references to the latest node names, and ExecuteAgent provides pass/fail signals during benchmark evaluation. During inference, FlowAgent generates $d$, RefineAgent corrects it to $I$0, and ExecuteAgent runs the result. The paper describes feedback as manifesting through RL improvements to FlowAgent, RefineAgent updates that prevent execution errors from outdated names, and ablations that isolate the contribution of each agent (Huang et al., 22 Mar 2025).

3. Link-level workflow generation

The central technical idea in ComfyGPT is to generate individual node links rather than entire workflows. Formally, the node set is $I$1, each node $I$2 has inputs $I$3 and outputs $I$4, and the workflow is represented as a link set

$I$5

This representation decomposes graph synthesis into typed connections and thereby narrows the error surface relative to direct JSON emission (Huang et al., 22 Mar 2025).

The baseline graph-level approach is described as direct LLM emission of the entire JSON graph, followed by schema validation and execution, with failures often arising from node-name hallucinations or mismatched I/O types. In ComfyGPT’s link-level approach, ReformatAgent transforms workflows into link lists, FlowAgent predicts a sequence $I$6, canonicalization enforces deterministic node identifiers such as CLIP Text Encoder_0 or KSampler_1, RefineAgent resolves invalid nodes using retrieval from $I$7, and ExecuteAgent reconstructs JSON for execution (Huang et al., 22 Mar 2025).

The paper gives the following inference sketch:

1. $I$8
2. if invalid_nodes(d): $I$9
3. json ← ConvertDiagramToJSON(d)
4. status ← ExecuteOnComfyUI(json)
5. return json, status

The reason this helps is stated in three parts. Precision improves because each link must respect node I/O names and invalid edges are easier to detect. The error surface is reduced because the model no longer emits full JSON objects with sprawling metadata. Context usage also improves because link lists are more compact than raw workflow JSON. This suggests that the contribution is not merely a new serialization format, but a change in the granularity at which the generation problem is posed (Huang et al., 22 Mar 2025).

The paper’s basic text-to-image example illustrates this representation with nodes such as CLIP Text Encoder_0, KSampler_0, VAELoader_0, VAEDecode_0, and SaveImage_0, linked by tuples such as [CLIP Text Encoder_0, CONDITIONING, KSampler_0, positive] and [KSampler_0, output, VAEDecode_0, samples], with parameters including sampler="Euler" and steps=30 (Huang et al., 22 Mar 2025).

4. Training regime, datasets, and evaluation metrics

ComfyGPT is trained and evaluated with FlowDataset and FlowBench. FlowDataset contains 13,571 workflow-description pairs collected from OpenArt, LibLib, ComfyWorkflows, and Civital. Its six core categories are Text-to-Image Generation, Image Editing, Style Transfer, 3D Generation, Video Editing/Generation, and Others; Image Editing is further subdivided into HD Upscaling/Image Restoration, Redrawing, Outpainting, Character-Based Guidance, Face Swap, and Background Change/Remove. Preprocessing includes JSON schema validation, removal or expansion of special nodes, deletion of redundant nodes such as Note and Reroute, graph-connectivity enforcement, preliminary execution validation, semantic enhancement of descriptions by ChatGPT-4.0 mini, and LLM-assisted categorization with manual consolidation. The training split contains 12,571 samples, and 1,000 reserved samples form FlowBench (Huang et al., 22 Mar 2025).

FlowBench is a 1,000-sample benchmark drawn from FlowDataset with proportional sampling by category and workflow length and with the additional construction condition that at least 70% of the data in each category is executable. The evaluation protocol requires models to generate workflows from descriptions and assesses them with four metrics: Format Validation (FV), Pass Accuracy (PA), Pass Instruct Alignment (PIA), and Pass Node Diversity (PND) (Huang et al., 22 Mar 2025).

The metrics are tailored to workflow generation rather than free-form text generation. FV checks node existence and I/O consistency and is defined as

$D=\{l_i\}$0

where $D=\{l_i\}$1 indicates whether all nodes exist and I/O specifications are consistent. PA measures execution success on the ComfyUI server:

$D=\{l_i\}$2

PIA measures semantic alignment of successfully executed workflows, using ChatGPT-4.0 mini to judge whether the node-name list can fulfill the description:

$D=\{l_i\}$3

PND measures diversity of node types across passed workflows and is defined as the size of the union of node-type sets over all passed workflows (Huang et al., 22 Mar 2025).

A notable design choice is that these metrics are used for evaluation, but not directly as RL reward components in the original ComfyGPT paper. The RL reward is strictly node-existence based. This is explicitly stated, and it distinguishes the system from later work such as ComfyUI-R1, which uses a fine-grained rule–metric hybrid reward, and ComfySearch, which uses hierarchical terminal rewards with validator-mediated trajectory supervision (Xu et al., 11 Jun 2025, Su et al., 7 Jan 2026).

5. FlowAgent optimization and empirical performance

FlowAgent is the primary generator in ComfyGPT. The final version uses Qwen2.5-14B as the backbone and is trained first with supervised fine-tuning and then with reinforcement learning using GRPO. For SFT, the model is trained on the 12,571-sample FlowDataset training split for 3 epochs with learning rate $D=\{l_i\}$4, batch size 1, and 4×80GB NVIDIA A100 GPUs. The objective is

$D=\{l_i\}$5

The goal is to teach the model both the diagram representation and the mapping from natural-language descriptions to link sequences (Huang et al., 22 Mar 2025).

The RL stage addresses node hallucination. Let $D=\{l_i\}$6 denote valid node names in the node database. For a generated diagram $D=\{l_i\}$7, the reward is

$D=\{l_i\}$8

Advantages are normalized within a group:

$D=\{l_i\}$9

The GRPO objective uses ratio clipping and a KL penalty. Training uses 8×80GB A100 GPUs, 300 RL steps, learning rate $d$0, batch size 8, group size $d$1, $d$2, and $d$3 (Huang et al., 22 Mar 2025).

On FlowBench, the reported few-shot baseline ceiling is Claude-3.7 Sonnet with FV 16.8%, PA 15.8%, PIA 15.8%, and PND 72. ComfyGPT with SFT only reaches substantially higher scores across several backbones: Llama-13B yields FV 87.0%, PA 85.2%, PIA 84.2%, PND 251; Baichuan2-13B yields FV 87.2%, PA 84.4%, PIA 82.9%, PND 294; ChatGLM3-6B yields FV 87.2%, PA 84.3%, PIA 82.9%, PND 227; and Qwen2.5-14B yields FV 89.5%, PA 85.6%, PIA 84.6%, PND 333. The Qwen2.5-14B model with GRPO reaches FV 90.0%, PA 86.0%, PIA 84.8%, and PND 321. Relative to the best few-shot baseline, the reported gains are +73.2 percentage points in FV, +70.2 percentage points in PA, +69.0 percentage points in PIA, and +249 distinct node types in PND (Huang et al., 22 Mar 2025).

The ablation results clarify the role of the four-agent design. FlowAgent alone yields FV 74.8%, PA 66.0%, PIA 64.2%, and PND 183. Adding ReformatAgent and ExecuteAgent increases these to FV 85.9%, PA 83.5%, PIA 82.4%, and PND 283. Adding RefineAgent further increases them to FV 89.5%, PA 85.6%, PIA 84.6%, and PND 333. The RL increment over SFT-only Qwen2.5-14B is smaller but consistent: FV increases by 0.5 percentage points, PA by 0.4, and PIA by 0.2, while PND decreases by 12. The paper interprets this as reduced hallucination and slightly lower diversity as the policy converges (Huang et al., 22 Mar 2025).

The system also improves over ComfyAgent on ComfyBench, where ComfyAgent achieves a 56.0% pass rate and ComfyGPT+GRPO achieves 81.0%, a gain of 25 percentage points. Inference uses 1×80GB A100 GPU with max tokens=8192, top-p=0.7, and temperature=0.95 (Huang et al., 22 Mar 2025).

6. Error modes, case studies, and later developments

The paper identifies several failure modes. SFT-only FlowAgent can hallucinate nonexistent nodes; GRPO and RefineAgent mitigate this. Invalid links can still arise through mismatched input/output names or types; FV detects these, but RefineAgent cannot correct inconsistent types when the underlying semantics are wrong. Parameter misconfiguration, such as incorrect steps, sampler settings, or latent dimensions, is not directly addressed by the RL reward. PIA also depends on an LLM-based yes/no judgment and can therefore be noisy, especially for ambiguous descriptions. The system struggles with extremely novel or rare node types absent from the node database $d$4, and with very long multi-stage workflows that still approach context limits despite the compact link representation (Huang et al., 22 Mar 2025).

The case studies are illustrative of intended use. One example describes generation of consistent-style images using an uploaded character profile and style reference, with LoRA support and HD repair if needed. The resulting refined graph includes nodes such as ImageResizeKJ_0, CLIPTextEncodeFlux_0, LoraLoader_0, UNETLoader_0, ModelSamplingFlux_0, BasicGuider_0, BasicScheduler_0, RandomNoise_0, SamplerCustomAdvanced_0, InpaintModelConditioning_0, VAELoader_0, VAEDecode_0, and SaveImage_0, with RefineAgent correcting ReplaceString to LogicUtil_ReplaceString. Another example is a simpler SD-like text-to-image pipeline using CLIP Text Encoder_0, KSampler_0, RandomNoise_0, BasicScheduler_0, VAELoader_0, VAEDecode_0, and SaveImage_0, with sampler="Euler" and steps=30 (Huang et al., 22 Mar 2025).

Later work places ComfyGPT in a broader methodological trajectory. ComfyUI-R1 reformulates automated workflow generation around long chain-of-thought reasoning, a code-level representation reversible to ComfyUI JSON, and a rule–metric hybrid GRPO reward; it reports 97% format validity, node-level F1 of 0.62, graph-level F1 of 0.51, and a ComfyBench pass rate of 0.67 for its 7B model (Xu et al., 11 Jun 2025). ComfySearch pushes further toward execution-grounded search with state-aware validation of atomic graph edits, in-place repair conditioned on validator diagnostics, and entropy-adaptive branching under rollout budgets; it reports 92.5% Pass and 71.5% Resolve on ComfyBench, and 0.82 overall on GenEval (Su et al., 7 Jan 2026). This suggests that ComfyGPT’s link-level generation and self-optimizing multi-agent loop helped define a design space that later systems explored through code serialization, explicit reasoning traces, and validation-guided search.

The future directions proposed for ComfyGPT itself are consistent with that trajectory. They include reward shaping with execution-based and instruction-alignment signals, extension of the link-level paradigm to other node-based systems such as audio pipelines and robotics toolchains, expansion of FlowDataset and FlowBench with richer parameter annotations and version-tracked node schemas, augmentation of RefineAgent with parameter validation and parameter inference, and environment-aware execution simulation to reduce dependence on specific ComfyUI server configurations (Huang et al., 22 Mar 2025).