ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback (2505.17908v1)

Abstract: With the rapid advancement of generative models, general-purpose generation has gained increasing attention as a promising approach to unify diverse tasks across modalities within a single system. Despite this progress, existing open-source frameworks often remain fragile and struggle to support complex real-world applications due to the lack of structured workflow planning and execution-level feedback. To address these limitations, we present ComfyMind, a collaborative AI system designed to enable robust and scalable general-purpose generation, built on the ComfyUI platform. ComfyMind introduces two core innovations: Semantic Workflow Interface (SWI) that abstracts low-level node graphs into callable functional modules described in natural language, enabling high-level composition and reducing structural errors; Search Tree Planning mechanism with localized feedback execution, which models generation as a hierarchical decision process and allows adaptive correction at each stage. Together, these components improve the stability and flexibility of complex generative workflows. We evaluate ComfyMind on three public benchmarks: ComfyBench, GenEval, and Reason-Edit, which span generation, editing, and reasoning tasks. Results show that ComfyMind consistently outperforms existing open-source baselines and achieves performance comparable to GPT-Image-1. ComfyMind paves a promising path for the development of open-source general-purpose generative AI systems. Project page: https://github.com/LitaoGuo/ComfyMind

An Analytical Overview of "ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback"

The paper "ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback" presents a new methodology aimed at overcoming the challenges of structuring open-source frameworks to facilitate robust general-purpose generation. Through the development of the ComfyMind system, the authors aspire to leverage collaborative AI for enhanced generative and editing capabilities across diverse modalities, transcending the limitations of traditional frameworks that often crumble under real-world complexities.

Core Innovations and Methodology

ComfyMind is constructed on the ComfyUI platform and introduces several foundational innovations. The first is the Semantic Workflow Interface (SWI), designed to abstract low-level node graphs into functional modules, thus simplifying high-level compositions and mitigating structural errors. The second is the Search Tree Planning with Localized Feedback Execution mechanism, which models the generation process as a hierarchical decision-making activity, enabling adaptative corrections. This dual-system aims to counter the fragility seen in prior models and support complex generative workflows.

The execution of the SWI involves a semantic-level LLM operation that minimizes reliance on platform-specific syntax. This boosts the robustness and flexibility of execution across workflows by allowing higher-level semantic operations. Planning via a search tree addresses the necessity for adaptive correction, as it allows tasks to be treated as modules, solving them by reasoning over workflow templates. Localized feedback within the decision process of each planning node sharpens the system's adaptability to failure without necessitating full-process regeneration, significantly enhancing robustness.

Quantitative Assessment and Performance

The authors validate ComfyMind across three benchmarks: ComfyBench, GenEval, and Reason-Edit. On ComfyBench, ComfyMind brilliantly achieved a complete 100% pass in task execution and markedly increased task resolution from 32.5% to 83.0%. This is a substantial gain over the ComfyAgent baseline, affirming that ComfyMind addresses intrinsic instability by wholly eliminating JSON-level failures.

In GenEval, used for assessing text-to-image generation fidelity, ComfyMind achieved an impressive overall score of 0.90, surpassing both SD3 and Janus-Pro-7B, and even outperforming OpenAI's GPT-Image-1 in five out of six evaluation dimensions. Finally, in Reason-Edit, ComfyMind achieved a GPT-Score of 0.906, again outperforming all open-source agents and aligning closely with proprietary solutions such as GPT-Image-1.

Implications and Future Directions

The implications of ComfyMind are both practical and theoretical. Practically, it pushes open-source systems a step closer to competing with closed-source models by enabling open-source methods to deal with the complexities of generative task execution across multiple domains, establishing a foundation for scalable generative AI solutions. Theoretically, it demonstrates the effectiveness of integrating semantic abstractions with tree-based planning to handle complex task executions, suggesting a broader application of hierarchical, modular planning in AI systems.

As AI continues to evolve, the approach outlined in this paper highlights a trajectory where increasingly autonomous systems can handle multifaceted tasks grounded in semantic reasoning and localized correction strategies. This paradigm not only enhances the robustness and scalability of AI systems but also sets a framework for future exploration into adaptive and general-purpose generative AI models. The integration of real-time adaptability and feedback in decision-making processes is poised to significantly elevate the standards and applicability of these systems in dynamic, complex environments.

Future developments may focus on expanding the versatility and scalability of ComfyMind's capabilities, refining its adaptability to emerging community-contributed workflows, and enhancing user interfaces to facilitate broader applicability across different user groups, including those in non-technical fields.