OpenCOLE: Towards Reproducible Automatic Graphic Design Generation (2406.08232v1)

Abstract: Automatic generation of graphic designs has recently received considerable attention. However, the state-of-the-art approaches are complex and rely on proprietary datasets, which creates reproducibility barriers. In this paper, we propose an open framework for automatic graphic design called OpenCOLE, where we build a modified version of the pioneering COLE and train our model exclusively on publicly available datasets. Based on GPT4V evaluations, our model shows promising performance comparable to the original COLE. We release the pipeline and training results to encourage open development.

• The paper introduces an open-source framework that addresses reproducibility barriers in automatic graphic design by using publicly accessible datasets.
• The paper details a modular architecture that combines LLMs like GPT-3.5 for design planning with fine-tuned SDXL1.0 for image generation.
• Experimental evaluations on the DESIGNERINTENTION dataset show that OpenCOLE achieves comparable design quality to COLE while enhancing transparency and accessibility.

OpenCOLE: Towards Reproducible Automatic Graphic Design Generation

The paper "OpenCOLE: Towards Reproducible Automatic Graphic Design Generation" proposes a novel framework named OpenCOLE, which addresses the reproducibility barriers present in current state-of-the-art methods for automatic graphic design generation. The authors present an open-source implementation that leverages publicly available datasets, facilitating wider accessibility and development within the community.

Introduction and Motivation

Graphic design plays a pivotal role in visual communication by integrating various multimodal elements such as text, images, and layout. The generation of these designs automatically has been an area of active research for decades. The recent advances in neural networks have spurred significant progress in specific design sub-tasks, including layout generation, font recommendation, and colorization. The demand for comprehensive and automated graphic design solutions has led to the development of frameworks like COLE. However, COLE’s reliance on proprietary datasets and its non-open nature pose significant challenges in replicating and extending its results.

Methodological Contributions

The primary contribution of OpenCOLE is the construction of an open-source framework that mirrors the architecture of COLE but employs only publicly accessible resources. This approach promotes reproducibility and democratizes the development of automatic graphic design systems. The architecture of OpenCOLE comprises three main modules:

1. Design Plan Generation Module: Utilizes an LLM, specifically GPT3.5 with in-context learning, to translate user intentions into a detailed design plan.
2. Image Generation Module: Fine-tunes SDXL1.0 to generate images corresponding to the layout described in the design plan.
3. Typography Generation Module: Adopts an LLM to generate typographic attributes, which are then integrated into the images.

Experimental Outcomes

The authors conducted rigorous experiments to evaluate OpenCOLE. The benchmarks used include the DESIGNERINTENTION dataset, which assesses graphic design quality based on design layout, content relevance, typography, graphics, and innovation. OpenCOLE demonstrated performance that closely matches the original COLE, particularly in the GPT4V-based evaluations. The results (as shown in the paper's quantitative evaluation table) reveal that OpenCOLE achieves commendable scores across all evaluated aspects, although there remains a performance gap when compared to models such as DALL-E3 and SDXL1.0 in overall quality.

Discussion and Future Directions

Although OpenCOLE marks significant progress towards open and reproducible graphic design generation, it also highlights some challenges. One crucial issue discussed is the dependency on GPT4V for evaluation, which may not always correlate perfectly with human judgment, especially in terms of text relevance and legibility within complex designs. The authors suggest that further development in evaluation methodologies is necessary to capture both the quantitative and qualitative aspects of generated designs more effectively.

Future developments in AI for automatic graphic design could benefit from integrating enhanced multi-modal understanding, improved fine-tuning techniques, and more sophisticated evaluation metrics that consider user intentions more comprehensively. By advancing these areas, we can expect to see models that not only generate high-quality designs but also ensure their elements are functionally and aesthetically coherent.

Overall, OpenCOLE represents a crucial step towards more inclusive and transparent research and development in the field of automatic graphic design generation. It opens up pathways for the community to build upon and innovate using the provided open-source resources.