DreamReward: Text-to-3D Generation with Human Preference (2403.14613v1)

Abstract: 3D content creation from text prompts has shown remarkable success recently. However, current text-to-3D methods often generate 3D results that do not align well with human preferences. In this paper, we present a comprehensive framework, coined DreamReward, to learn and improve text-to-3D models from human preference feedback. To begin with, we collect 25k expert comparisons based on a systematic annotation pipeline including rating and ranking. Then, we build Reward3D -- the first general-purpose text-to-3D human preference reward model to effectively encode human preferences. Building upon the 3D reward model, we finally perform theoretical analysis and present the Reward3D Feedback Learning (DreamFL), a direct tuning algorithm to optimize the multi-view diffusion models with a redefined scorer. Grounded by theoretical proof and extensive experiment comparisons, our DreamReward successfully generates high-fidelity and 3D consistent results with significant boosts in prompt alignment with human intention. Our results demonstrate the great potential for learning from human feedback to improve text-to-3D models.

• The paper introduces DreamReward, a framework that integrates human preference feedback into text-to-3D generative modeling.
• The methodology employs Reward3D, trained on 25k expert-graded pairs, and DreamFL, a tuning algorithm enhancing multi-view diffusion models.
• Empirical results show significant improvements over baselines, with higher GPTEval3D, CLIP, and ImageReward scores.

DreamReward: Enhancing Text-to-3D Generative Models with Human Preferences

Introduction to DreamReward

Text-to-3D generation has garnered significant interest, offering applications in a myriad of sectors including, but not limited to, entertainment, architecture, and virtual reality. Despite this, the fidelity and alignment of 3D generated content to human expectations remain a challenge. In addressing these limitations, this paper introduces DreamReward, a novel framework formulated to refine text-to-3D generative models utilizing human preference feedback. This framework entails the construction of Reward3D, the first general-purpose text-to-3D human preference model, followed by the implementation of Reward3D Feedback Learning (DreamFL), a direct tuning algorithm aiming at optimizing multi-view diffusion models.

Constructing Reward3D: The Human Preference Model

The formulation of Reward3D as a pivotal component of DreamReward signifies the foundational step towards optimizing text-to-3D models according to human preferences. This process commences with the strategic curation of a dataset from a meticulously designed annotation pipeline, which yielded 25k expertly graded comparison pairs. From these comparisons, the Reward3D model training was conducted, setting a precedent as the inception of a text-to-3D preference model with a focus on encoding human judgments regarding text-to-3D content quality, alignment, and consistency.

Training Reward3D:

• Dataset and Annotations: Utilizing a subset of prompts extracted from Cap3D and aided by a clustering algorithm, 2530 prompt sets were delineated, covering a wide range of themes and subjects.
• Model Architecture and Training: Inspired by advancements in reinforcement learning from human feedback in NLP and text-to-image domains, Reward3D was trained to distinguish varying quality levels among 3D content corresponding to identical textual prompts.

DreamFL: Direct Tuning with Human Feedback

Leveraging the Reward3D model, DreamFL introduces a direct tuning mechanism towards the betterment of text-to-3D generative models. This endeavor encapsulates a profound theoretical analysis, laying the groundwork for an optimization approach that directly utilizes human preference encoded through the Reward3D scores.

Key Insights and Formulation of DreamFL:

• The fundamental premise of DreamFL rests on the discrepancy between the distributions obtained from pre-trained diffusion models and the ideal distribution that mirrors human preferences closely.
• Through rigorous mathematical derivation, DreamFL proposes an optimization method that efficiently bridges this gap, incorporating human feedback into the SDS optimization loop.

Empirical Results and Analysis

DreamReward was subjected to a series of extensive experiments, pitting it against leading text-to-3D models across various metrics designed to evaluate the alignment with human intentions and overall 3D content fidelity. The results showcased a significant improvement in generating high-fidelity, multi-view consistent 3D models that better aligned with human preferences.

• Quantitative Metrics and Comparisons: DreamReward consistently outperformed baselines across several evaluation metrics including GPTEval3D, CLIP scores, and ImageReward scores.
• Qualitative Evaluations: Illustrated examples further substantiated the consistent superiority of DreamReward in generating 3D models that are both visually appealing and closely aligned with the provided textual descriptions.

Conclusion and Future Directions

DreamReward introduces a pioneering approach in incorporating human preferences into the optimization of text-to-3D generative models. This is achieved through the innovative development of Reward3D and the subsequent formulation of DreamFL for direct model tuning. The promising results open up several avenues for future research, emphasizing the potential for further exploration in merging human feedback with generative AI models to enhance their performance and relevance in practical applications. Future work may delve into expanding the diversity of the annotated dataset and exploring novel architectures for the Reward3D model to encapsulate more intricate human preferences.