DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization (2502.04370v1)

Abstract: Text-to-3D generation automates 3D content creation from textual descriptions, which offers transformative potential across various fields. However, existing methods often struggle to align generated content with human preferences, limiting their applicability and flexibility. To address these limitations, in this paper, we propose DreamDPO, an optimization-based framework that integrates human preferences into the 3D generation process, through direct preference optimization. Practically, DreamDPO first constructs pairwise examples, then compare their alignment with human preferences using reward or large multimodal models, and lastly optimizes the 3D representation with a preference-driven loss function. By leveraging pairwise comparison to reflect preferences, DreamDPO reduces reliance on precise pointwise quality evaluations while enabling fine-grained controllability through preference-guided optimization. Experiments demonstrate that DreamDPO achieves competitive results, and provides higher-quality and more controllable 3D content compared to existing methods. The code and models will be open-sourced.

• The paper introduces DreamDPO, a framework that directly aligns text-to-3D generation with human preferences using direct preference optimization, bypassing reliance on traditional quality metrics.
• DreamDPO generates paired 3D examples, ranks them based on preferences using reward models or LMMs, and optimizes the 3D output via a modulation-based loss function focused on preference discrepancies.
• Empirical results show DreamDPO outperforms 13 state-of-the-art methods across multiple metrics like text-asset alignment and texture quality, offering enhanced control and adaptability for future 3D generation research.

DreamDPO: Aligning Text-to-3D Generation with Human Preferences

The paper delineates DreamDPO, a framework aimed at the direct incorporation of human preferences into text-to-3D generation processes. Existing text-to-3D methodologies, despite substantial achievements, frequently face challenges in aligning generated content with nuanced human subjective preferences. DreamDPO addresses this gap by integrating human preferences directly into the 3D asset generation pipeline using an optimization-based approach.

Key Methodological Advances

DreamDPO distinguishes itself through a series of innovative steps that deviate from previous score-based evaluation dependencies:

1. Pairwise Example Construction: By applying diverse Gaussian noise additions to rendered 3D images at specified diffusion timesteps, DreamDPO generates diverse pairs of 3D representations. This step is crucial for the differentiation of these images according to human aesthetic preferences.
2. Preference Ranking: Utilizing either reward models or large multimodal models (LMMs), DreamDPO ranks these pairs based on their alignment with human preferences. This ranking reflects the human subjective quality judgments embedded in either pre-trained models or large-scale datasets incorporating human annotations.
3. Preference-Guided Optimization: DreamDPO optimizes the 3D representations via a preference-driven loss function, which minimizes the difference between winning and losing examples from human pairwise comparisons. The loss function is modulation-based, whereby the gradient computed selectively accounts for scenarios when preference discrepancies exceed a predefined threshold. This helps resolve chaotic gradients that may arise from overly similar example evaluations.

Empirical Outcomes

Empirically, DreamDPO delivers pronounced advancements:

• It surpasses 13 state-of-the-art methods across six quantitative metrics within the framework of GPTEval3D, which include factors such as text-asset alignment, 3D plausibility, texture quality, and text-geometry coherence.
• The qualitative results highlight its capability to produce not only more human-aligned 3D assets but also achieve fine-grained control over texture and geometric fidelity.

Implications and Future Directions

The method's success suggests broader implications for AI-driven 3D content generation:

• Adaptability: DreamDPO provides a flexible framework capable of integrating various preference models, laying a foundation for broader applicability to different domains and user-specific requirements.
• Theoretical Underpinnings: By reducing the reliance on precise quality evaluations, DreamDPO invites discussion on the necessity and design of loss functions suitable for preference-driven models, potentially influencing future theoretical developments.
• Future Exploration: Future research could explore integration with more advanced generative backbones and reward frameworks, possibly involving real-time human preference feedback during the generation process or improving controllability via better multimodal understanding and alignment mechanisms.

In conclusion, DreamDPO represents a significant step toward aligning automated generation of 3D content with human subjective preferences, advancing both the practical capability and theoretical understanding of preference-guided AI systems. This not only enhances the usability and acceptance of such systems in creative domains but also opens avenues for new research at the intersection of human-AI interaction and multimodal machine learning.