Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation (2410.00890v3)

Abstract: Generating high-quality 3D content from text, single images, or sparse view images remains a challenging task with broad applications. Existing methods typically employ multi-view diffusion models to synthesize multi-view images, followed by a feed-forward process for 3D reconstruction. However, these approaches are often constrained by a small and fixed number of input views, limiting their ability to capture diverse viewpoints and, even worse, leading to suboptimal generation results if the synthesized views are of poor quality. To address these limitations, we propose Flex3D, a novel two-stage framework capable of leveraging an arbitrary number of high-quality input views. The first stage consists of a candidate view generation and curation pipeline. We employ a fine-tuned multi-view image diffusion model and a video diffusion model to generate a pool of candidate views, enabling a rich representation of the target 3D object. Subsequently, a view selection pipeline filters these views based on quality and consistency, ensuring that only the high-quality and reliable views are used for reconstruction. In the second stage, the curated views are fed into a Flexible Reconstruction Model (FlexRM), built upon a transformer architecture that can effectively process an arbitrary number of inputs. FlemRM directly outputs 3D Gaussian points leveraging a tri-plane representation, enabling efficient and detailed 3D generation. Through extensive exploration of design and training strategies, we optimize FlexRM to achieve superior performance in both reconstruction and generation tasks. Our results demonstrate that Flex3D achieves state-of-the-art performance, with a user study winning rate of over 92% in 3D generation tasks when compared to several of the latest feed-forward 3D generative models.

• The paper demonstrates that Flex3D significantly improves 3D reconstruction by curating high-quality candidate views and integrating them through a flexible reconstruction model.
• Flex3D employs a transformer-based FlexRM that processes an arbitrary number of input views using a tri-plane representation and 3D Gaussian Splatting for detailed output.
• The framework achieves state-of-the-art performance with a 92% user study win rate, highlighting its potential impact on AR, VR, gaming, and other 3D applications.

An Overview of Flex3D: Feed-Forward 3D Generation With Flexible Reconstruction Model And Input View Curation

The paper presents Flex3D, a novel framework for generating high-quality 3D models from text inputs, single images, or sparse view images. Flex3D addresses the challenges inherent in existing 3D generation techniques, which often struggle with limited and fixed input views, resulting in suboptimal 3D reconstructions when synthesized views lack quality. The proposed method innovates by allowing an arbitrary number of high-quality input views, thus enhancing the 3D generation process.

Methodology

The Flex3D framework comprises two primary stages: candidate view generation and curation, followed by 3D reconstruction using a flexible reconstruction model (FlexRM).

1. View Generation and Curation: The first stage combines multi-view image and video diffusion models, finely tuned to produce a diverse set of candidate views from an initial text or image prompt. These views are then curated to select the most consistent and high-quality representations. This selection is crucial, as poor-quality views can significantly impair the final 3D reconstruction.
2. Flexible Reconstruction Model (FlexRM): The second stage involves FlexRM, which is capable of processing an arbitrary number of input views. Built upon a transformer architecture, FlexRM uses a tri-plane representation to output detailed 3D Gaussian points. This approach enhances the fidelity and efficiency of the 3D reconstruction.

FlexRM distinguishes itself by its scalability (handling variable input views) and memory efficiency. This is achieved through stronger camera conditioning within the model and an innovative method of combining the tri-plane features with 3D Gaussian Splatting—a technique that facilitates rapid and high-quality rendering of 3D objects.

Results and Contributions

The paper reports that Flex3D achieves state-of-the-art performance in 3D generation tasks, with the framework exhibiting a notable user paper winning rate of over 92% compared to existing feed-forward 3D generative models. This indicates a substantial improvement in both reconstruction quality and realism of 3D assets generated from diverse input sources.

The major contributions of the work include:

• View Generation Strategy: The introduction of a view generation pipeline capable of producing a pool of candidate views, followed by an effective selection process, ensures that only the best views contribute to the 3D reconstruction.
• FlexRM Design: A flexible reconstruction model that utilizes a transformer-based architecture to interpret input views' camera information more accurately and efficiently decode tri-plane features into 3D Gaussian points.
• Innovative Training Strategy: Use of a training procedure that simulates imperfections in input data, thereby making the reconstruction process more robust to real-world data inconsistencies.

Implications and Speculations

The implications of Flex3D are significant for fields that rely heavily on 3D content, such as gaming, augmented and virtual reality, and robotics. By allowing more flexible input configurations and improving the reconstruction quality, Flex3D sets a new benchmark for feed-forward 3D model generation.

Future developments in AI could further enhance Flex3D by integrating more advanced diffusion models or incorporating multi-modality inputs beyond text and images. As the demand for real-time 3D object generation continues to grow, Flex3D’s adaptable approach could pave the way for more robust applications in digital content creation.

In summary, this work exemplifies a critical advancement in the domain of 3D generation, both broadening the application scope and strengthening the quality of outputs possible from limited input data.