X-Ray: A Sequential 3D Representation For Generation
Abstract: We introduce X-Ray, a novel 3D sequential representation inspired by the penetrability of x-ray scans. X-Ray transforms a 3D object into a series of surface frames at different layers, making it suitable for generating 3D models from images. Our method utilizes ray casting from the camera center to capture geometric and textured details, including depth, normal, and color, across all intersected surfaces. This process efficiently condenses the whole 3D object into a multi-frame video format, motivating the utilize of a network architecture similar to those in video diffusion models. This design ensures an efficient 3D representation by focusing solely on surface information. Also, we propose a two-stage pipeline to generate 3D objects from X-Ray Diffusion Model and Upsampler. We demonstrate the practicality and adaptability of our X-Ray representation by synthesizing the complete visible and hidden surfaces of a 3D object from a single input image. Experimental results reveal the state-of-the-art superiority of our representation in enhancing the accuracy of 3D generation, paving the way for new 3D representation research and practical applications.
- Pu-dense: Sparse tensor-based point cloud geometry upsampling. IEEE Trans. Image Process., 2022.
- DeepFloyd Lab at StabilityAI. DeepFloyd IF: a novel state-of-the-art open-source text-to-image model with a high degree of photorealism and language understanding. https://www.deepfloyd.ai/deepfloyd-if, 2023. Retrieved on 2023-11-08.
- Stable video diffusion: Scaling latent video diffusion models to large datasets. CoRR, 2023.
- Efficient geometry-aware 3D generative adversarial networks. In CVPR, 2022.
- Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.08051, 2022.
- Taming transformers for high-resolution image synthesis, 2020.
- Get3d: A generative model of high quality 3d textured shapes learned from images. In Advances In Neural Information Processing Systems, 2022.
- Ref-neus: Ambiguity-reduced neural implicit surface learning for multi-view reconstruction with reflection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4251–4260, 2023.
- Generative sparse detection networks for 3d single-shot object detection. In European conference on computer vision, 2020.
- Openlrm: Open-source large reconstruction models. https://github.com/3DTopia/OpenLRM, 2023.
- LRM: large reconstruction model for single image to 3d. CoRR, abs/2311.04400, 2023.
- Efficientnerf efficient neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12902–12911, 2022.
- Trivol: Point cloud rendering via triple volumes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20732–20741, 2023a.
- Point2pix: Photo-realistic point cloud rendering via neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8349–8358, 2023b.
- Make-a-shape: a ten-million-scale 3d shape model. CoRR, 2024.
- Screened poisson surface reconstruction. ACM Trans. Graph., 2013.
- 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
- Segment anything. arXiv:2304.02643, 2023.
- Point cloud upsampling via disentangled refinement. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
- Neural sparse voxel fields. NeurIPS, 2020.
- Point-voxel CNN for efficient 3d deep learning. In NeurIPS, 2019.
- Diffusion probabilistic models for 3d point cloud generation. In CVPR, 2021.
- Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
- Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019.
- Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
- Fast training of diffusion transformer with extreme masking for 3d point clouds generation. arXiv preprint arXiv: 2312.07231, 2023.
- Point-e: A system for generating 3d point clouds from complex prompts. CoRR, abs/2212.08751, 2022.
- Scalable diffusion models with transformers. In ICCV, 2023.
- Wuerstchen: An efficient architecture for large-scale text-to-image diffusion models, 2023.
- Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NeurIPS, 2017.
- High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, 2022.
- Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
- Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
- Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054, 2024.
- Triposr: Fast 3d object reconstruction from a single image. arXiv preprint arXiv:2403.02151, 2024.
- Single-view view synthesis with multiplane images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
- Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021.
- DMV3D: denoising multi-view diffusion using 3d large reconstruction model. CoRR, abs/2311.09217, 2023.
- PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
- Pu-net: Point cloud upsampling network. In CVPR, 2018.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.