FIRE3D: Scene Reconstruction in Under a Minute
FIRE3D transforms casual RGB video into complete, textured 3D environments without manual object annotation. By combining 3D instance perception with hierarchical latent compression and batched generative reconstruction, the system produces simulation-ready scenes containing more than 12 objects in under 60 seconds—achieving a 480× speedup over optimization-based methods while maintaining object-level editability and multi-view consistency.Script
Building a 3D scene from video usually means hours of optimization, manual masks, or sequential object-by-object reconstruction. FIRE3D does it in under a minute, converting casual RGB captures into editable, textured environments without a single bounding box.
The architecture lifts image features into a world-aligned 3D point cloud, then uses a query-based transformer to predict instance masks and canonical object poses in a single pass. This joint 3D treatment couples geometry and pose through the same spatial representation, ensuring multi-view consistency without ever cropping objects from individual frames.
The hierarchical compression VAE shrinks object latents by 32 times, trading modest fidelity for massive parallelism. A single A100 can now decode more than 16 objects concurrently, whereas the uncompressed representation maxes out at two.
Canonicalized point clouds feed cascaded flow-matching models that generate structure, shape, and material in sequence. Batched CUDA operations handle mesh extraction, UV unwrapping, and texture baking for all objects at once, delivering more than 10 times speedup over sequential execution.
On Imaginarium, FIRE3D achieves a 480 times speedup over HoloScene and a 32 times speedup over SimRecon, reconstructing scenes with more than 12 objects in under 60 seconds. It improves scene-level Chamfer Distance and F1 while sacrificing some normal consistency and PSNR, a trade-off that prioritizes editability and runtime over rendering fidelity.
FIRE3D makes object-level scene reconstruction practical by integrating perception, compression, and generation around a shared 3D representation. To explore the full paper and create your own videos, visit EmergentMind.com.
