SceneFactory: A Workflow-centric and Unified Framework for Incremental Scene Modeling

Abstract: We present SceneFactory, a workflow-centric and unified framework for incremental scene modeling, that conveniently supports a wide range of applications, such as (unposed and/or uncalibrated) multi-view depth estimation, LiDAR completion, (dense) RGB-D/RGB-L/Mono/Depth-only reconstruction and SLAM. The workflow-centric design uses multiple blocks as the basis for constructing different production lines. The supported applications, i.e., productions avoid redundancy in their designs. Thus, the focus is placed on each block itself for independent expansion. To support all input combinations, our implementation consists of four building blocks that form SceneFactory: (1) tracking, (2) flexion, (3) depth estimation, and (4) scene reconstruction. The tracking block is based on Mono SLAM and is extended to support RGB-D and RGB-LiDAR (RGB-L) inputs. Flexion is used to convert the depth image (untrackable) into a trackable image. For general-purpose depth estimation, we propose an unposed & uncalibrated multi-view depth estimation model (U$^2$-MVD) to estimate dense geometry. U$^2$-MVD exploits dense bundle adjustment to solve for poses, intrinsics, and inverse depth. A semantic-aware ScaleCov step is then introduced to complete the multi-view depth. Relying on U$^2$-MVD, SceneFactory both supports user-friendly 3D creation (with just images) and bridges the applications of Dense RGB-D and Dense Mono. For high-quality surface and color reconstruction, we propose Dual-purpose Multi-resolutional Neural Points (DM-NPs) for the first surface accessible Surface Color Field design, where we introduce Improved Point Rasterization (IPR) for point cloud based surface query. ...

• The paper introduces SceneFactory, a modular framework that integrates independent blocks for robust incremental scene modeling without redundancy.
• It employs an innovative U²-MVD model for unposed multi-view depth estimation, achieving competitive results on datasets like KITTI and Replica.
• The framework delivers high-quality surface and color reconstructions in SLAM applications, offering versatility for robotics and dynamic scene analysis.

SceneFactory: A Unified Framework for Incremental Scene Modeling

Overview

The paper introduces SceneFactory, a versatile and modular framework designed for incremental scene modeling. It supports a plethora of applications including multi-view depth estimation, LiDAR completion, RGB-D, RGB-L, Mono, and Depth-only reconstruction, as well as SLAM. Its workflow-centric design employs multiple blocks, which can be independently expanded and combined to avoid redundancy and facilitate ease of use.

Modular Design

Building Blocks

SceneFactory incorporates four primary blocks:

1. Mono-SLAM Block: Utilizes minimal sensor input for tracking and mapping.
2. Depth Estimation Block: Handles dense depth estimation and completion.
3. Flexion Block: Converts depth images to flexion images for improved feature matching.
4. Scene Reconstruction Block: Generates high-quality surface and color reconstructions using multi-resolution neural points.

Each block can function independently or combine with others for complex tasks.

Unposed and Uncalibrated Multi-View Depth Estimation (U$^2$-MVD)

The authors propose an innovative depth estimation model, U$^2$-MVD, to estimate dense geometry using dense bundle adjustment (DBA). This model does not require pre-existing camera poses or intrinsic parameters, making it highly flexible. The ScaleCov step completes the multi-view depth by leveraging deep learned covariances to fill in missing regions.

Practical Applications and Results

High-Quality Surface and Color Reconstruction

The framework leverages Dual-purpose Multi-resolutional Neural Points (DM-NPs) and Improved Point Rasterization (IPR) for efficient surface query and visually appealing renderings. SceneFactory achieves outstanding results in both surface light field and monocular RGB-D SLAM reconstructions, surpassing state-of-the-art methods in many tasks.

Numerical Results

Table 1 in the paper compares multi-view depth estimation across several datasets, highlighting SceneFactory's competitiveness:

• KITTI: SceneFactory achieves the lowest mean relative error and the highest inlier ratio in most tests.
• ScanNet: Performance is slightly lower than some deep-learning models like DUSt3R, primarily due to ScanNet’s motion blur and rotational challenges.
• ETH3D, DTU, and Tanks{content}Temple: SceneFactory generally performs robustly across various settings, showing its wide applicability.

Table 2 demonstrates SceneFactory's superior performance in surface light field tasks, achieving higher PSNR and SSIM scores with lower LPIPS than competitors on the Replica and ScanNet datasets.

Broader Implications and Future Directions

SceneFactory's modular and workflow-centric design streamlines the process of building and extending scene modeling pipelines. This flexibility can significantly benefit fields like robotics, where quick adaptation to different sensing setups and environmental conditions is crucial.

Speculative Future Developments:

• Deformable Reconstruction: Extending SceneFactory to handle non-rigid scenes.
• Active SLAM: Incorporating decision-making processes for better path planning and sensor usage.
• Scene Understanding: Integrating higher-level semantic understanding into the scene modeling process.

Conclusion

SceneFactory presents a flexible, modular approach to incremental scene modeling, accommodating various inputs and applications with high performance and practicality. It stands as a robust competitor to existing tightly-coupled methods and sets a new standard for scene modeling frameworks. The open access to its codebase can stimulate further research and development in this domain.