Residual-Based Mixed Representation

• Residual-based mixed representation is a dual-component approach that fuses a coarse TSDF grid with an implicit neural residual module to model high-fidelity signals in RGB-D reconstructions.
• It enhances reconstruction detail, convergence speed, and computational efficiency by dedicating the explicit grid to low-frequency geometry and the neural module to high-frequency corrections.
• The method also refines camera pose optimization via residual bundle adjustment, ensuring globally consistent reconstructions through adaptive gradient amplification and local moving volumes.

A residual-based mixed representation combines two or more complementary forms of representation—typically separating a coarse, explicit (often grid-based) base with a finer, implicit (neural or parametric) residual component—to efficiently model complex high-fidelity signals or functions. In the context of large-scale online RGB-D reconstruction, as exemplified by RemixFusion (Lan et al., 23 Jul 2025), this paradigm integrates a fast, memory-efficient explicit structure (e.g., a truncated signed distance function, TSDF, grid) with a lightweight, detail-enriching implicit neural module that learns residual corrections. This division of representational labor addresses the limitations faced by purely explicit or purely implicit approaches—namely, the scalability and smoothing issues of neural implicit mapping and the memory or computational burden of high-resolution explicit mapping—leading to improvements in reconstruction detail, convergence speed, memory efficiency, and camera pose optimization.

1. Decomposition of Scene Representation

RemixFusion tackles dense online RGB-D reconstruction by separating the scene description into a sum of two components:

• Explicit Coarse Representation: A TSDF voxel grid (𝒱_coarse), constructed via standard volumetric fusion algorithms, that rapidly and efficiently encodes the broad geometric structure of the environment in real time.
• Implicit Residual Module: A neural network module (typically using hash embeddings and a small MLP decoder) that learns only the difference—i.e., the residual—needed to upgrade the coarse explicit map to include high-frequency geometric and appearance details.

The overall scene function is mathematically formalized as:

$$\mathcal{f} = \mathcal{f}_c \oplus \mathcal{f}_\Delta$$

where $\mathcal{f}_c$ is the explicit TSDF map and $\mathcal{f}_\Delta$ is the neural residual correction, with $\oplus$ denoting the sum or aggregation at arbitrary query locations.

At a queried 3D point $p$, the predicted attribute $O(p)$ (geometry or color) is given by:

$$O(p) = \beta(p) + \mathcal{D}(\beta(p), \varrho(p), \mathcal{H}_\Delta(p))$$

where $\beta(p)$ is the trilinear interpolation result from the explicit TSDF grid, $\varrho(p)$ is a positional encoding, $\mathcal{H}_\Delta(p)$ is a hash-embedding-based neural feature, and $\mathcal{D}$ is the decoder for residual correction.

2. Rationale and Advantages

The core motivation for the residual-based mixed representation is to reduce the learning and memory burden of neural reconstruction by offloading as much as possible onto an explicit, efficient component. The explicit TSDF grid absorbs the low-frequency, large-scale geometry, requiring minimal neural effort or memory. The neural module, correspondingly, is dedicated only to learning high-frequency corrections, which:

• Enhances reconstruction fidelity, particularly for fine shapes, textures, and sharp transitions.
• Avoids the over-smoothing observed with fully implicit neural methods.
• Shortens convergence and training times, since the neural component optimizes over a narrower, high-frequency function class.
• Decreases computational and memory costs relative to dense explicit grids of equivalent detail.

Experiments indicate that this approach outperforms state-of-the-art online reconstruction systems—both those relying purely on explicit TSDF fusion and those using full neural implicit mapping—in metrics of mapping and tracking accuracy, detail, and resource efficiency.

3. Camera Pose Optimization via Residual Bundle Adjustment

RemixFusion extends the residual-based mixed formulation to multi-frame pose optimization (bundle adjustment). Instead of directly optimizing absolute camera poses as independent variables, RemixFusion:

• Learns a small pose residual (a correction) with a dedicated MLP for each pose, such that the final pose is expressed as the initial estimate from the front-end tracker plus a correction estimated through joint optimization.
• Back-propagates rendering and geometric losses through both the pose corrections and the scene representation, enforcing global consistency across multiple frames.

This approach enables regularization and coupling across views, avoids the pitfalls of optimizing unconstrained poses, and accelerates optimization convergence, producing better-aligned 3D reconstructions and smoother trajectories.

4. Adaptive Gradient Amplification

A practical challenge in large-scale and texture-sparse environments is the presence of weak or uneven gradients in the loss (especially away from the reconstructed surface or in poorly textured regions). RemixFusion addresses this with an adaptive gradient amplification technique:

• During optimization, gradients originating near the zero-crossing (surface interface) in the TSDF grid are selectively boosted.
• This is achieved through a clamping function using an amplified threshold factor:

$$\hat{\beta}(p) = \Upsilon\left([\beta(p) \cdot (tr_c / tr_i)],\ k \cdot \tau_c\right)$$

where $\tau_c$ is the TSDF truncation threshold and $k > 1$ is an adaptive multiplier.

This amplification encourages the optimizer to more actively correct poses and details in regions critical to geometric consistency, reducing the risk of local minima and improving the global optimum of the reconstruction and trajectory.

5. Local Moving Volume and Efficiency

To support large-scale and online deployment, RemixFusion employs a divide-and-conquer factorization using a local moving volume:

• A “window” of limited spatial extent (local subvolume) follows the current camera position. Reconstruction and tracking optimizations are restricted to this region, allowing for:
  • Local updates of both explicit and implicit representation, dramatically lowering computation and memory requirements.
  • Scalable performance regardless of global scene size.
• The total scene is thus factorized into many such subvolumes, each reconstructed using the same residual-based mixed representation.

This architectural choice enables real-time performance, efficient memory reuse, and robust handling of extensive environments.

6. Quantitative Performance and Comparative Outcomes

RemixFusion demonstrates superior performance to prior baselines (explicit and implicit):

• Mapping accuracy: Higher completeness and sharpness, as indicated by metrics including D-L1 and F1-score.
• Tracking precision: Lower absolute trajectory error due to improved surface fidelity and more robust pose optimization.
• Computation and resource use: Smaller network sizes/training burden, reduced GPU memory, and faster updates owing to the explicit–implicit decomposition and local volume approach.

7. Broader Implications and Extensions

The residual-based mixed representation strategy generalizes beyond RGB-D reconstruction:

• The decoupling of explicit and residual learning can be applied in other domains where coarse-to-fine modeling is beneficial, such as image and video super-resolution, large-scale SLAM, neural field rendering, and simulation.
• The framework’s pose correction mechanism and adaptive gradient strategies address longstanding challenges in multi-frame optimization for simultaneous localization and mapping (SLAM) and scene reconstruction.

In summary, RemixFusion’s residual-based mixed representation integrates explicit and implicit components in a principled fashion, yielding an efficient, scalable, and detail-preserving solution for large-scale online scene reconstruction and camera pose estimation in RGB-D settings (Lan et al., 23 Jul 2025).