FoundationSSC: Decoupled 3D Scene Completion

• FoundationSSC is a camera-based 3D semantic scene completion framework that decouples semantic and geometric priors from stereo RGB inputs using a dual decoupling strategy for enhanced voxel prediction.
• It employs source-level decoupling with a frozen foundation encoder and refines features via a Geometry-Aware Context Adapter and Disparity-to-Depth Volume Mapping to preserve uncertainty.
• The framework fuses dual 3D feature volumes using Axis-Aware Fusion, achieving state-of-the-art performance on SemanticKITTI and SSCBench benchmarks while improving long-tail class metrics.

Searching arXiv for FoundationSSC and closely related camera-based 3D semantic scene completion papers to ground citations. FoundationSSC is a camera-based 3D semantic scene completion framework that addresses the joint prediction of occupancy and semantics in a dense voxel grid from stereo RGB input. Introduced in “Unleashing Semantic and Geometric Priors for 3D Scene Completion” (Chen et al., 19 Aug 2025), it is organized around a “dual decoupling” principle: semantics and geometry are separated first at the source level, through distinct outputs of a frozen foundation encoder, and then at the pathway level, through task-specific refinement modules before 3D lifting and fusion. The framework is positioned against prior camera-based SSC systems that rely on a coupled encoder for both semantic context and geometric reasoning, and it reports simultaneous improvements in semantic and geometric metrics on SemanticKITTI and SSCBench-KITTI-360 (Chen et al., 19 Aug 2025).

1. Task setting and motivating design principle

3D semantic scene completion predicts a dense 3D voxel representation in which each voxel is assigned both occupancy and, if occupied, a semantic category. In the camera-based setting, the input is RGB imagery rather than LiDAR, which makes the problem significantly harder because the system must infer geometry and semantics for both observed and occluded regions. FoundationSSC is evaluated in autonomous-driving scenes, where the target is a complete semantic occupancy volume over the driving environment (Chen et al., 19 Aug 2025).

The framework is motivated by a structural critique of prior camera-based SSC. Earlier systems commonly used a single coupled image encoder to provide features for semantic reasoning and geometric reasoning simultaneously. The paper argues that this induces an inherent trade-off: semantic reasoning favors high-level, category-level, context-rich features, whereas geometric reasoning requires precise local structure, spatial correspondence, and depth-sensitive detail. The central claim is not merely that semantics and geometry are both important, but that they impose conflicting representational demands when forced through one shared feature source (Chen et al., 19 Aug 2025).

FoundationSSC therefore proposes “dual decoupling.” At the source level, semantic and geometric priors originate from different outputs of a frozen foundation encoder. At the pathway level, these priors are refined by different specialized modules before being lifted into 3D. This is the framework’s defining design principle. A broader pattern is visible across foundation-model-assisted dense prediction: recent systems in weakly supervised semantic segmentation and few-shot segmentation also rely on strong frozen priors plus lightweight task-specific adaptation, although they do not address the semantic-geometric conflict in 3D SSC directly (Yang et al., 2023, Chang et al., 2024).

2. Source-level decoupling through a frozen foundation encoder

FoundationSSC uses a pretrained, frozen FoundationStereo model as an integrated foundation encoder. The choice is justified by its combination of semantic representational strength inherited from DINOv2 / DepthAnythingV2 lineage and strong stereo geometric reasoning. The encoder is not fine-tuned for SSC; rather, it serves as a fixed prior source whose outputs are routed into distinct downstream branches (Chen et al., 19 Aug 2025).

The encoder provides three outputs. First, it produces monocular image features, denoted $\mathbf{F}^{2D}$, formed from multi-scale features of the frozen DepthAnythingV2 backbone and fused through a lightweight FPN. These features constitute the primary semantic prior. Second, it provides disparity cost volume features, denoted $\mathbf{V}_{disp}$, from FoundationStereo’s attentive hybrid cost filtering module. Third, it predicts a dense depth map $\mathbf{Z}$, obtained by converting the final disparity map to metric depth using camera intrinsics. The paper does not provide the explicit disparity-to-depth formula (Chen et al., 19 Aug 2025).

This separation is what the paper calls source-level decoupling. Semantics and geometry do not begin from one entangled tensor. Instead, semantic reasoning starts from $\mathbf{F}^{2D}$, while geometric reasoning starts from $\mathbf{V}_{disp}$ and $\mathbf{Z}$. The paper emphasizes that a cost volume is materially richer than a deterministic depth estimate because it preserves probabilistic matching ambiguity and uncertainty rather than collapsing them into a single map (Chen et al., 19 Aug 2025).

3. Pathway-level decoupling: semantic and geometric refinement

Pathway-level decoupling assigns different refinement mechanisms to the semantic and geometric priors. The semantic branch uses the Geometry-Aware Context Adapter (GCA), while the geometric branch uses Disparity-to-Depth Volume Mapping (DDVM) (Chen et al., 19 Aug 2025).

In the semantic branch, GCA is introduced because foundation visual features remain primarily 2D-centric and do not, by themselves, encode explicit 3D structural awareness. GCA constructs a geometry prior matrix $\mathbf{M}^{g}$ from the dense depth map $\mathbf{Z}$ by combining a depth relationship matrix $\mathbf{M}^{d}$ and a spatial distance matrix $\mathbf{M}^{s}$:

$\mathbf{V}_{disp}$0

Here, $\mathbf{V}_{disp}$1 is learnable, and $\mathbf{V}_{disp}$2. This matrix then modulates self-attention:

$\mathbf{V}_{disp}$3

The effect is to downweight interactions between image positions that are close in 2D but inconsistent in depth. Different heads use different decay rates, following DFormerV2, and attention is implemented with an axes decomposition strategy for efficiency (Chen et al., 19 Aug 2025).

In the geometric branch, DDVM addresses a specific incompatibility between stereo cost volumes and Lift-Splat-Shoot-style lifting. The input disparity volume

$\mathbf{V}_{disp}$4

is flattened spatially to

$\mathbf{V}_{disp}$5

then transformed by a learned non-linear function:

$\mathbf{V}_{disp}$6

After reshaping back to

$\mathbf{V}_{disp}$7

a shallow 3D CNN encoder $\mathbf{V}_{disp}$8 refines it, and softmax along depth yields a depth distribution:

$\mathbf{V}_{disp}$9

The paper states that $\mathbf{Z}$0 is implemented by stacked channel-mapper blocks containing FFN, layer normalization, and residual connection. The point of DDVM is not merely representation conversion, but retention of uncertainty and ambiguity from stereo matching, which the paper argues are lost when geometry is reduced to a single depth map (Chen et al., 19 Aug 2025).

4. Hybrid lifting to 3D and anisotropic feature fusion

FoundationSSC follows CGFormer in using a hybrid view transformation, but it explicitly preserves both lifted representations instead of discarding one as an intermediate. The semantic branch outputs context features

$\mathbf{Z}$1

and the geometric branch outputs categorical depth distributions

$\mathbf{Z}$2

Their outer product forms a frustum representation

$\mathbf{Z}$3

which is voxel-pooled into a 3D feature volume

$\mathbf{Z}$4

This is the Lift-Splat-Shoot-style branch. A second branch uses a voxel transformer, seeded by sparse occupied voxel proposals from the depth map and initial LSS features, to produce

$\mathbf{Z}$5

The paper characterizes $\mathbf{Z}$6 as preserving finer near-field geometry and $\mathbf{Z}$7 as providing stronger contextual reasoning for distant and occluded regions (Chen et al., 19 Aug 2025).

These two 3D feature volumes are then combined by Axis-Aware Fusion (AAF), which is designed around the anisotropy of driving scenes. Rather than using isotropic 3D channel attention with global pooling, AAF constructs three parallel fusion units aligned with the $\mathbf{Z}$8, $\mathbf{Z}$9, and $\mathbf{F}^{2D}$0 planes. Each unit has a local 3D-convolution pathway and a global anisotropic-pooling pathway. The fusion rule is:

$\mathbf{F}^{2D}$1

The learned gate $\mathbf{F}^{2D}$2 selects, for each axis-aware unit, how much should be drawn from the LSS volume versus the voxel-transformer volume. The paper’s claim is that this anisotropic merging better respects directional structure than isotropic 3D pooling (Chen et al., 19 Aug 2025).

5. Optimization, benchmarks, and ablation evidence

FoundationSSC uses the same core SSC losses as MonoScene plus auxiliary terms. The training objective includes weighted cross-entropy loss $\mathbf{F}^{2D}$3, geometric affinity loss $\mathbf{F}^{2D}$4, semantic affinity loss $\mathbf{F}^{2D}$5, a 2D semantic segmentation auxiliary loss $\mathbf{F}^{2D}$6, and a depth loss $\mathbf{F}^{2D}$7. The total loss is written as

$\mathbf{F}^{2D}$8

The paper notes a notation inconsistency: the prose refers to $\mathbf{F}^{2D}$9 while the equation uses $\mathbf{V}_{disp}$0. The reported weights are $\mathbf{V}_{disp}$1 and $\mathbf{V}_{disp}$2 (Chen et al., 19 Aug 2025).

The framework is evaluated on SemanticKITTI and SSCBench-KITTI-360 using IoU for geometric occupancy and mIoU for semantic completion. The reported benchmark results are as follows.

Dataset	IoU	mIoU
SemanticKITTI	48.12	19.32
SSCBench-KITTI-360	48.61	21.78

On SemanticKITTI, these numbers exceed the prior best overall method, SOAP, by +2.03 IoU and +0.23 mIoU. On SSCBench-KITTI-360, FoundationSSC reports state-of-the-art performance on both metrics (Chen et al., 19 Aug 2025).

The SemanticKITTI class-wise highlights reported in the paper include building 30.5, truck 9.0, bicycle 5.6, motorcycle 6.9, other-vehicle 10.0, and fence 23.1. The paper specifically emphasizes gains on long-tail classes such as bicycle and motorcycle. On KITTI-360, notable values include car 30.5, motorcycle 8.4, truck 24.9, other-vehicle 13.0, road 63.6, and building 43.6 (Chen et al., 19 Aug 2025).

The ablation study on SemanticKITTI validation is central to the paper’s argument that dual decoupling, rather than any single component, drives the improvement.

Setting	IoU	mIoU
Baseline	45.28	16.53
+ FE	46.61	18.59
+ FE + GCA	47.17	18.95
+ FE + DDVM	47.93	19.11
+ FE + GCA + DDVM	47.84	19.56
+ FE + GCA + DDVM + AAF	47.91	20.36

These numbers show that the foundation encoder alone gives +1.33 IoU and +2.06 mIoU over the baseline, GCA adds +0.56 IoU and +0.36 mIoU, DDVM adds +1.32 IoU and +0.52 mIoU, and AAF raises semantic performance from 19.56 to 20.36 mIoU. The paper also reports a dedicated DDVM comparison: “Depth Refinement” obtains 47.87 IoU and 19.83 mIoU, “Cost Volume + AR” obtains 47.76 IoU and 19.59 mIoU, and “Cost Volume + DDVM” obtains 47.91 IoU and 20.36 mIoU. This is used to argue that analytical disparity-to-depth conversion is insufficient and that learned mapping better preserves useful geometric uncertainty (Chen et al., 19 Aug 2025).

6. Significance, relation to adjacent work, and unresolved details

Within the broader literature on foundation-model-assisted dense prediction, FoundationSSC is notable for how it uses foundation priors rather than for merely using them. Related dense prediction systems in weak supervision and few-shot segmentation also exploit frozen or largely frozen foundation features plus lightweight task-specific modules, but FoundationSSC extends this pattern into volumetric scene completion by explicitly separating semantic and geometric sources, refining them in different pathways, and fusing them only after 3D lifting (Yang et al., 2023, Chang et al., 2024).

A common misconception in this area is that adding external semantic priors or stereo depth information to an otherwise coupled encoder is sufficient to resolve the semantic-geometric conflict. The paper’s position is more specific: such additions still inherit the structural bottleneck of a shared feature source. FoundationSSC’s contribution is to relocate the decoupling upstream, so that semantics and geometry are disentangled both in their origin and in their pathway-specific processing (Chen et al., 19 Aug 2025).

The framework also suggests a broader methodological implication. The backbone comparison reports that all tested visual foundation model backbones outperform EfficientNet-B7, while DINOv2-L gives the best mIoU and DepthAnythingV2-L gives higher IoU but slightly lower mIoU. The paper interprets this as a specialization-generalization trade-off, which supports the decision to retain a decoupled design rather than collapsing semantic and geometric reasoning into one unified feature stream (Chen et al., 19 Aug 2025).

Several implementation details remain unspecified in the paper excerpt. The provided text does not report optimizer type, learning rate, epoch count, batch size, image resolution, voxel grid size, disparity bin count, depth bin count, augmentation policy, runtime, latency, memory, or FLOPs for the full FoundationSSC system. The paper also does not provide a dedicated limitations section or explicit failure-case analysis in the available text. This suggests that the main unresolved questions are not about the existence of gains, which are quantitatively established, but about reproducibility detail, computational cost, and the precise regimes in which decoupling is most beneficial (Chen et al., 19 Aug 2025).

FoundationSSC therefore occupies a specific place in the evolution of camera-based SSC. Its central scientific claim is that the long-standing semantic-versus-geometry trade-off of coupled architectures is not fundamental; it can be mitigated by decoupling priors at the source, refining them in task-specialized pathways, and merging the resulting 3D features with axis-aware anisotropic fusion. The reported simultaneous gains in IoU and mIoU are the paper’s empirical evidence for that claim (Chen et al., 19 Aug 2025).