FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction

Abstract: Existing learning-based occupancy prediction methods rely on large-scale 3D annotations and generalize poorly across environments. We present FreeOcc, a training-free framework for open-vocabulary occupancy prediction from monocular or RGB-D sequences. Unlike prior approaches that require voxel-level supervision and ground-truth camera poses, FreeOcc operates without 3D annotations, pose ground truth, or any learning stage. FreeOcc incrementally builds a globally consistent occupancy map via a four-layer pipeline: a SLAM backbone estimates poses and sparse geometry; a geometrically consistent Gaussian update constructs dense 3D Gaussian maps; open-vocabulary semantics from off-the-shelf vision-LLMs are associated with Gaussian primitives; and a probabilistic Gaussian-to-occupancy projection produces dense voxel occupancy. Despite being entirely training-free and pose-agnostic, FreeOcc achieves over $2\times$ improvements in IoU and mIoU on EmbodiedOcc-ScanNet compared to prior self-supervised methods. We further introduce ReplicaOcc, a benchmark for indoor open-vocabulary occupancy prediction, and show that FreeOcc transfers zero-shot to novel environments, substantially outperforming both supervised and self-supervised baselines. Project page: https://the-masses.github.io/freeocc-web/.

• The paper introduces a training-free system that uses SLAM and 3D Gaussian mapping to generate geometrically consistent, semantic-rich occupancy maps.
• It leverages vision-language models for open-vocabulary reasoning, eliminating the need for dense annotations and pose-specific training.
• Quantitative results demonstrate significant IoU and mIoU improvements, highlighting robust generalization in novel and zero-shot environments.

Training-Free Open-Vocabulary Occupancy Prediction for Embodied AI: An Analysis of FreeOcc

Introduction

The paper "FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction" (2604.28115) introduces a paradigm shift in 3D semantic occupancy prediction for embodied agents. It addresses the critical challenge of scene understanding without reliance on dense annotations or environment-specific training, aiming to advance autonomy in robotics by delivering an online system capable of constructing geometrically consistent, semantic-rich occupancy maps from monocular or RGB-D observations. FreeOcc sets itself apart by being both training-free and pose-agnostic while providing open-vocabulary queryability—contrasting with existing approaches that depend heavily on closed-set, voxel-level supervision and precise pose data.

Background and Motivation

Recent advances in embodied semantics and 3D scene representation have been propelled by supervised learning methods, which require costly occupancy and semantic labels, and by self-supervised techniques, which, while alleviating some annotation burdens, still depend on accurate poses and often lack robust cross-domain generalization. These pipelines generally deploy 3D Gaussian Splatting (3DGS) for continuous surface modeling or voxel grids for explicit volumetric semantics—each with attendant limitations. Notably, previous learning-based methods degrade sharply when exposed to novel environments due to overfitting to the training domain, restricting their applicability to real-world, open-set scenarios in robotics and AR.

FreeOcc directly targets these deficiencies by eschewing any offline training or dataset-specific pose priors. It offers an online, incremental solution that fuses SLAM-based geometry, 3D Gaussian scene modeling, and vision-LLM (VLM) semantics into a multi-layer architecture supporting open-vocabulary reasoning.

System Architecture and Methodology

Four-Layer Incremental Mapping Pipeline

FreeOcc's architecture is structured as a four-layer hierarchical pipeline:

1. SLAM Backbone (Layer 1): Egocentric RGB(-D) image sequences are first processed with a robust SLAM algorithm (specifically DROID-SLAM), yielding globally consistent camera trajectories and sparse point cloud reconstructions. This ensures a stable spatial reference frame for downstream processing.
2. Geometrically Consistent Gaussian Mapping (Layer 2): The sparse SLAM map is densified using 3D Gaussian primitives, each parameterized by position, anisotropic scale, orientation, color, opacity, and—in FreeOcc—a language-aligned feature vector. Unlike traditional 3DGS systems optimized for rendering consistency, FreeOcc couples Gaussian parameter updates with the SLAM back end, preserving geometric consistency over incremental mapping.
3. Open-Vocabulary Semantic Association (Layer 3): Semantics are assigned using pre-trained, training-free VLMs (such as open-vocabulary segmentation models). 2D per-pixel embeddings are lifted into 3D via SLAM-derived depth/geometry and associated with the corresponding Gaussian, resulting in language-embedded (LE) Gaussians. This allows for later semantic queries in arbitrary language space, transcending fixed-label taxonomy.
4. Probabilistic Gaussian-to-Occupancy Projection (Layer 4): A volumetric occupancy field is created by projecting Gaussian support into a voxel grid, aggregating both geometric occupancy and semantic information probabilistically. Open-vocabulary semantic scores are computed for each occupied voxel via similarity with text embeddings from the user's query.

Geometric and Semantic Fidelity

A central technical contribution is the geometrically anchored update strategy for the Gaussian map, which resolves ambiguities inherent in photometric losses of conventional 3DGS (where multiple Gaussian parameterizations can explain the same visual data). By constraining Gaussian centers to SLAM point positions and aligning ellipsoidal extents to sensor rays, the system efficiently converges to geometrically plausible solutions, thereby improving downstream occupancy predictions.

For semantic association, lifting vision-language embeddings from images to 3D space ensures the system is not confined to the closed sets inherent in supervised segmentation and enables robust, open-vocabulary reasoning for embodied perception.

Experimental Evaluation

Datasets and Benchmarks

FreeOcc is evaluated on both the established EmbodiedOcc-ScanNet benchmark and the new ReplicaOcc benchmark, the latter introduced to test cross-dataset generalization in open-vocabulary settings.

• EmbodiedOcc-ScanNet: Standard for semantic occupancy, heavily used for both training and evaluation in supervised/self-supervised pipelines.
• ReplicaOcc: A test-only dataset derived from Replica, featuring 44 semantic categories per scene for open-vocabulary evaluation, specifically designed to test zero-shot generalization.

Quantitative Results: Generalization and Robustness

• On EmbodiedOcc-ScanNet, FreeOcc (monocular/RGB-D) outperforms self-supervised GaussianOcc and GaussTR by a margin of more than 2x in IoU and mIoU, despite not using any task-specific training or pose supervision. For instance, FreeOcc achieves IoU/mIoU of 31.29/13.86 (mono) and 34.40/15.84 (RGB-D), while baselines attain 10.17/4.34 or 15.63/4.95 respectively.
• On ReplicaOcc (zero-shot transfer), FreeOcc exhibits a significant leap: IoU/mIoU of 46.81/16.93 (mono) and 55.65/20.90 (RGB-D). Competing methods collapse in this regime (IoU/mIoU near 0.00), underscoring both the poor generalization of supervised/self-supervised approaches and the robustness of FreeOcc's training-free methodology.
• Ablation studies confirm that the geometrically consistent update and geometry-aware initialization drive substantial improvements, both in accuracy and computational efficiency.

Qualitative and Open-Vocabulary Results

• FreeOcc recovers coherent, detailed occupancy maps with correct semantic localization, even in cases where supervised ground-truth labels do not align with true object boundaries (e.g., correctly identifying a window misclassified as a wall in ground truth).
• The system supports open-vocabulary 3D queries, retrieving semantically diverse objects (e.g., "basket", "indoor-plant", "clock") in both synthetic and real-world environments.

System Flexibility

FreeOcc's modular framework supports SWAP-in of stronger SLAM backbones or alternative VLMs, enhancing geometry or semantics, respectively, without end-to-end retraining. Monocular and RGB-D sensing modes are both supported. Real-world deployment experiments (e.g., handheld RGB-D mapping with online Qwen3-VL cues) validate the feasibility of streaming, annotation-free, 3D open-vocabulary reasoning.

Limitations

• SLAM sensitivity: Mapping fidelity remains dependent on SLAM performance; geometric drift and sensor failures degrade occupancy map quality.
• Semantic consistency: Variability in VLM outputs causes temporal inconsistency in per-frame semantic associations, leading to noisy 3D semantic features.
• Fine Object Resolution: Grid resolution limits fidelity when mapping small objects, suggesting adaptive grid representations as a future avenue.
• Semantic mIoU Gap: On fixed-label datasets, mIoU is lower compared to fully supervised methods, mainly due to semantic misalignment and absent supervision.

Implications and Future Directions

FreeOcc eliminates the need for costly dataset-specific annotations and retraining, providing an immediately deployable solution for robotics, AR/VR, and spatial computing applications in unseen environments. Its open-vocabulary semantics and training-free geometry pipeline signify a decoupling of embodied 3D reasoning from inflexible, closed-set taxonomies and overfitted cross-domain mappings.

Extending FreeOcc with semantic and geometric feedback into the SLAM optimization loop, integrating confidence-weighted temporal aggregation, and exploring adaptive, hierarchical occupancy grids may further boost robustness and expressivity. The system's ability to leverage external advancements (better SLAM, improved VLMs) without structural changes is particularly advantageous for scalability in the ever-evolving ecosystem of visual and spatial foundation models.

Conclusion

FreeOcc (2604.28115) establishes a new state-of-the-art paradigm for embodied 3D open-vocabulary occupancy prediction, offering strong geometric and semantic accuracy in a training-free, annotation-free, and generalizable fashion. It addresses core limitations of learning-based approaches, enabling practical embodied agents to perform robust, language-driven spatial reasoning in both simulated and uncontrolled real-world settings. This framework signals a shift towards deployable, foundation model-aligned 3D scene understanding, paving the way for more adaptable and scalable embodied AI systems.