HM3D: Habitat–Matterport 3D Dataset

• HM3D is a large-scale 3D indoor dataset with 1,000 scenes, 10,600 rooms, and 112,500 m² of navigable space, encompassing diverse environments like residences, stores, and offices, designed for embodied AI research.
• It offers high reconstruction completeness, visualized through a below 5% defects rate in 560 of its 1,000 scenes, outperforming previous datasets vith FID vlaue of 20.5 against Gibson and MP3D
• The dataset, integrated with the Habitat simulator, includes over 10,000,000 scenarios for PointGoal Navigation. The agent's success and navigational skills achieve an SPL of 0.93 performance on Gibson and 0.92 success and SPL on HM3D
• question_q1_1: Which methods were used to ensure the geographic and architectural
• question_q2:
• question_q3:
• question_q4: Which strategies were suggested for
• question_q5:

HM3D, or Habitat–Matterport 3D, is a large-scale dataset of 1,000 textured, building-scale 3D reconstructions of real indoor environments for embodied AI. Its scenes include multi-floor residences, stores, offices, restaurants, workplaces, gyms, nightclubs, and art studios, and are distributed for direct use in the Habitat simulator. HM3D was introduced to address the limited scale, incomplete geometry, and visual unreality of earlier indoor 3D datasets, providing 112.5k m² of navigable space across more than 10,600 rooms and approximately 1,920 floors (Ramakrishnan et al., 2021).

1. Origin, objectives, and dataset composition

HM3D was collected by Matterport in collaboration with the Habitat team. Matterport users were explicitly asked to contribute scans for open-source academic and non-commercial embodied-AI research. All scans were acquired with a Matterport Pro2 tripod-based depth sensor. Matterport’s reconstruction pipeline performed RGB-D alignment, surface meshing, and texturing, producing environments represented primarily as textured 3D meshes rather than point clouds, floor plans, or panorama-only resources (Ramakrishnan et al., 2021).

The dataset was curated from a larger candidate pool in two stages. Fifteen volunteer annotators rated candidate scenes on a 1-to-5 scale according to holes and cracks, furnishing density and realism, closed doors, reconstruction quality, and “interactive potential.” Quantitative navigability information was also used to identify disconnected floors and inaccessible regions. Three expert annotators then selected the final scenes, prioritizing reconstruction quality while preserving diversity in floor count. Curation, annotation, and release required an estimated 800+ hours of human effort.

The released collection contains:

• 1,000 scenes;
• more than 10,600 rooms;
• approximately 1,920 building floors;
• 112.5k m² of navigable area;
• scenes from 38 countries and 181 geographic regions;
• locations in 43 U.S. states;
• buildings ranging from one to eight floors;
• buildings containing one to 93 rooms;
• floor areas from 49 m² to 2,172 m².

The environments are furnished real interiors rather than artist-authored synthetic scenes. Their diversity is intended to provide variation in layout, clutter, room connectivity, floor count, architecture, and visual appearance. The resulting resource is designed primarily for embodied navigation and static-scene understanding rather than unrestricted simulation of physical interaction.

2. Scale, navigability, and structural complexity

HM3D defines several geometric and navigational quantities. Floor area is the sum, over floors, of the area of the two-dimensional convex hull of navigable locations on each floor:

$$A_{\mathrm{floor}}=\sum_f \operatorname{Area}\left(\operatorname{ConvHull}(\mathcal{N}_f)\right).$$

Navigable area is the region accessible to a cylindrical robot with radius 0.1 m and height 1.5 m, computed using Habitat’s navigation mesh. It excludes unreachable regions, narrow passages, blocked spaces, and areas occupied by obstacles.

Navigation complexity is the maximum ratio of geodesic distance to Euclidean distance over pairs of navigable locations:

$$C_{\mathrm{nav}}=\max_{p,q\in\mathcal{N}} \frac{d_{\mathrm{geo}}(p,q)}{d_{\mathrm{Euc}}(p,q)}.$$

Scene clutter is the ratio between raw mesh area within 0.5 m of navigable regions and total navigable area:

$$C_{\mathrm{clutter}}= \frac{\text{raw mesh area within }0.5\text{ m of navigable regions}} {\text{navigable area}}.$$

The principal structural comparison is:

Dataset	Scenes	Navigable area	Navigation complexity
Replica	18	0.56k m²	5.99
MP3D	90	30.22k m²	17.09
Gibson	571	81.84k m²	14.25
ScanNet	1,613	10.52k m²	3.78
HM3D	1,000	112.50k m²	13.31

Relative to Gibson, HM3D has 1.7 times the floor area and 1.4 times the navigable area. Relative to the high-quality Gibson 4+ subset, it has approximately 20 times the floor area and 15.6 times the navigable area. HM3D’s navigation complexity is comparable to MP3D and Gibson, while exceeding that of room-scale datasets such as Replica, RoboTHOR, and ScanNet.

The dataset’s size was also associated with downstream scaling. Training PointGoal Navigation agents on 80, 400, and 800 HM3D training scenes produced performance that scaled nearly linearly with total navigable area, with a reported Pearson correlation of $\rho=0.88$ between training navigable area and navigation performance (Ramakrishnan et al., 2021). Correlations between navigable area and validation SPL ranged from 0.82 to 0.97 across train/validation combinations. By contrast, mean KID showed weak correlations with SPL, approximately from -0.30 to 0.1, and defect percentage was generally uncorrelated with SPL in simulation.

3. Reconstruction completeness and visual fidelity

HM3D was evaluated against reconstruction artifacts involving missing surfaces, holes, black cracks, and untextured regions. The evaluation densely samples navigable space using 1 m × 1 m cells. Four viewpoints are generated per cell with headings of 0°, 90°, 180°, and 270°, fixed azimuth, a 90° field of view, and 256 × 256 RGB-D rendering.

A rendered view is defective when either more than 5% of depth pixels have zero value or more than 5% of RGB pixels are completely black. The scene-level defect percentage is:

$$\%\text{defects} = 100\cdot \frac{\#\{\text{defective sampled views}\}} {\#\{\text{sampled views}\}}.$$

HM3D has 34–91% fewer reconstruction artifacts than prior photorealistic datasets, depending on the comparison dataset and measure. It contains 560 scenes with fewer than 5% defective views, compared with 175 such Gibson scenes. ScanNet contains more scans numerically but exhibits severe defects in almost all scenes. Replica has only 18 scenes, and some have approximately 80% defective views because roofs are missing.

Completeness matters for embodied learning because holes and absent surfaces can permit agents to see through walls or ceilings and exploit unrealistic simulator shortcuts. Such defects can produce discrepancies between simulated behavior and physical deployment.

Visual fidelity was evaluated by rendering RGB images from reconstructed meshes and comparing them with real-camera imagery from high-resolution 360° equirectangular panoramas in Gibson and MP3D. The study used Fréchet Inception Distance (FID) and Kernel Inception Distance (KID), for which lower values indicate closer distributions.

Dataset	FID vs Gibson real	KID ×10³ vs Gibson real	FID vs MP3D real	KID ×10³ vs MP3D real
Replica	34.9	15.8	42.8	19.3
MP3D	43.8	32.9	24.4	17.3
Gibson	39.3	29.8	38.0	25.5
ScanNet	126.7	106.0	121.3	97.4
HM3D	20.5	15.8	20.5	12.8

The paper reports 20–85% higher visual fidelity than prior datasets based on lower distributional distances. Cross-dataset distances between Gibson-real and MP3D-real images were themselves nonzero, with FID 11.2 and KID $6.2\times10^{-3}$, establishing a domain-shift floor for these comparisons.

4. Habitat integration, splits, and PointGoal Navigation

HM3D is distributed as a Habitat-ready resource containing meshes, textures, metadata, navigation information, and scene-loading support. The standard scene split contains:

$$800\ \text{training scenes},\qquad 100\ \text{validation scenes},\qquad 100\ \text{test scenes}.$$

The released PointNav episode organization contains 8.0 million training episodes, 2,500 validation episodes, and 2,500 test episodes. Training uses 10,000 episodes per training scene, while validation and test use 25 episodes per scene. Full Gibson training episodes were also regenerated, yielding 4.11 million Gibson training episodes.

In PointGoal Navigation, an agent receives a relative goal coordinate $(\Delta x,\Delta y)$ in addition to RGB or depth observations and GPS+Compass localization. The action set is:

$$\{\text{MOVE FORWARD},\text{TURN LEFT},\text{TURN RIGHT},\text{STOP}\}.$$

Forward movement is 0.25 m, turning is 10°, and an episode succeeds when the agent stops within 0.2 m of the goal. The main metrics are Success Rate and SPL:

$$\mathrm{SPL} = \frac{1}{N} \sum_{i=1}^{N} S_i \frac{L_i}{\max(P_i,L_i)},$$

where $$C_{\mathrm{nav}}=\max_{p,q\in\mathcal{N}} \frac{d_{\mathrm{geo}}(p,q)}{d_{\mathrm{Euc}}(p,q)}.$$0 indicates success, $$C_{\mathrm{nav}}=\max_{p,q\in\mathcal{N}} \frac{d_{\mathrm{geo}}(p,q)}{d_{\mathrm{Euc}}(p,q)}.$$1 is the shortest-path distance, and $$C_{\mathrm{nav}}=\max_{p,q\in\mathcal{N}} \frac{d_{\mathrm{geo}}(p,q)}{d_{\mathrm{Euc}}(p,q)}.$$2 is traveled path length.

The reported agent uses a ResNet-50 visual backbone, an MLP for GPS+Compass features, a two-layer LSTM with 512-dimensional hidden states, fully connected policy and value heads, stochastic action sampling, and 1.5 billion training frames. Training uses 256 parallel environments across eight nodes, with four GPUs per node and eight environments per worker.

Depth agents trained on HM3D achieve 1.00 Success and 0.93 SPL on Gibson, 0.94 Success and 0.83 SPL on MP3D, and 0.99 Success and 0.92 SPL on HM3D. RGB agents achieve 1.00/0.90 on Gibson, 0.85/0.71 on MP3D, and 0.98/0.87 on HM3D. HM3D-trained agents had not seen Gibson or MP3D scenes during training. The paper characterizes HM3D as “pareto optimal” in the sense that agents trained on HM3D achieve the best reported performance across Gibson, MP3D, and HM3D test domains; this is not a strict proof of Pareto-frontier optimality in a formal multi-objective optimization problem (Ramakrishnan et al., 2021).

5. HM3D-derived datasets and embodied-AI applications

HM3D has been reused as an environment source for tasks extending beyond PointNav.

Object-centric reconstruction

HM3D-ABO combines HM3D indoor scenes with Amazon-Berkeley Objects (ABO) assets. It places realistic household objects into realistic HM3D environments and provides multi-view RGB observations, camera poses, depth maps, object masks, and watertight object meshes (Yang et al., 2022).

The object meshes are converted to watertight geometry by rendering hundreds of depth maps, applying truncated signed-distance-field fusion, and extracting surfaces with marching cubes. The dataset contains 1,966 objects, 500 indoor scenes, and 3,196 scene-object configurations, with final RGB images at 640 × 480 resolution. Most objects belong to chair and table categories. Its benchmarks address absolute camera-pose estimation and few-view object reconstruction.

Vision-and-language navigation

HM3D-AutoVLN automatically converts 900 unlabeled HM3D buildings into vision-and-language navigation supervision (Chen et al., 2022). It constructs navigation graphs with 36,562 navigable nodes/viewpoints, approximately 172,000 pseudo-labeled 3D objects, and 217,703 object–instruction–trajectory triplets.

The pipeline uses Habitat, 2D panoptic segmentation with Mask2Former trained on ADE20K, RGB-D projection into 3D, 0.1 m voxelization, cross-view probability aggregation, GPT-2-based instruction generation, and shortest-path trajectories. Cross-view consistency increases manually assessed semantic-label correctness from 37.4% for direct 2D predictions to 58.3%.

Pretraining with HM3D-AutoVLN improves DUET on unseen REVERIE and SOON environments. SPL increases by 7.12 points on REVERIE validation-unseen and by 8.11 points on SOON validation-unseen. The results indicate that environmental diversity, rather than merely increasing trajectory count within a small set of buildings, is important for generalization.

Open-vocabulary ObjectNav

HM3D-OVON mines HM3DSem annotations to create open-vocabulary Object Goal Navigation episodes (Yokoyama et al., 2024). It uses 145 training scenes and 36 validation scenes, with no scene or object-instance overlap between training and evaluation. Goals are expressed as free-form object categories and are divided into seen, synonym, and hard-unseen categories using SentenceBERT similarity.

Each goal must occupy at least 5% of the Stretch robot’s camera view from at least one vantage point within 1 m. Episodes require a start position and a nearest-goal distance between 1 m and 30 m. The agent uses RGB, depth, relative odometry, and language goals, with a 500-step limit and a 1 m success radius.

The strongest detector-augmented baseline reported in the supplied material achieves 38.5% Success on seen categories, 39.0% on synonym categories, and 37.1% on hard-unseen categories. The benchmark demonstrates that HM3D can test semantic generalization in addition to geometric navigation.

Object navigation, memory, and question answering

HM3D has supported numerous zero-shot and memory-augmented ObjectNav systems. VoroNav uses a Reduced Voronoi Graph, semantic mapping, panoramic observations, BLIP captions, and GPT-3.5 reasoning; it reports 42.0 Success and 26.0 SPL on the HM3D benchmark (Wu et al., 2024). Later systems introduce episodic memory, adaptive exploration, probabilistic scene graphs, evidence accumulation, candidate-level interfaces, and multi-view target verification.

MemoryEQA constructs MT-HM3D, a benchmark of 1,587 question-answer pairs involving multiple objects and regions. Questions require comparison, relationship reasoning, counting, and attribute reasoning across HM3D observations (Zhai et al., 20 May 2025). MemoryEQA reports higher success and lower normalized exploration steps than planner-centric baselines, although the paper contains inconsistencies in scene counts and headline improvement values.

FARM uses Habitat-Sim to render posed RGB-D streams from HM3D and evaluates relational object retrieval with IRef-VLA queries (He et al., 13 Jun 2026). On 30 complex HM3D scenes, FARM reaches 7.9% Accuracy@1, 18.2% Recall@5, 26.9% Recall@10, and 12.5% mean reciprocal rank at visible-mask IoU threshold 0.10. The low absolute values reflect the difficulty of large HM3D scenes, where correct-object candidate recall is substantially lower than in ScanNet.

6. Limitations, governance, and research significance

HM3D’s geographic and architectural distribution is not globally representative. Its scenes were contributed by users who could afford a Matterport Pro2 sensor, had internet access for cloud upload, and lived in regions where the sensor was available. The dataset may therefore overrepresent particular economic, geographic, and architectural populations. Agents trained on HM3D may perform best in buildings resembling the sampled subset.

The original dataset is primarily static and geometric. It does not provide semantic object labels or physical interaction annotations in its initial release, nor does it model dynamic changes involving people, pets, or moved objects. Subsequent resources such as HM3DSem, HM3D-OVON, HM3D-AutoVLN, HM3D-MFMON, and HM3D-DYNA add semantic, language, sequential, multi-floor, or dynamic components, but each addresses only a subset of these limitations.

Privacy protection was implemented through consent from contributing Matterport users and inspection for faces, people, documents, diplomas, licenses, addresses, phone numbers, email addresses, notes, employer information, license plates, and street names. Identified information was labeled and blurred with pixel-wise masks before being used to recreate meshes. The reported process is commercially reasonable but is not described as a guarantee that all sensitive information is absent.

Licensing is also component-specific. HM3D is available free for academic, non-commercial research under Matterport’s academic-use terms and is subject to Matterport’s end-user license agreement. Derived resources may additionally involve the licenses of ABO, Habitat-Sim, Blender, Matterport3D, Gibson, ScanNet, Replica, or other associated asset packages.

The principal significance of HM3D is the joint provision of large physical scale, navigable building structure, reconstruction completeness, photorealistic appearance, and geographic and architectural diversity. Its use in PointNav, VLN, ObjectNav, open-vocabulary grounding, memory-centric EQA, object retrieval, and dynamic or multi-floor navigation has made it a common substrate for evaluating embodied agents under increasingly realistic spatial and semantic conditions. At the same time, HM3D remains a simulator-oriented representation of real spaces: it does not by itself establish physical-robot performance, complete semantic understanding, dynamic-world competence, or geographically unbiased embodied intelligence.