---
title: Matterport3D Simulator
url: https://www.emergentmind.com/topics/matterport3d-simulator
type: topic
---

# Matterport3D Simulator

The Matterport3D Simulator is a large-scale reinforcement learning environment providing visually and geometrically accurate agent interactions in real-world indoor spaces. Designed to facilitate embodied vision-based AI research, it leverages true RGB-D imagery and high-resolution 3D meshes from the Matterport3D dataset, supporting language-guided navigation tasks and benchmarks. The simulator is a foundation for embodied vision-and-language tasks and is closely associated with the Room-to-Room (R2R) visually-grounded navigation benchmark [1711.07280].

## 1. Environment Construction and Data Foundations

The simulator is constructed over 90 actual building-scale indoor scenes sampled from the Matterport3D dataset, including houses, offices, hotels, and churches. This dataset consists of 194,400 RGB-D frames, organized into 10,800 panoramic viewpoints. Panoramas uniformly sample the walkable floorplan at approximately 2.25 m intervals, each stored as a cube-map (comprising 18 perspective images) with calibrated 6 DoF pose. High-resolution textured meshes, enriched with room and object instance segmentations, underlie all rendering and agent navigation [1711.07280]. 

A navigation graph $G = (V, E)$ is constructed with vertices $V$ as all panoramic viewpoints. An undirected edge $⟨v_i, v_j⟩∈E$ exists if a straight-line raytrace in the mesh between $v_i$ and $v_j$ is unobstructed and $‖v_i - v_j‖_2 ≤ 5\,\mathrm{m}$. Manual postprocessing removes edges through artifacts such as windows and mirrors.

Agent observations at each state are generated by projecting the precomputed cube-map into a configurable pinhole camera model. Future extensions are planned to expose depth and semantic segmentation channels directly from mesh labels.

## 2. State Representation and Observations

The simulator implements a partially observable Markov decision process (POMDP) formalism. The system state at time $t$ is defined as
$$
s_t = ⟨v_t, ψ_t, θ_t⟩ ∈ S,
$$
where $v_t∈V$ indexes the agent's current 3D panorama (vertex), $ψ_t ∈ [0, 2π)$ is the heading (yaw), and $θ_t ∈ [−π/2, +π/2]$ is the elevation (pitch).

Agent observations are
- $o_t = \mathrm{RGB}(o_t)$, the pinhole projected 3-channel image,
- Optionally, $o_t^D$ depth observations (not enabled by default).

Feature extraction is typically performed via mean-pooled ResNet-152 embeddings, yielding $\varphi(o_t)\in ℝ^d$ [1711.07280].

## 3. Action Space and Transition Dynamics

The action space is both discrete and state-dependent. At each step:

- $A(s_t) = W_{t+1} \cup \{\mathrm{look~left}, \mathrm{look~right}, \mathrm{look~up}, \mathrm{look~down}\} \cup \{\mathrm{STOP}\}$,
- $W_{t+1} \subseteq V$ contains panoramic vertices visible and reachable given the agent's 3D camera frustum and mesh connectivity (preventing nonphysical transitions).

Formally,
$$
W_{t+1} = \{v_t\} \cup \{v_i ∈ V \mid ⟨v_t, v_i⟩∈E \wedge v_i ∈ P_t\}
$$
where $P_t$ is the camera frustum (with margin). This prevents teleporting through walls or ceilings.

Transitions are deterministic:
$$
a_t = (\mathrm{move~to}~v_i,~\Deltaψ,~\Deltaθ)\implies s_{t+1} = \langle v_i,\,ψ_t + \Deltaψ,\,θ_t + \Deltaθ \rangle.
$$

The reference seq-to-seq agent employs a reduced discrete action set: {forward, left, right, up, down, stop}, with "forward" mapped to the most central reachable viewpoint (projected image center alignment). Rotation actions are fixed at ±30° in yaw or pitch.

## 4. Reinforcement Learning Formulation and Metrics

The navigation task is formalized as an MDP $\langle S, A, P, R, \gamma\rangle$:

- $P$ is the deterministic transition defined by the navigation graph and viewpoint transformations.
- A typical discount factor is $\gamma≈0.99$.
- A commonly used shaped reward: 
$$
r(s_t, a_t) = \mathrm{dist}_G(v_t, v^*) - \mathrm{dist}_G(v_{t+1}, v^*),
$$
where $v^*$ is the goal node, and $\mathrm{dist}_G$ is the shortest-path distance in $G$, rewarding approach towards the target.

Success criterion and oracle metrics for Room-to-Room evaluation:

| Metric          | Definition                                                                                 |
|-----------------|-------------------------------------------------------------------------------------------|
| Path length     | $\sum_{t=0}^{T-1}‖v_t−v_{t+1}‖_2$                                                        |
| Navigation error| $e = \mathrm{dist}_G(v_T, v^*)$                                                          |
| Success rate    | $\%$ of episodes with $e < 3\,\mathrm{m}$ and agent chooses STOP                         |
| Oracle Success  | $\%$ of episodes where $\min_{t \leq T} \mathrm{dist}_G(v_t, v^*) < 3\,\mathrm{m}$, STOP ignored |

The trajectory length ratio is often reported: agent path length over the shortest optimal path [1711.07280].

## 5. Room-to-Room (R2R) Benchmark Dataset

The Room-to-Room (R2R) dataset is the foundational vision-and-language navigation benchmark for the Matterport3D Simulator:

- 7,189 start-goal pairs sampled between different rooms, constrained to shortest paths ≥5 m (4–6 graph edges, mean ≈10 m).
- Each path is annotated with three free-form, crowd-sourced natural language navigation instructions using an AMT WebGL “fly-through” interface, yielding 21,567 instructions (average length: 29 words; vocabulary ≈3.1 k tokens).
- Data split: Train Seen (61 scenes, 14,025 instructions), Validation Seen (same 61 scenes, 1,020 instructions), Validation Unseen (11 scenes, 2,349 instructions), Test Unseen (18 scenes, 4,173 instructions—accessible only via evaluation server) [1711.07280].

## 6. Simulator Implementation and Interfaces

The Matterport3D Simulator core is implemented in C++/OpenGL, with Python bindings designed for compatibility with major deep learning and RL frameworks (Caffe, TensorFlow, ParlAI, OpenAI Gym). Configuration is controlled via TOML/JSON or CLI flags, supporting options such as image resolution, field of view, observation modalities (depth, semantic segmentation), and navigation graph thresholds.

Typical Python workflow:

```python
import matterport3d_simulator as M
sim = M.Simulator(config_file="config.toml")
scan_id, start_view_id = sim.reset(scene="17DRP5sb8fy", start_vertex=v0, start_yaw=ψ0)
state = sim.get_state()
for t in range(max_steps):
    # state.rgb is H×W×3 array; state.view_ids is list of reachable vertices
    action = agent.act(state.rgb, state.view_ids, language_instruction)
    if action == STOP: break
    state = sim.step(action)
final_dist = sim.distance_to_goal()
```

Additional interface features include `sim.render_map()` for interactive WebGL data annotation and `sim.get_action_space()` to enumerate current move/rotation options [1711.07280].

## 7. Extensions and Future Directions

Planned and possible extensions for the Matterport3D Simulator include:

- RGB-D and semantic/instance segmentation observations using mesh labels.
- Support for new embodied tasks, including navigation-instruction generation (inverse VLN), embodied Visual Question Answering (EQA), and question/answer dialog during agent movement.
- Real-to-sim transfer: fine-tuning policies on real Kinect scans for deployment on physical robots.
- Alternative control modalities: continuous agent control leveraging the full 3D mesh (parallel to the MINOS simulator), which supports continuous physics, multimodal sensors (surface normal, contact), and richer dynamics [1712.03931].
- Curriculum learning (with progressive subgoals), hierarchical RL, and multi-agent coordination [1711.07280].

These extensibility points position the simulator as a versatile testbed for developing and benchmarking embodied agents in visually realistic indoor environments.

Source: https://www.emergentmind.com/topics/matterport3d-simulator