Matterport3D Simulator

• Matterport3D Simulator is a simulation platform that replicates real-world indoor spaces using high-resolution RGB-D imagery and 3D mesh reconstructions.
• It builds a deterministic navigation graph from panoramic viewpoints, enabling embodied vision-and-language tasks such as room-to-room navigation.
• The simulator integrates reinforcement learning with discrete action spaces, precise reward shaping, and support for transfer to physical robots.

The Matterport3D Simulator is a large-scale reinforcement learning environment providing visually and geometrically accurate agent interactions in real-world indoor spaces. Designed to facilitate embodied vision-based AI research, it leverages true RGB-D imagery and high-resolution 3D meshes from the Matterport3D dataset, supporting language-guided navigation tasks and benchmarks. The simulator is a foundation for embodied vision-and-language tasks and is closely associated with the Room-to-Room (R2R) visually-grounded navigation benchmark (Anderson et al., 2017).

1. Environment Construction and Data Foundations

The simulator is constructed over 90 actual building-scale indoor scenes sampled from the Matterport3D dataset, including houses, offices, hotels, and churches. This dataset consists of 194,400 RGB-D frames, organized into 10,800 panoramic viewpoints. Panoramas uniformly sample the walkable floorplan at approximately 2.25 m intervals, each stored as a cube-map (comprising 18 perspective images) with calibrated 6 DoF pose. High-resolution textured meshes, enriched with room and object instance segmentations, underlie all rendering and agent navigation (Anderson et al., 2017).

A navigation graph $G = (V, E)$ is constructed with vertices $V$ as all panoramic viewpoints. An undirected edge $⟨v_i, v_j⟩∈E$ exists if a straight-line raytrace in the mesh between $v_i$ and $v_j$ is unobstructed and $‖v_i - v_j‖_2 ≤ 5\,\mathrm{m}$. Manual postprocessing removes edges through artifacts such as windows and mirrors.

Agent observations at each state are generated by projecting the precomputed cube-map into a configurable pinhole camera model. Future extensions are planned to expose depth and semantic segmentation channels directly from mesh labels.

2. State Representation and Observations

The simulator implements a partially observable Markov decision process (POMDP) formalism. The system state at time $t$ is defined as

$s_t = ⟨v_t, ψ_t, θ_t⟩ ∈ S,$

where $v_t∈V$ indexes the agent's current 3D panorama (vertex), $ψ_t ∈ [0, 2π)$ is the heading (yaw), and $θ_t ∈ [−π/2, +π/2]$ is the elevation (pitch).

Agent observations are

• $o_t = \mathrm{RGB}(o_t)$, the pinhole projected 3-channel image,
• Optionally, $o_t^D$ depth observations (not enabled by default).

Feature extraction is typically performed via mean-pooled ResNet-152 embeddings, yielding $\varphi(o_t)\in ℝ^d$ (Anderson et al., 2017).

3. Action Space and Transition Dynamics

The action space is both discrete and state-dependent. At each step:

• $$A(s_t) = W_{t+1} \cup \{\mathrm{look~left}, \mathrm{look~right}, \mathrm{look~up}, \mathrm{look~down}\} \cup \{\mathrm{STOP}\}$$,
• $W_{t+1} \subseteq V$ contains panoramic vertices visible and reachable given the agent's 3D camera frustum and mesh connectivity (preventing nonphysical transitions).

Formally,

$$W_{t+1} = \{v_t\} \cup \{v_i ∈ V \mid ⟨v_t, v_i⟩∈E \wedge v_i ∈ P_t\}$$

where $P_t$ is the camera frustum (with margin). This prevents teleporting through walls or ceilings.

Transitions are deterministic:

$$a_t = (\mathrm{move~to}~v_i,~\Deltaψ,~\Deltaθ)\implies s_{t+1} = \langle v_i,\,ψ_t + \Deltaψ,\,θ_t + \Deltaθ \rangle.$$

The reference seq-to-seq agent employs a reduced discrete action set: {forward, left, right, up, down, stop}, with "forward" mapped to the most central reachable viewpoint (projected image center alignment). Rotation actions are fixed at ±30° in yaw or pitch.

4. Reinforcement Learning Formulation and Metrics

The navigation task is formalized as an MDP $\langle S, A, P, R, \gamma\rangle$:

• $P$ is the deterministic transition defined by the navigation graph and viewpoint transformations.
• A typical discount factor is $\gamma≈0.99$.
• A commonly used shaped reward:

$$r(s_t, a_t) = \mathrm{dist}_G(v_t, v^*) - \mathrm{dist}_G(v_{t+1}, v^*),$$

where $v^*$ is the goal node, and $\mathrm{dist}_G$ is the shortest-path distance in $G$, rewarding approach towards the target.

Success criterion and oracle metrics for Room-to-Room evaluation:

Metric	Definition
Path length	$\sum_{t=0}^{T-1}‖v_t−v_{t+1}‖_2$
Navigation error	$e = \mathrm{dist}_G(v_T, v^*)$
Success rate	$\%$ of episodes with $e < 3\,\mathrm{m}$ and agent chooses STOP
Oracle Success	$\%$ of episodes where $$\min_{t \leq T} \mathrm{dist}_G(v_t, v^*) < 3\,\mathrm{m}$$, STOP ignored

The trajectory length ratio is often reported: agent path length over the shortest optimal path (Anderson et al., 2017).

5. Room-to-Room (R2R) Benchmark Dataset

The Room-to-Room (R2R) dataset is the foundational vision-and-language navigation benchmark for the Matterport3D Simulator:

• 7,189 start-goal pairs sampled between different rooms, constrained to shortest paths ≥5 m (4–6 graph edges, mean ≈10 m).
• Each path is annotated with three free-form, crowd-sourced natural language navigation instructions using an AMT WebGL “fly-through” interface, yielding 21,567 instructions (average length: 29 words; vocabulary ≈3.1 k tokens).
• Data split: Train Seen (61 scenes, 14,025 instructions), Validation Seen (same 61 scenes, 1,020 instructions), Validation Unseen (11 scenes, 2,349 instructions), Test Unseen (18 scenes, 4,173 instructions—accessible only via evaluation server) (Anderson et al., 2017).

6. Simulator Implementation and Interfaces

The Matterport3D Simulator core is implemented in C++/OpenGL, with Python bindings designed for compatibility with major deep learning and RL frameworks (Caffe, TensorFlow, ParlAI, OpenAI Gym). Configuration is controlled via TOML/JSON or CLI flags, supporting options such as image resolution, field of view, observation modalities (depth, semantic segmentation), and navigation graph thresholds.

Typical Python workflow:

import matterport3d_simulator as M sim = M.Simulator(config_file="config.toml") scan_id, start_view_id = sim.reset(scene="17DRP5sb8fy", start_vertex=v0, start_yaw=ψ0) state = sim.get_state() for t in range(max_steps): # state.rgb is H×W×3 array; state.view_ids is list of reachable vertices action = agent.act(state.rgb, state.view_ids, language_instruction) if action == STOP: break state = sim.step(action) final_dist = sim.distance_to_goal()

Additional interface features include sim.render_map() for interactive WebGL data annotation and sim.get_action_space() to enumerate current move/rotation options (Anderson et al., 2017).

7. Extensions and Future Directions

Planned and possible extensions for the Matterport3D Simulator include:

• RGB-D and semantic/instance segmentation observations using mesh labels.
• Support for new embodied tasks, including navigation-instruction generation (inverse VLN), embodied Visual Question Answering (EQA), and question/answer dialog during agent movement.
• Real-to-sim transfer: fine-tuning policies on real Kinect scans for deployment on physical robots.
• Alternative control modalities: continuous agent control leveraging the full 3D mesh (parallel to the MINOS simulator), which supports continuous physics, multimodal sensors (surface normal, contact), and richer dynamics (Savva et al., 2017).
• Curriculum learning (with progressive subgoals), hierarchical RL, and multi-agent coordination (Anderson et al., 2017).

These extensibility points position the simulator as a versatile testbed for developing and benchmarking embodied agents in visually realistic indoor environments.