Multimodal Mars Synthesis (M3arsSynth)

• The paper introduces M3arsSynth, an automated pipeline leveraging stereo rover imagery and advanced vision models to synthesize accurate 3D Martian environments.
• It employs robust techniques including depth estimation, camera pose recovery, and 3D Gaussian splatting to ensure photorealistic rendering and metric consistency.
• The system generates a diverse dataset of over 10,000 Martian locations, supporting applications in planetary exploration, navigation, and synthetic video generation.

Multimodal Mars Synthesis (M3arsSynth) is an automated pipeline for reconstructing and synthesizing photorealistic, metrically accurate 3D Martian environments. Leveraging stereo imagery from Mars rover missions, M3arsSynth constructs a large-scale, multimodal video dataset of over 10,000 distinct Martian locations, each presented with precise geometric, visual, and semantic information. The resulting data form the backbone for MarsGen, a Martian terrain video generator capable of synthesizing visually realistic and geometrically consistent videos conditioned on user input such as initial images, camera trajectories, or textual prompts (Li et al., 10 Jul 2025).

1. Data Acquisition and Preprocessing

M3arsSynth ingests raw stereo-navigation images sourced from the NASA Planetary Data System (PDS). Primary data sources include left/right "Navcam" pairs from the Curiosity and Perseverance rovers, but the pipeline accommodates any PDS-archived stereo pairs, including those from mast-cam and hazard-avoidance cameras.

A highly-structured automatic curation process mitigates the lack of PDS calibration metadata and the prevalence of noisy, redundant, corrupted, or thumbnail imagery:

1. Discard images below resolution or size thresholds, and reject frames with low RGB variance.
2. Remove near-duplicate images using perceptual hashes and Hamming distance thresholds to ensure semantic diversity.
3. Eliminate frames with blurred scene content, measured by Laplacian variance below a sharpness cutoff.
4. Filter images showing strong color saturation, flat histograms, or large uniform patches—frequent artifacts from occlusions or sensor errors.

Residual edge cases (e.g., rover mast shadows, wheel intrusions, specular highlights) are handled with a semi-automatic process. Grounded-SAM generates segmentation masks based on prompts such as "rover wheel," which human annotators review, yielding a terrain-only corpus for downstream reconstruction.

2. Depth Estimation and Camera Pose Recovery

M3arsSynth replaces classical block-matching stereo with recent vision foundation models for intrinsic calibration, metric depth estimation, and relative pose calculation:

• Intrinsic Calibration (VGGT): Each view is processed by the Visual Geometry Grounded Transformer (VGGT), which infers the $3 \times 3$ camera intrinsic matrix

$$K = \begin{pmatrix} f_x & 0 & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{pmatrix}.$$

• Dense Metric Depth (Metric3D v2): Using Metric3D v2, a dense per-pixel depth map $d(u,v)$ is predicted. Pixels are back-projected into $\mathbb{R}^3$ camera space:

$P_c(u,v) = d(u,v)\;K^{-1}[u,v,1]^\top.$

• Relative Pose (PnP with GIM): Sparse 2D correspondences are detected between stereo pairs with the Generalizable Image Matcher (GIM). The relative rotation $R_{\rm rel}$ and translation $\mathbf{t}_{\rm rel}$ are solved by minimizing the standard reprojection error:

$$\min_{R_{\rm rel},\,\mathbf{t}_{\rm rel}} \sum_i \bigl\| \mathbf{p}_{2,i} - \pi(K\,[R_{\rm rel}\mid\mathbf{t}_{\rm rel}]\,\mathbf{P}_{1,i}) \bigr\|_2^2,$$

with $\pi$ for perspective division.

• Depth-Scale Alignment: To ensure metrical consistency, predicted depths at matched pixels are globally rescaled by solving

$$\min_{s,b} \sum_j \bigl(s\,d_{1,j} + b - d_{2,j}\bigr)^2$$

and updating $$K = \begin{pmatrix} f_x & 0 & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{pmatrix}.$$0.

This integrated process yields dense, metrically consistent point clouds and relative poses for each stereo acquisition.

3. 3D Scene Reconstruction via Gaussian Splatting

Rather than meshing, M3arsSynth employs 3D Gaussian Splatting (3DGS) for continuous scene modeling:

• Primitive Initialization: Each back-projected pixel forms the center of a 3D Gaussian primitive; scale is set by nearest-neighbor depth.
• Photometric + Depth Regularization: Primitive positions, scales, and colors are optimized by minimizing

$$K = \begin{pmatrix} f_x & 0 & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{pmatrix}.$$1

where

$$K = \begin{pmatrix} f_x & 0 & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{pmatrix}.$$2

• Bilateral-Grid Appearance Transform: To counteract view-dependent appearance variation (e.g., camera ISP, solar angle), a learnable per-view bilateral grid (dimensions $$K = \begin{pmatrix} f_x & 0 & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{pmatrix}.$$3; each cell encodes a $$K = \begin{pmatrix} f_x & 0 & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{pmatrix}.$$4 affine transform) is optimized under total-variation regularization. Rendered colors pass through this grid prior to photometric loss computation.

This approach yields highly detailed, photorealistic 3D reconstructions suitable for novel view synthesis.

4. Rendering and Multimodal Output Generation

The optimized 3DGS representation supports rendering sequences for arbitrary, "depth-adaptive" camera trajectories:

$$K = \begin{pmatrix} f_x & 0 & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{pmatrix}.$$5

where near-field regions are contracted and far-field are expanded to ensure uniform information density. For each camera pose $$K = \begin{pmatrix} f_x & 0 & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{pmatrix}.$$6,

• RGB frames are rendered by splatting Gaussians and applying the bilateral grid.
• Per-frame metric depth maps are generated from the corresponding virtual view.
• Per-pixel normal maps are computed via finite differencing or analytic derivative of Gaussian splats.

The scene representation thus directly supports generation of multiple synchronized modalities per frame without explicit UV unwrapping.

5. Dataset Composition and Evaluation

M3arsSynth outputs a large-scale, diverse dataset:

Property	Value/Description
Distinct Scenes	>10,000 Martian locations (metric-scale reconstruction)
Frames per Scene	40–100 (at original Navcam resolution)
Trajectory Types	Orbit, lateral pan, rising/descending
Modalities per Frame	RGB, depth, normal map, 6-DoF pose, text description

Additionally, each rendered frame is paired with a natural-language caption generated by ChatGPT-4o, conditioned on the imagery and terrain class.

Evaluation Metrics:

• 3D model physical accuracy is evaluated using reprojection error during reconstruction.
• Generated video consistency is measured by 2D Warp Error:
  • Self-reprojection loss for frame $$K = \begin{pmatrix} f_x & 0 & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{pmatrix}.$$7:

  $$K = \begin{pmatrix} f_x & 0 & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{pmatrix}.$$8 - Cross-view reprojection between frames $$K = \begin{pmatrix} f_x & 0 & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{pmatrix}.$$9 and $d(u,v)$0:

  $d(u,v)$1 - Overall 2D Warp Error:

  $d(u,v)$2

• Visual fidelity is measured by FID and FVD.
• Novel-view accuracy uses PSNR, SSIM, and LPIPS.
• MarsGen models trained on the M3arsSynth dataset outperform Earth-trained video synthesis baselines across these metrics.

6. Significance and Applications

M3arsSynth provides an automated, scalable solution to the challenge of physically accurate Martian scene synthesis, significantly exceeding the domain coverage, spatial fidelity, and modality breadth of prior Mars scene datasets. By generating metric-scale 3D reconstructions and supporting a rich set of rendering modalities—including per-view RGB, depth, surface normals, and text captions—M3arsSynth underpins advanced Martian world models and robust controllable video generation for both scientific and engineering applications. The approach directly addresses the data scarcity and domain gap between Martian and terrestrial datasets, providing a foundation for planetary navigation, mission rehearsal, geomorphological analysis, and synthetic data-driven machine learning pipelines for extraterrestrial robotic systems (Li et al., 10 Jul 2025).