M3DMap: Multimodal 3D Mapping Framework

Updated 4 July 2026

M3DMap is a framework that constructs dynamic, object-aware 3D maps by integrating images, point clouds, and text data.
It employs a modular design with key components for segmentation/tracking, odometry, map updating, and multimodal retrieval to support varied robotics tasks.
The system distinguishes between accumulated and instantaneous maps to capture both persistent scene structure and time-specific dynamics.

Searching arXiv for the primary paper and closely related multimodal 3D mapping work cited in the provided material. M3DMap is a modular framework for object-aware multimodal 3D mapping in static and dynamic environments. It is introduced as a response to the absence of a universal representation for dynamic 3D scenes that incorporates multimodal data such as images, point clouds, and text, and it is positioned simultaneously as a taxonomy of recent multimodal 3D mapping methods and as an original method for constructing multimodal 3D maps that support practical tasks ranging from 3D object grounding to mobile manipulation (Yudin, 23 Aug 2025).

1. Definition and scope

M3DMap, expanded as Multimodal 3D Mapping, is presented as a framework for constructing 3D maps that encode not only geometry but also objects, semantics, multimodal features, and queryability. Its central concern is dynamic environments, where scene structure cannot be treated as a single immutable geometric entity. The paper distinguishes dynamic settings by including objects moving inside the robot’s field of view, objects moving outside the field of view, and interactive objects changing state. This distinction motivates a map design that stores both persistent scene structure and time-specific scene states (Yudin, 23 Aug 2025).

The framework is explicitly multimodal. Images provide appearance and semantics; point clouds or LiDAR provide reliable 3D geometry and spatial structure; and text supports open-vocabulary querying, scene grounding, reasoning, and integration with LLMs and VLMs. A central premise is that no fixed category inventory is sufficient for practical robotics, so multimodal maps should align 3D entities with language and visual features rather than only with closed-set labels (Yudin, 23 Aug 2025).

The paper assigns M3DMap a dual role. First, it proposes a taxonomy that classifies multimodal 3D mapping methods by scene type, representation, learning method, and application. Second, it introduces an original modular method consisting of four interconnected components: a neural multimodal object segmentation and tracking module, an odometry estimation module, a module for 3D map construction and updating with various implementations depending on the desired scene representation, and a multimodal data retrieval module (Yudin, 23 Aug 2025).

2. Formal problem formulation

The framework assumes one or more RGB-D or stereo cameras producing sequences of images $I$ and depth maps $D$ , one or more LiDARs producing point cloud sequences $PCL$ , optional GNSS+RTK pose sequences $Pose$ , optional IMU-derived velocity $Vel$ and acceleration $Acc$ , and multimodal queries $Q_{Multi}$ from a configuration terminal, including text, images, or point clouds (Yudin, 23 Aug 2025).

Its global output is divided into an accumulated map and instantaneous maps: $(Map_A, Map_I)=\mathbb{M}(I,D,PCL,[Pose],[Vel],[Acc],[Q_{Multi}]),$ where $Map_A$ is the accumulated map at step $N$ , while $D$ 0 stores instantaneous maps at selected time steps. This distinction is central to the treatment of dynamic scenes, because the accumulated map alone does not capture current object state or short-term scene changes (Yudin, 23 Aug 2025).

The architecture is further decomposed into four module-level functions: $D$ 1

$D$ 2

$D$ 3

$D$ 4

Here, $D$ 5 denotes the set of objects, $D$ 6 is the estimated pose produced by the odometry module, and $D$ 7 is the result of retrieval from the multimodal map. The paper does not provide a full probabilistic state-space model, explicit object-state equations, or a unified coordinate-frame derivation beyond these module abstractions. This suggests that M3DMap is intended as a modular systems blueprint rather than a single monolithic estimator (Yudin, 23 Aug 2025).

3. Taxonomy of multimodal 3D mapping methods

The taxonomy proposed in the paper organizes the field along four axes. The first is scene type, separating static scenes from dynamic scenes. Dynamic scenes are explicitly defined to include moving objects within the field of view, moving objects outside the field of view, and interactive objects with changing state (Yudin, 23 Aug 2025).

The second axis is scene or map representation. Dense representations include Gaussian splatting-based, NeRF-based, SDF-based, voxel-based, point-based, point-voxel-based, and neural implicit representations. Sparse representations include knowledge graphs, object-based maps, object-based maps with hierarchy, ontology-based maps, and sparse place-based memory. The paper notes that object-based methods are graph-like but omit explicit relational edges (Yudin, 23 Aug 2025).

The third axis is learning method, divided into approaches without learning, modular approaches, and end-to-end methods. End-to-end is defined as single neural models where gradients can flow from the output back to raw input data during training. The paper observes that dense dynamic methods are dominated by end-to-end systems, whereas M3DMap is explicitly modular (Yudin, 23 Aug 2025).

The fourth axis is application domain, comprising seven task categories: 3D Object Grounding, Vehicle Localization and Mapping, 4D Scene Reconstruction, State Prediction, High-Level Action Planning, Low-Level Vehicle Control, and Question Answering (Yudin, 23 Aug 2025).

This taxonomy is used to position M3DMap as a system spanning static and dynamic scenes, dense and sparse constructions depending on implementation, modular learning-based design, and multiple downstream applications. The paper therefore treats M3DMap not as one fixed representation but as a framework capable of instantiating several representation families, including point cloud or voxel maps, scene graphs, NeRF-based maps, Gaussian splatting-based maps, and sparse place-based maps (Yudin, 23 Aug 2025).

4. Modular architecture

The architecture is organized around four primary modules: object segmentation and tracking, odometry estimation, 3D map construction and update, and multimodal retrieval. Information flows from sensors to recognition and odometry, then to mapping, and finally to retrieval over the produced maps (Yudin, 23 Aug 2025).

Object segmentation and tracking

The recognition stage is subdivided into image-based recognition, point-cloud-based recognition, fusion, and tracking. Image-based recognition is written as

$D$ 8

and may operate with a fixed label set, an open dictionary, or visual queries. The paper states that outputs can be 2D bounding boxes, 3D bounding boxes, or masks, and may also include CLIP features, DINOv2 features, and textual descriptions (Yudin, 23 Aug 2025).

Point-cloud-based recognition is written as

$D$ 9

and can use the estimated pose $PCL$ 0 to compact or densify point clouds over time. The outputs can include 3D bounding boxes, 3D masks, and 3D object descriptors from encoders such as Uni3D and MinkLoc3Dv2 (Yudin, 23 Aug 2025).

Fusion is expressed as

$PCL$ 1

with camera–LiDAR calibration supplying the geometric link between modalities. Tracking then assigns persistent identities: $PCL$ 2 The paper states that association uses algorithms based on the Hungarian algorithm, and that for 3D tracking the cost includes object overlap while accounting for the robot’s 6DoF motion via $PCL$ 3. No single fixed network architecture or explicit tracking loss is prescribed; instead, M3DMap exposes a modular slot for these choices (Yudin, 23 Aug 2025).

Odometry estimation

Odometry is defined as

$PCL$ 4

Two implementations are emphasized for dynamic environments. The first performs localization in prior LiDAR maps while removing points belonging to dynamic objects. The second combines GNSS+RTK navigation data with LiDAR odometry, including trainable fusion via a modified UKF and neural Kalman filter adaptation (Yudin, 23 Aug 2025).

This design reflects the paper’s claim that dynamic objects should be filtered out before localization when possible, and that sensor-fusion odometry is necessary in settings where either GNSS or LiDAR alone becomes unreliable. The paper does not provide a unified optimization objective for the odometry module, which again reinforces the interpretation of M3DMap as a framework that aggregates several implementation styles (Yudin, 23 Aug 2025).

Map construction and updating

Map construction is defined as

$PCL$ 5

The module supports multiple representation families depending on task requirements. The paper explicitly lists point cloud or voxel-based maps, scene graph-based maps, NeRF-based maps, Gaussian splatting-based maps, and sparse place-based maps (Yudin, 23 Aug 2025).

The decisive mechanism for dynamic environments is the coexistence of the accumulated map $PCL$ 6 and the instantaneous maps $PCL$ 7. Static scene structure can be fused into $PCL$ 8, while dynamic objects and time-dependent configurations are represented by up-to-date $PCL$ 9 at selected time steps. The paper does not specify a universal occupancy update rule, TSDF equation, rendering loss, or graph-construction formula for this module; its level of description remains architectural (Yudin, 23 Aug 2025).

Multimodal retrieval

Retrieval is written as

$Pose$ 0

The paper states that retrieval may return submaps, target objects, places, and more general task-relevant data in response to multimodal queries. Two implementations are discussed: encoder-based retrieval over images, point clouds, semantic masks, and text; and LLM-based retrieval over text representations of scene graphs (Yudin, 23 Aug 2025).

This module is what turns the map into a usable memory structure rather than only a reconstruction product. In that respect, M3DMap is close in spirit to systems that treat 3D maps as queryable memory, including feature-bank representations such as "M3: 3D-Spatial MultiModal Memory" (Zou et al., 20 Mar 2025), although M3DMap is broader in its explicit treatment of dynamics, tracking, and odometry (Yudin, 23 Aug 2025).

5. Theoretical propositions and empirical evidence

A distinctive part of the paper is its theory section, which argues that multimodal fusion and the combination of raw data with encoded features can improve recognition quality. Let

$Pose$ 1

and let a fused model produce

$Pose$ 2

Recognition quality is defined as

$Pose$ 3

with threshold $Pose$ 4, and the paper derives conditions under which

$Pose$ 5

It states a theorem asserting the existence of recognition models for which the fused output has better recognition quality than either unimodal output, and corollaries extending the claim to multimodal fusion and to the combination of raw sensory input with encoded features such as foundation-model embeddings (Yudin, 23 Aug 2025).

Two fusion forms are analyzed. In the linear case,

$Pose$ 6

a sufficient condition is

$Pose$ 7

In the attention-like case,

$Pose$ 8

the paper derives a sufficient condition denoted $Pose$ 9, involving a lower bound on $Vel$ 0 and positive correlation between the attention-like term and $Vel$ 1. The stated implication is that both MLP-style fusion and attention or cross-attention are theoretically compatible with the paper’s multimodal performance claims (Yudin, 23 Aug 2025).

Empirically, the paper assembles module-level evidence rather than a single end-to-end benchmark. In segmentation and grounding, prompt-tuned open-vocabulary systems improve over original prompts, and zero-shot 3D semantic segmentation quality increases from $Vel$ 2 for ConceptFusion and $Vel$ 3 for ConceptGraphs to $Vel$ 4 for BBQ-CLIP on ScanNet (Yudin, 23 Aug 2025). In odometry, the best Husky prior-map localization configuration improves translation error from $Vel$ 5 m for basic LMR to $Vel$ 6 m with odometry, multiframe processing, and masking, while rotation error improves from $Vel$ 7 deg to $Vel$ 8 deg (Yudin, 23 Aug 2025). In semantic occupancy mapping on Semantic KITTI, obstacle classification precision increases from $Vel$ 9 for raw point-cloud occupancy mapping to $Acc$ 0 when semantic labels are incorporated (Yudin, 23 Aug 2025).

The paper also cites strong scene-graph results, including ScanRefer $Acc$ 1 for 3DGraphLLM with Llama3-8B and ScanRefer $Acc$ 2 for GPT4Scene-HDM (Yudin, 23 Aug 2025). For place recognition on Oxford RobotCar, MSSPlace-LIST, which uses image, semantic label map, LiDAR, and text, achieves $Acc$ 3 and $Acc$ 4, which the paper uses as evidence that more modalities improve retrieval quality (Yudin, 23 Aug 2025). In future map prediction, adding trajectory encoding improves observed and occluded AUC as well as flow EPE, supporting the use of tracklets in dynamic-scene forecasting (Yudin, 23 Aug 2025). In manipulation and planning, scene-graph-based variants sharply outperform baselines without scene graphs, with one reported jump from $Acc$ 5 to $Acc$ 6 in the LAMDEN elevator environment when a scene graph and PCA are used (Yudin, 23 Aug 2025).

A plausible implication is that M3DMap’s modularity is meant to preserve access to these distinct empirical advantages rather than to force all tasks into a single representation. This also explains why the framework is framed as representation-flexible rather than as a single dense or sparse map type.

6. Relationship to adjacent research, limitations, and significance

M3DMap is situated relative to several adjacent research directions. Compared with dense reconstruction-centric methods such as Gaussian-splatting or NeRF-based semantic scene systems, M3DMap is broader because it integrates segmentation, tracking, odometry, mapping, and retrieval in one modular architecture (Yudin, 23 Aug 2025). Compared with sparse scene-graph systems, it retains support for localization and alternative dense representations. Compared with end-to-end autonomous driving planners, it is more representation-flexible and less tied to a single downstream task (Yudin, 23 Aug 2025).

The framework also overlaps with several systems that emphasize one of its components. "CogniMap3D" provides persistent static-scene memory, revisit handling, and dynamic-scene filtering in monocular mapping (Wang et al., 13 Jan 2026), while "MAP-ADAPT" shows how semantic and geometric criteria can drive quality-adaptive 3D mapping in a unified TSDF representation (Zheng et al., 2024). "MM3DGS SLAM" illustrates dense multimodal SLAM using vision, depth, and inertial sensing with a 3D Gaussian map (Sun et al., 2024), and "M3: 3D-Spatial MultiModal Memory" develops a scene-level multimodal memory over 3D Gaussian Splatting using Principal Scene Components and Gaussian Memory Attention (Zou et al., 20 Mar 2025). M3DMap’s novelty, in the terms of its own paper, lies in attempting to unify such capabilities under an object-aware modular framework for dynamic environments rather than in defining a single new map primitive (Yudin, 23 Aug 2025).

The paper explicitly identifies limitations. It notes the need to synchronize all modules and the difficulty of achieving real-time operation (Yudin, 23 Aug 2025). It also highlights broader field-level bottlenecks: the lack of unified datasets and benchmarks that jointly cover recognition, tracking, grounding, reasoning, and planning; the scarcity of annotated dynamic indoor scene datasets; and the absence of universal multimodal dynamic 3D scene representations (Yudin, 23 Aug 2025). These remarks are consistent with the framework’s status as a blueprint that aggregates proven ingredients but does not yet collapse them into a universally benchmarked end-to-end system.

In significance, M3DMap is best understood as a systems-level proposal for treating 3D mapping as an object-centric perception–memory–retrieval problem rather than as geometry reconstruction alone. Its key conceptual move is the explicit combination of multimodal recognition, dynamic-scene-aware odometry, flexible map construction, and multimodal retrieval under a formal split between accumulated and instantaneous maps. This suggests a research direction in which 3D maps become persistent, queryable, and task-oriented multimodal structures for robotics and autonomous transportation rather than static geometric models (Yudin, 23 Aug 2025).