--- title: 'OpenMulti: Distributed Instance Mapping' url: https://www.emergentmind.com/topics/openmulti type: topic --- # OpenMulti: Distributed Instance Mapping OpenMulti is an open-vocabulary instance-level multi-agent distributed implicit mapping framework for multi-agent distributed collaborative mapping. It was introduced to address the limitation that existing approaches lack instance-level awareness and semantic understanding of environments, which constrains their usefulness for downstream applications. The framework combines Cross-Agent Instance Alignment with Cross Rendering Supervision, maintains object-level NeRFs annotated with rich open-vocabulary semantics, and supports instance-level retrieval tasks and semantic annotations for downstream applications. Experimental results report that OpenMulti outperforms related algorithms in both fine-grained geometric accuracy and zero-shot semantic accuracy [2509.01228]. ## 1. Problem setting and conceptual scope OpenMulti is presented as the first framework to enable open-vocabulary, instance-level, distributed implicit mapping among multiple agents. Its design targets collaborative mapping scenarios in which multiple agents autonomously collect observations, maintain distributed scene representations, and require consistent object-level semantics across the team. The central claim is that prior distributed mapping methods do not provide both instance-level awareness and open-vocabulary semantic understanding, whereas OpenMulti explicitly integrates both within a distributed implicit mapping pipeline [2509.01228]. The framework is organized around four stated contributions. First, it introduces **Cross-Agent Instance Alignment**, a module for consistent, confidence-based instance-level map alignment across multiple agents via an Instance Collaborative Graph. Second, it introduces **Cross Rendering Supervision**, a supervision strategy based on ray-based rendering consistency rather than only parameter sharing. Third, it constructs **instance-level, open-vocabulary mapping**, in which each agent maintains object-level NeRFs annotated with rich semantics through VLM-derived features. Fourth, it reports superior performance in geometry and zero-shot semantics, while also being more communication-efficient and robust to network failures. This suggests that OpenMulti is positioned not merely as a geometric mapping system, but as a distributed semantic scene representation framework oriented toward retrieval and downstream robot reasoning [2509.01228]. ## 2. Pipeline, data processing, and feature construction OpenMulti consists of three main stages: **Multi-Agent Data Collection & Preprocessing**, **Cross-Agent Instance Alignment**, and **Collaborative Optimization and Learning**. In the first stage, each agent autonomously gathers RGB/depth images and pose data, and performs open-vocabulary instance segmentation and feature extraction. The summary specifies CropFormer for instance segmentation, CLIP for visual features from mask crops, TAP for caption generation, and SBERT for caption encoding. Each instance is assigned a unique identifier, and the resulting visual and textual features are stored in a feature code book [2509.01228]. Within each agent, the system first performs **Cross-Frame Fusion**. Instance masks $\{m^{inst}_{k,t,m}\}$ are clustered temporally into unified object tracks $O_{k,i}$, with fused features $f_{k,i}^{clip}$ and $f_{k,i}^{caption}$. This stage converts per-frame detections into persistent object-level entities before inter-agent reconciliation begins. The use of both CLIP and caption embeddings is central to the later open-vocabulary retrieval mechanism, because each object is represented by both visual and textual descriptors rather than by a fixed closed-set category label. The preprocessing stage therefore establishes two properties required by later modules. The first is object persistence within an agent, achieved through temporal fusion. The second is multimodal semantic representation, achieved through the feature code book. A plausible implication is that OpenMulti treats the distributed map as a collection of instance-indexed neural fields enriched by external VLM feature spaces rather than as a monolithic scene-level reconstruction. ## 3. Cross-Agent Instance Alignment and the Instance Collaborative Graph The Cross-Agent Instance Alignment module is the mechanism that ensures different agents agree on which observations correspond to the same physical object instance. It begins with **confidence-based instance propagation**. For mask confidence, the framework defines $$ C_{k,t,i} = \begin{cases} 0 & O_{k,t,i} \notin I_{k,t}^c \\ \frac{S(O_{k,t,i})}{S(I^c_{k,t})} & O_{k,t,i} \notin I_{k,t}^{c,center} \\ 1 & O_{k,t,i} \in I_{k,t}^{c,center} \end{cases} $$ and averages this over time as $$ C_{k,i} = \frac{1}{T}\sum_{t=1}^{T} C_{k,t,i}. $$ The confidence reflects the position of an observed object in the image and is subsequently used to select reference instances during propagation [2509.01228]. For **cross-agent alignment**, each instance mask $O_{k,i}$ is converted to a downsampled point cloud $P_{k,i}$. Spatial overlap between instances from different agents is measured using Intersection-over-Union and Intersection-over-Base: $$ IoU_{k_1,i}^{k_2,j} = \frac{|P_{k_1,i}\cap P_{k_2,j}|}{|P_{k_1,i}\cup P_{k_2,j}|}, \qquad IoB_{k_1,i}^{k_2,j} = \frac{|P_{k_1,i}\cap P_{k_2,j}|}{|P_{k_1,i}|}. $$ Edges are created in the **Instance Collaborative Graph** between highly overlapping instances, and the resulting connected structure forms clusters of observations believed to correspond to the same object across agents [2509.01228]. Within each cluster, the most confident instance provides semantic and fusion updates for the others. The summary identifies three specific correction cases: **Semantic Error/Viewpoint Loss**, where low-confidence instances have IDs and features replaced by the reference; **Over-Segmentation**, where overlapping masks are merged; and **Under-Segmentation**, where the reference point cloud is projected onto other masks for segmentation refinement. After this propagation, the system reports a one-to-one consistent instance alignment across all agents, described as an injective mapping state. This is a distinctive feature of OpenMulti: alignment is defined at the instance level rather than only at the parameter or feature-aggregation level. ## 4. Distributed implicit representation and cross rendering supervision OpenMulti represents the scene through **instance-level NeRFs**. Each agent maintains a separate small MLP NeRF per object instance, denoted $F_{k,i}$, rather than a single global scene NeRF. For coordinates within object $O_{k,i}$, $$ F_{k,i}(\vec{x}) \to (\vec{c}, \sigma), $$ where $\vec{c}$ is color and $\sigma$ is density. Along a ray, the ray termination probability at depth $p$ is $$ T_p = \sigma(\vec{x}_p)\prod_{q