OpenMulti: Distributed Instance Mapping

• OpenMulti is an open-vocabulary instance-level distributed mapping framework that combines object-level neural fields with rich semantic annotations.
• It employs cross-agent instance alignment and cross rendering supervision to ensure consistent, robust, and fine-grained geometric and semantic mapping across agents.
• Experimental results demonstrate superior performance in geometric accuracy, zero-shot semantic segmentation, and communication efficiency compared to prior methods.

OpenMulti is an open-vocabulary instance-level multi-agent distributed implicit mapping framework for multi-agent distributed collaborative mapping. It was introduced to address the limitation that existing approaches lack instance-level awareness and semantic understanding of environments, which constrains their usefulness for downstream applications. The framework combines Cross-Agent Instance Alignment with Cross Rendering Supervision, maintains object-level NeRFs annotated with rich open-vocabulary semantics, and supports instance-level retrieval tasks and semantic annotations for downstream applications. Experimental results report that OpenMulti outperforms related algorithms in both fine-grained geometric accuracy and zero-shot semantic accuracy (Dou et al., 1 Sep 2025).

1. Problem setting and conceptual scope

OpenMulti is presented as the first framework to enable open-vocabulary, instance-level, distributed implicit mapping among multiple agents. Its design targets collaborative mapping scenarios in which multiple agents autonomously collect observations, maintain distributed scene representations, and require consistent object-level semantics across the team. The central claim is that prior distributed mapping methods do not provide both instance-level awareness and open-vocabulary semantic understanding, whereas OpenMulti explicitly integrates both within a distributed implicit mapping pipeline (Dou et al., 1 Sep 2025).

The framework is organized around four stated contributions. First, it introduces Cross-Agent Instance Alignment, a module for consistent, confidence-based instance-level map alignment across multiple agents via an Instance Collaborative Graph. Second, it introduces Cross Rendering Supervision, a supervision strategy based on ray-based rendering consistency rather than only parameter sharing. Third, it constructs instance-level, open-vocabulary mapping, in which each agent maintains object-level NeRFs annotated with rich semantics through VLM-derived features. Fourth, it reports superior performance in geometry and zero-shot semantics, while also being more communication-efficient and robust to network failures. This suggests that OpenMulti is positioned not merely as a geometric mapping system, but as a distributed semantic scene representation framework oriented toward retrieval and downstream robot reasoning (Dou et al., 1 Sep 2025).

2. Pipeline, data processing, and feature construction

OpenMulti consists of three main stages: Multi-Agent Data Collection & Preprocessing, Cross-Agent Instance Alignment, and Collaborative Optimization and Learning. In the first stage, each agent autonomously gathers RGB/depth images and pose data, and performs open-vocabulary instance segmentation and feature extraction. The summary specifies CropFormer for instance segmentation, CLIP for visual features from mask crops, TAP for caption generation, and SBERT for caption encoding. Each instance is assigned a unique identifier, and the resulting visual and textual features are stored in a feature code book (Dou et al., 1 Sep 2025).

Within each agent, the system first performs Cross-Frame Fusion. Instance masks $\{m^{inst}_{k,t,m}\}$ are clustered temporally into unified object tracks $O_{k,i}$, with fused features $f_{k,i}^{clip}$ and $f_{k,i}^{caption}$. This stage converts per-frame detections into persistent object-level entities before inter-agent reconciliation begins. The use of both CLIP and caption embeddings is central to the later open-vocabulary retrieval mechanism, because each object is represented by both visual and textual descriptors rather than by a fixed closed-set category label.

The preprocessing stage therefore establishes two properties required by later modules. The first is object persistence within an agent, achieved through temporal fusion. The second is multimodal semantic representation, achieved through the feature code book. A plausible implication is that OpenMulti treats the distributed map as a collection of instance-indexed neural fields enriched by external VLM feature spaces rather than as a monolithic scene-level reconstruction.

3. Cross-Agent Instance Alignment and the Instance Collaborative Graph

The Cross-Agent Instance Alignment module is the mechanism that ensures different agents agree on which observations correspond to the same physical object instance. It begins with confidence-based instance propagation. For mask confidence, the framework defines

$$C_{k,t,i} = \begin{cases} 0 & O_{k,t,i} \notin I_{k,t}^c \ \frac{S(O_{k,t,i})}{S(I^c_{k,t})} & O_{k,t,i} \notin I_{k,t}^{c,center} \ 1 & O_{k,t,i} \in I_{k,t}^{c,center} \end{cases}$$

and averages this over time as

$C_{k,i} = \frac{1}{T}\sum_{t=1}^{T} C_{k,t,i}.$

The confidence reflects the position of an observed object in the image and is subsequently used to select reference instances during propagation (Dou et al., 1 Sep 2025).

For cross-agent alignment, each instance mask $O_{k,i}$ is converted to a downsampled point cloud $P_{k,i}$. Spatial overlap between instances from different agents is measured using Intersection-over-Union and Intersection-over-Base:

$$IoU_{k_1,i}^{k_2,j} = \frac{|P_{k_1,i}\cap P_{k_2,j}|}{|P_{k_1,i}\cup P_{k_2,j}|}, \qquad IoB_{k_1,i}^{k_2,j} = \frac{|P_{k_1,i}\cap P_{k_2,j}|}{|P_{k_1,i}|}.$$

Edges are created in the Instance Collaborative Graph between highly overlapping instances, and the resulting connected structure forms clusters of observations believed to correspond to the same object across agents (Dou et al., 1 Sep 2025).

Within each cluster, the most confident instance provides semantic and fusion updates for the others. The summary identifies three specific correction cases: Semantic Error/Viewpoint Loss, where low-confidence instances have IDs and features replaced by the reference; Over-Segmentation, where overlapping masks are merged; and Under-Segmentation, where the reference point cloud is projected onto other masks for segmentation refinement. After this propagation, the system reports a one-to-one consistent instance alignment across all agents, described as an injective mapping state. This is a distinctive feature of OpenMulti: alignment is defined at the instance level rather than only at the parameter or feature-aggregation level.

4. Distributed implicit representation and cross rendering supervision

OpenMulti represents the scene through instance-level NeRFs. Each agent maintains a separate small MLP NeRF per object instance, denoted $F_{k,i}$, rather than a single global scene NeRF. For coordinates within object $O_{k,i}$0,

$O_{k,i}$1

where $O_{k,i}$2 is color and $O_{k,i}$3 is density. Along a ray, the ray termination probability at depth $O_{k,i}$4 is

$O_{k,i}$5

and the rendered outputs include

$O_{k,i}$6

Multi-instance compositing is performed using depth-aware volume rendering, stated to follow VMAP (Dou et al., 1 Sep 2025).

Local optimization is driven by an occupancy, depth, and color objective. The total local data loss is

$O_{k,i}$7

To support distributed learning, OpenMulti also uses a parameter consistency loss

$O_{k,i}$8

which exchanges NeRF weights among agents. The summary notes that this is a standard approach in distributed NeRF optimization, referencing DiNNO and MACIM as comparison points (Dou et al., 1 Sep 2025).

The system’s central additional supervision mechanism is Cross Rendering Supervision, motivated by the blind-zone optimization trap. In blind zones, an agent has no data-driven supervision, so parameter consistency alone may not prevent holes or discontinuities in rendered geometry. OpenMulti therefore enforces rendered-depth agreement directly:

$O_{k,i}$9

The key idea is that a collaborating agent shares ray directions for regions it has observed; both agents render along those rays, and the discrepancy is minimized. According to the summary, this supervision forces agents to predict consistent scene geometry everywhere, even in unobserved regions, thereby reducing blind-zone artifacts (Dou et al., 1 Sep 2025).

5. Open-vocabulary semantics, code book design, and retrieval

OpenMulti associates each object instance with open-vocabulary semantic features stored in the feature code book. The stored descriptors include CLIP-based visual features and SBERT-encoded caption features derived from TAP-generated descriptions. The code book supports both semantic annotation and retrieval, so the representation is not restricted to reconstruction or occupancy estimation (Dou et al., 1 Sep 2025).

The retrieval mechanism computes cosine similarities between a query and stored instance features. The reported scoring function is

$f_{k,i}^{clip}$0

where $f_{k,i}^{clip}$1 weights the visual and caption modalities. Queries may therefore be image-based, text-based, or description-based. The framework explicitly states support for queries by name, description, or visual similarity, and each instance receives rich multimodal tags for downstream robotic tasks (Dou et al., 1 Sep 2025).

This design links OpenMulti to open-vocabulary scene understanding rather than closed-set semantic mapping. The retrieval layer is instance-centric, not voxel-centric, and this is consistent with the system’s earlier use of object-level NeRFs and cross-agent instance alignment. A plausible implication is that OpenMulti’s semantic representation is intended to remain stable across views and agents because it is anchored to the aligned instance identity rather than to transient per-frame detections.

6. Experimental results, ablations, and reported significance

The reported quantitative results cover both geometry and semantics. For geometry, OpenMulti is compared with Di-NeRF*, MACIM, DiNNO_Inst, DSGD_Inst, and a single-agent baseline. For semantics, it is compared with OpenObj, ConceptGraph, LERF, and 3D-OVS. The principal reported numbers are as follows (Dou et al., 1 Sep 2025).

Setting	Metric	OpenMulti
Geometry	Acc. cm	2.47
Geometry	Comp. cm	3.52
Geometry	Comp. Ratio <5cm %	81.52
Semantics	F-mIoU (↑)	46.18
Semantics	F-mAcc (↑)	51.92

The geometry table in the summary reports OpenMulti as best on accuracy, completeness, and reconstruction ratio. The semantics table reports OpenMulti ahead of OpenObj, ConceptGraph, LERF, and 3D-OVS on zero-shot semantic segmentation. The stated interpretation is that these gains arise from effective cross-agent semantic correction, instance-level granularity, and stronger distributed geometry learning (Dou et al., 1 Sep 2025).

The framework also reports a runtime comparison: 4.76 min for OpenMulti versus 12.43 min for OpenObj. In addition, robustness to communication failures is emphasized. Under severe communication drop, OpenMulti incurs 15.2% mapping completeness loss, compared with 39.2% for Di-NeRF*. The ablation study further states that removing instance-based mapping or Cross Rendering Supervision degrades accuracy and completeness, which is used as evidence that both the instance-level decomposition and the rendering-consistency loss are functionally necessary components of the method (Dou et al., 1 Sep 2025).

The significance claimed for OpenMulti is fourfold: it establishes the first end-to-end, scalable system for open-vocabulary, instance-level, distributed implicit mapping; it integrates VLM-derived semantics with neural fields; it enables rich semantic querying and compositional scene representation through instance-level distributed collaboration; and it improves mapping accuracy, semantics, speed, and communication robustness. Within the scope of the reported paper, OpenMulti is therefore best understood as a distributed object-centric neural scene representation system that unifies multi-agent mapping, open-vocabulary semantics, and retrieval-oriented scene understanding (Dou et al., 1 Sep 2025).