Sensor array and camera fusion via unbalanced optimal transport for 3D source localization

Abstract: We address the problem of localizing multiple sources in 3D by combining sensor array measurements with camera observations. We propose a fusion framework extending the covariance matrix fitting method with an unbalanced optimal transport regularization term that softly aligns sensor array responses with visual priors while allowing flexibility in mass allocation. To solve the resulting largescale problem, we adopt a greedy coordinate descent algorithm that efficiently updates the transport plan. Its computational efficiency makes full 3D localization feasible in practice. The proposed framework is modular and does not rely on labeled data or training, in contrast with deep learning-based fusion approaches. Although validated here on acoustic arrays, the method is general to arbitrary sensor arrays. Experiments on real data show that the proposed approach improves localization accuracy compared to sensor-only baselines.

• The paper introduces a CMF extension using unbalanced optimal transport that integrates visual priors into acoustic localization.
• It employs a greedy coordinate descent algorithm to achieve robust 3D source localization with lower mean squared error, notably improving depth accuracy.
• Experimental results validate the fusion approach against array-only baselines, demonstrating resilience to noise and changes in sensor geometry.

Sensor Array and Camera Fusion via Unbalanced Optimal Transport for 3D Source Localization

Introduction

The paper presents a framework for 3D source localization that fuses heterogeneous modalities, specifically sensor array signals and camera observations, via unbalanced optimal transport (UOT) regularization. By extending the classical covariance matrix fitting (CMF) method, the approach enables robust, modular, and training-free integration of visual priors into the acoustic localization process. This UOT-based regularization allows for soft spatial alignment between modalities, directly influencing the optimization process rather than employing rigid post-hoc fusion or requiring data-intensive supervised learning.

Methodology

The core problem addressed is the inherent limitation of sensor arrays, particularly in depth estimation, versus the complementary geometric and semantic cues offered by cameras. The approach integrates visual detections into the source power estimation process by augmenting CMF with a UOT term that softly enforces spatial alignment without necessitating stringent mass conservation. This is motivated by the fact that cameras provide unreliable estimates of signal magnitude and distance.

Given two heterogeneous distributions—visual detections represented as discrete sets in 3D and the acoustic power distribution over a volumetric grid—the UOT-based regularization incorporates visual priors by minimizing a transport cost related to angular deviations in 3D space. The resulting objective function is:

$$\min_{\mathbf{b} \geq 0} \left\| \mathbf{G} \operatorname{diag}(\mathbf{b}) \mathbf{G}^H - \mathbf{\hat{\Sigma}} \right\|_F^2 + \lambda\, \mathrm{UOT}(\nu_{\mathbf{a}}, \eta_{\mathbf{b}})$$

Where:

• $\mathbf{G}$: sensor transfer matrix
• $\mathbf{b}$: acoustic source power over grid points
• $\mathbf{\hat{\Sigma}}$: empirical covariance matrix
• $\mathrm{UOT}$: unbalanced optimal transport cost between camera-derived and grid distributions.

The UOT term leverages a cost matrix based on angular distances, mitigating the unreliability of camera-derived depth cues by enforcing that all points along a visual ray incur identical cost.

Optimization over the large-scale joint variable space is conducted using a greedy coordinate descent algorithm, enabling feasible runtimes for full 3D grids without recourse to expensive Sinkhorn solvers or stochastic training.

Figure 1: Spatial configuration indicating the relative positions of the sensor array, camera, and sources.

Figure 2: Experimental setup with spatial alignment and arrangement of microphones, sources, and camera.

Experimental Results

Validation employs a real-world dataset obtained in a gymnasium, using a 32-microphone array and four loudspeakers. The array covers a $2.4\times2\times0.7$ m$^3$ volume, discretized into $4.2\times10^5$ points, with visual detections serving as priors. Manual annotation replaces automated detection, focusing on validating the fusion method in a controlled scenario.

The method demonstrates clear improvements over array-only CMF baselines, especially in depth accuracy and reduction of spurious estimates. Quantitatively, source localization mean squared error (MSE) is consistently lower across varying SNRs and source distances.

Figure 3: Full point cloud comparison, illustrating CMF’s diffuse and artifact-prone estimates versus CMF-UOT’s concentrated and accurate localization.

Figure 4: Mean squared localization error versus SNR, showing performance gains of CMF-UOT over CMF, especially at low SNR.

Figure 5: Mean squared localization error as a function of source-array distance, highlighting robustness of the fusion approach even as array resolution degrades with distance.

Notably, CMF-UOT is resilient both to noise and increased distances, where CMF alone fails, as evidenced by persistent accuracy under escalating acoustic ambiguity. The computational cost remains practical—approximately 2.45 seconds per localization—underscoring the utility of the coordinate descent optimization for large grids.

Implications and Future Directions

The presented framework demonstrates the feasibility of integrating visual priors into sensor array source localization, bypassing the need for labeled data or retraining characteristic of deep learning-based late or early fusion. The modularity allows substitution of both sensor- and visual-front ends, enabling adaptation to alternative sensor types (e.g., radar arrays) and object detection frameworks.

Practically, this enables robust 3D localization in scenarios where array geometry, camera placement, or sensing environments change, mitigating the brittleness of purely data-driven fusion. Theoretically, the integration of UOT regularization sets a precedent for handling mismatches in cross-modal information granularity, especially when one modality lacks reliable quantitative attributes.

Open questions and speculative directions include:

• Extension to dynamic, non-static scenes with moving sources and cameras.
• Enhancement of transport constraints to manage severe visual occlusions.
• Automatic adaptation of regularization and unbalance parameters ($\lambda$, $\mu$), potentially informed by sensor confidence or noise estimation.
• Application beyond acoustics, where similar geometric-complementary sensing pairs exist.

Conclusion

By introducing an unbalanced optimal transport regularization into the CMF framework, the paper establishes a practical, modular, and efficient methodology for 3D source localization through sensor array and camera fusion. Experimental evidence underlines superior spatial resolution, especially in depth, compared to sensor-only approaches. The method sidesteps data dependence, offers resilience to geometric configuration changes, and provides a computationally viable solution for real-world volumetric localization. Extensions to more complex environments and modalities are natural avenues for further research.