On Single Source Robustness in Deep Fusion Models

Abstract: Algorithms that fuse multiple input sources benefit from both complementary and shared information. Shared information may provide robustness against faulty or noisy inputs, which is indispensable for safety-critical applications like self-driving cars. We investigate learning fusion algorithms that are robust against noise added to a single source. We first demonstrate that robustness against single source noise is not guaranteed in a linear fusion model. Motivated by this discovery, two possible approaches are proposed to increase robustness: a carefully designed loss with corresponding training algorithms for deep fusion models, and a simple convolutional fusion layer that has a structural advantage in dealing with noise. Experimental results show that both training algorithms and our fusion layer make a deep fusion-based 3D object detector robust against noise applied to a single source, while preserving the original performance on clean data.

• The paper introduces the MaxSSN loss to directly optimize worst-case prediction errors under single-source perturbations, achieving up to a 24% improvement in robustness.
• The study develops two training algorithms, TrainSSN and TrainSSNAlt, alongside a Latent Ensemble Layer (LEL) that maintains clean-data accuracy while boosting fault tolerance.
• Empirical and theoretical analyses reveal that conventional fusion models are highly sensitive to dominant sensor failures, underscoring the need for balanced multimodal resilience.

Single Source Robustness in Deep Fusion Models

Problem Definition and Motivation

Deep fusion models are widely adopted in safety-critical domains such as autonomous driving, leveraging complementary and shared information from heterogeneous sensors (e.g., camera, LIDAR). This work precisely defines and investigates "single source robustness," the capacity for a deep fusion model to withstand localized corruption affecting only one input source, a failure mode observed in real-world deployments. Standard fusion approaches optimize global prediction performance but fail to guarantee balanced resilience with respect to individual source faults. The paper establishes—starting from a tractable linear regime—that such unbalanced robustness is the rule, not the exception, under conventional training objectives.

Analysis in Linear Fusion Regimes

To clarify the origins of the vulnerability, the authors introduce a canonical linear model: two input sources each possess source-specific data and overlap on a latent shared representation. The regression target is an additive function of all components. The authors demonstrate that stacking features from both sources and learning a linear predictor under MSE minimization admits a degeneracy: the model can preferentially rely on the shared component of a particular input, rendering sensitivity highly asymmetric to source-specific perturbations. Analytical solutions demonstrate that, for data distributions where one source dominates, noise injected into that source results in significant prediction degradation, while the model remains robust to the other. This directly contradicts the safety motivation for multi-source fusion.

MaxSSN Loss: Explicit Objective for Single Source Robustness

Addressing the observed deficit, the authors propose the MaxSSN loss, an explicit surrogate for quantifying worst-case prediction error under single-source corruption. For a model $f$ with $n_s$ inputs, MaxSSN is defined as:

$$L_{MaxSSN}(f) = \max_{i} \mathbb{E}_{\epsilon_i}\left[L\left(y, f(x_1, \ldots, x_i + \epsilon_i, \ldots, x_{n_s})\right)\right]$$

where each $x_i$ is perturbed independently. This objective directly optimizes the maximal degradation over all $n_s$ possible source corruptions, ensuring well-calibrated vulnerability profiles. Theoretical analysis shows that, compared to standard training under all-source noise, models trained to minimize MaxSSN attain strictly lower worst-case error, with the improvement characterized analytically for the canonical model. Critically, the formulation also preserves clean-data accuracy, thereby sidestepping adversarial trade-offs observed in naively robust methods.

Algorithms for Deep Fusion: Training Procedures

The authors develop two training algorithms to minimize MaxSSN in deep architectures:

• TrainSSN: For every mini-batch, the network is exposed to $n_s$ variants, each with noise injected into a distinct source; the update direction targets the input with maximal loss.
• TrainSSNAlt: A more efficient stochastic variant that alternately augments only a single input per batch, rotating through sources.

Both algorithms interleave clean-data batches to avoid sacrificing uncorrupted performance. Empirically, both prove effective; TrainSSN slightly outperforms under strict evaluation, but TrainSSNAlt is computation-efficient and sufficient for small $n_s$.

Feature Fusion Layer Design: Latent Ensemble Layer

Beyond objective functions, model architecture contributes to robustness. The authors present the Latent Ensemble Layer (LEL), a learnable fusion operator generalizing beyond element-wise mean or concatenation. LEL enables flexible, channel-wise mixing, enforced with sparsity regularization, permitting the network to learn source/channel-specific mixing strategies that admit survival of source-private features. Experimentally, LEL displays an inherent tendency towards single source robustness even without robust training—contrasting with simpler fusion schemes that promote harmful parameter entanglement and redundancy.

Empirical Evaluation

Experiments focus on 3D object detection for autonomous driving (KITTI dataset, AVOD architecture). Models are subjected to two corruption types: Gaussian noise and sensor downsampling, each applied singly to camera or LIDAR.

Key results show:

• TrainSSN and LEL each substantially raise minAP (minimum AP over all single-source corruptions) and lower maxDiffAP (difference between best/worst AP among single-source corruptions).
• Under Gaussian noise, MaxSSN-trained models see up to a $24\%$ absolute improvement in worst-case AP relative to baselines; LEL yields robustification even without explicit MaxSSN minimization.
• Importantly, clean accuracy is preserved, confirming the efficacy of the interleaving training protocol.
• Under structured corruption (downsampling), robustification persists, indicating the approach generalizes beyond i.i.d. noise.

These observations hold for both feature mean fusion and LEL, though LEL architectures display higher overall robustness, especially in high-noise or extreme downsampling settings.

Theoretical and Practical Implications

The MaxSSN approach formally decouples robustness evaluation from global noise distributions, making robustness guarantees interpretable and directly actionable for practitioners. The findings underscore the inadequacy of conventional robust training (focused on all-source perturbations) for safety-critical sensor fusion. On the architectural side, structured fusion operators like LEL mitigate adversarial redundancy and can serve as a drop-in primitive for robust multimodal learning.

Theoretically, this analysis exposes fundamental limitations imposed by expressivity and loss function choice in multi-source deep models. The direct connection to multicollinearity in classical statistics and adversarial robustness in deep learning highlights avenues for integrating ideas across these domains. The study focuses on single source corruption but is extensible to $k$-of-$n_s$ faults; future work may generalize MaxSSN to arbitrary corruption subsets, dynamic sensor reliability, and adversarial settings, as discussed in the appendix.

Conclusion

This work advances the formalization and practical improvement of robustness in deep fusion models, targeting the challenging and realistic case of single-source corruption. The combination of MaxSSN loss, tailored training protocols, and structured fusion (LEL) delivers significant gains in both worst-case and balanced prediction accuracy without degrading clean performance. This study guides future research in robust multimodal learning, sensor fault tolerance, and architecture design for distributed perception systems.