Domain-Specialized Object Detection via Model-Level Mixtures of Experts

Abstract: Mixture-of-Experts (MoE) models provide a structured approach to combining specialized neural networks and offer greater interpretability than conventional ensembles. While MoEs have been successfully applied to image classification and semantic segmentation, their use in object detection remains limited due to challenges in merging dense and structured predictions. In this work, we investigate model-level mixtures of object detectors and analyze their suitability for improving performance and interpretability in object detection. We propose an MoE architecture that combines YOLO-based detectors trained on semantically disjoint data subsets, with a learned gating network that dynamically weights expert contributions. We study different strategies for fusing detection outputs and for training the gating mechanism, including balancing losses to prevent expert collapse. Experiments on the BDD100K dataset demonstrate that the proposed MoE consistently outperforms standard ensemble approaches and provides insights into expert specialization across domains, highlighting model-level MoEs as a viable alternative to traditional ensembling for object detection. Our code is available at https://github.com/KASTEL-MobilityLab/mixtures-of-experts/.

• The paper introduces a novel MoE framework that dynamically routes inputs to YOLO-based, domain-specialized experts, enhancing interpretability and performance under domain shifts.
• It leverages domain-aware splitting of the BDD100K dataset to train specialized detectors, achieving higher mAP compared to traditional ensembles.
• The work employs a gating network with entropy regularization to balance expert contributions, offering actionable insights for safety-critical applications like autonomous driving.

Domain-Specialized Object Detection via Model-Level Mixtures of Experts: A Technical Overview

Introduction

The paper "Domain-Specialized Object Detection via Model-Level Mixtures of Experts" (2604.18256) addresses the challenge of achieving high-performance and interpretable object detection in safety-critical, diverse real-world environments. The authors propose a model-level Mixture-of-Experts (MoE) framework for object detection, combining multiple strongly specialized YOLO-based detectors with a learned gating mechanism. This architecture is designed to enable modular specialization, dynamic expert selection, and transparent expert attribution for detection outcomes, contrasting with traditional, post-hoc heuristic ensembling strategies that obscure model contributions.

The approach is systematically evaluated on the BDD100K dataset, partitioned by semantic domains (daytime vs. nighttime), and compared against the strongest single-model baselines and static ensemble techniques. Comprehensive ablations and interpretability analyses are performed to elucidate the behavior of the gating mechanism, the benefits and limitations of domain-based specialization, and the nature of expert disagreement and domain overlap.

Model-Level Mixtures of Experts in Object Detection

Conventional ensemble methods, although beneficial for increasing predictive accuracy, are limited by postprocessing heuristics such as Non-Maximum Suppression (NMS), which obfuscate attribution and hinder interpretability. Recent advances in layer-level MoEs within large-scale transformer architectures have demonstrated success in enabling sparse, input-dependent expert activation; however, model-level MoEs, where entire models serve as specialized experts, are particularly suited for tasks demanding both robust modularity and interpretable decision policies.

The proposed MoE architecture integrates multiple independently trained YOLO-based detectors—each fine-tuned on semantically disjoint dataset splits—with a gating network that computes input-dependent routing weights. This structure—illustrated in (Figure 1)—enables the fusion of structured, dense object detection outputs prior to application of NMS or other fusion methods, preserving the traceability of expert contributions.

Figure 1: Proposed MoE architecture consisting of two expert models.

Each expert model processes the raw image and produces object class probabilities, BBox regressions, and objectness scores. Their respective intermediate feature maps are concatenated and passed to the gate, which outputs a soft distribution over experts. The resulting weighted detections are then postprocessed using domain-optimized fusion algorithms (e.g., Non-Maximum Weighted Fusion, NMW).

Dataset Splitting and Expert Specialization

The evaluation leverages the BDD100K driving dataset, characterized by significant domain shifts along the time-of-day axis. This dataset is partitioned by metadata fields into daytime and nighttime splits, with additional subsets for dawn/dusk and undefined. The split distribution and representative images are visualized in (Figure 2) and (Figure 3).

Figure 2: Distribution of BDD100K data according to the metadata parameter timeofday.

Figure 3: Exemplary images with different values of the timeofday metadata parameter in the BDD100K dataset.

This splitting supports the training of expert models that strongly specialize: each expert achieves optimal in-domain performance but suffers from domain shift on OOD (Out-of-Distribution) domains.

Training Methodology

The primary training strategy involves freezing expert weights and training only the gating network using detection loss. To mitigate expert collapse (over-reliance on a single expert), several balancing losses (importance, KL-divergence, entropy-based, and sample-wise entropy regularization) are explored. Empirical evidence supports the effectiveness of sample-wise entropy regularization for stable and balanced expert usage.

Domain-aware gate training (using explicit domain labels as gate supervision) is found to slightly degrade mAP compared to unsupervised gating, suggesting that hard-coded domain boundaries may limit the gate’s ability to exploit the nuanced visual cues important for fine-grained routing decisions.

Fusion Strategies and Numerical Performance

Comprehensive experiments demonstrate that the MoE substantially outperforms static ensembles (e.g., simple concatenation, SoftNMS, WBF) on both in-domain and OOD splits for all YOLO model variants considered. Notably, across all evaluated fusion methods, NMW fusion with mAP-based reweighting yields the strongest ensemble results.

The MoE consistently delivers higher mAP than fixed ensembles, with the most substantial improvements on in-domain data (see Table 1 for illustrative mAP comparisons per domain). However, models trained on the combined, unsplit dataset remain the absolute top performers, reflecting residual limitations associated with hard expert specialization and gating-induced fusion. The following detection visualization, using the BDD100K data, demonstrates differences among specialized experts, classical ensembles, and the MoE:

Figure 4: Predictions of different models on an exemplary image from the BDD100K dataset.

Gating, Ablations, and Interpretability

The gating network is ablated across several axes:

• Gate architecture (from single FC to multi-conv + FC layers),
• Number and granularity of gates (single, spatial, classwise),
• Feature extraction layer sampling points (early, middle, late layers).

The best results are obtained with a compact gate using deep intermediate feature input from the backbone and a single image-level gating decision.

Critical analysis of the gate’s output distributions across all data splits—shown in (Figure 5)—demonstrates domain-aligned, input-dependent weighting, and reveals that, for ambiguous or OOD images, the network naturally adopts softer, less polarized gating, reflecting epistemic uncertainty:

Figure 5: Distribution of expert weights, predicted by a simple gate in a YOLOv7x-based MoE with samplewise entropy balancing loss and no domain-aware training.

Further, the disagreement analysis at the bounding box level (Figure 6) quantifies the types and locations of expert disagreement and exclusive expert detections. This enables tracing the ultimate MoE prediction back to its expert provenance on a per-object basis—delivering a granular form of local interpretability surpassing that of standard ensembles.

Figure 6: Disagreement analysis helps trace MoE decisions back to expert decisions. BBoxes with full expert agreement are green, BBoxes of expert disagreement on label are yellow, and BBoxes of objects not detected by either expert are red.

Practical and Theoretical Implications

Practically, the model-level MoE paradigm delivers substantial benefits for domains where specialization and interpretability are paramount (e.g., autonomous driving, medical imaging). It enables performance that is competitive with strong ensembling, but with the added value of exposing a transparent, input-dependent division of labor among expert detectors. The ability to trace predictions to domain-relevant experts provides intuitive debugging and reliability insights, especially in safety-critical applications.

Theoretically, the results validate the hypothesis that model-level, input-adaptive routing can approach, and sometimes surpass, static combined-model performance—particularly when meaningful domain splits are available. However, optimal performance in the presence of ambiguous or OOD data remains contingent on the gate’s capacity to interpolate between expert specializations, and on the coverage of domains represented by the experts.

Future research directions include scaling MoEs to larger, more finely partitioned expert pools, integrating more complex or uncertainty-aware gating functions, and extending the architecture to generic detection backbones. Additionally, automatic discovery of optimal expert specialization could further replace the current reliance on explicit metadata-based splits.

Conclusion

This work provides a comprehensive empirical and analytical investigation of model-level Mixture-of-Experts architectures for object detection. The combination of dynamically routed, domain-specialized detectors with explicit, interpretable fusion yields consistent improvements over classic ensembling strategies, both in accuracy and in attributional transparency. While specialization plus adaptive gating approaches the performance of strong baselines trained on all available data, the true added value is in modular interpretability and controllable expertise—properties critical for modern real-world detection systems.