FusionGatedFIRNet: Multi-Expert Object Detector

• The paper introduces a transfer learning paradigm that fuses outputs from domain-specific experts via a gating network, achieving improved mAP (e.g., from 87.9 to 90.4) on surveillance data.
• The methodology employs a ResNet-50 based feature extractor with softmax normalization to generate adaptive weights, enabling dynamic fusion in heterogeneous multi-camera environments.
• The approach enhances scalability and privacy by transferring model parameters instead of raw images, proving effective in resource-constrained settings and real-world applications.

FusionGatedFIRNet is a model fusion architecture designed for object detection in multi-domain scenarios, particularly video surveillance environments characterized by numerous, heterogeneous camera installations. The core idea is to enable transfer learning from multiple source domains (each represented by its own trained model or "expert") to a single target domain, bypassing the limitations of traditional one-to-one domain adaptation and addressing the unique challenges of privacy and scalability in real-world deployments.

1. Architectural Overview of the Gating Network

The FusionGatedFIRNet approach centers on a gating network that mediates between multiple source domain models and the target domain. The gating network is built atop a deep neural network backbone, such as ResNet-50 pre-trained on ImageNet, with its final layer modified to output as many units as there are source models. For a given input image $x$, the gating function $G(x)$ is defined as

$G(x) = \operatorname{softmax}(R(x; \theta)),$

where $R(x; \theta)$ represents the feature extraction backbone with trainable parameters $\theta$, and the output layer dimension is set to match the number of source models $n$. The softmax normalization ensures the output weights form a valid probability distribution over the ensemble of experts.

Given source models $\{E_i(\cdot)\}_{i=1}^n$, the fused prediction $Y$ for an input $x$ is calculated as: $Y = \sum_{i=1}^n G(x)_i \cdot E_i(x).$ This architecture allows the gating network to act as an intelligent, input-conditioned ensemble aggregator.

2. Model Fusion Methodology

The FusionGatedFIRNet workflow is organized in two main phases: gating network training and inference.

• Training Phase:
  • Each source domain yields a pre-trained expert $E_i$, e.g., RetinaNet models optimized for individual video cameras.
  • A limited target domain dataset is used to supervise the gating network. For each image $x$ in the target set, predictions from each $E_i$ are weighted by $G(x)_i$, aggregated, and compared against ground truth annotations.
  • The loss for a mini-batch combines regression and classification objectives:
  • Regression: $$l_{\text{reg}} = \text{SmoothL1}(Y_{\text{reg}}, T_{\text{reg}})$$.
  • Classification: $$l_{\text{cls}} = \text{focal\_weight} \cdot \text{BCE}(Y_{\text{cls}}, T_{\text{cls}})$$, where BCE is binary cross-entropy.
  • The total loss $L = l_\text{reg} + l_\text{cls}$ is backpropagated to update $\theta$.
• Inference Phase:
  • For each incoming image, the gating network yields $G(x)$.
  • Each expert processes $x$ independently; the gating weights are applied to their outputs.
  • Fused outputs (bounding boxes and scores) undergo non-maximum suppression (NMS) to generate the final predictions.

This methodology enables dynamic, per-input fusion, adapting ensemble contributions to the specific input characteristics.

3. Transfer Learning Scenario and Novelty

FusionGatedFIRNet establishes a new transfer learning paradigm for scenarios hosting a large, heterogeneous collection of source domains. Unlike traditional transfer learning, which adapts a source model to a target domain via fine-tuning, this architecture enables knowledge transfer by fusing the outputs of multiple domain-specific models. Crucially, this mechanism does not require sharing raw image data, only model parameters.

Key features of this scenario include:

• Provision for multiple source domains, each supplying a distinct bias-specific expert.
• Automated, input-dependent selection and fusion of models, eliminating the need for manual domain matching.
• Applicability in privacy-preserving contexts, since model transfer suffices and direct access to sensitive data is avoided.
• Enhanced scalability, as new target deployments can leverage the ensemble of existing experts through the gating network.

This framework is particularly suited for surveillance systems where each camera represents a unique domain, and legal or ethical constraints prohibit sharing of sensitive footage.

4. Experimental Evaluation

The effectiveness of FusionGatedFIRNet was assessed on a subset of the UA-DETRAC dataset, simulating real-world surveillance with 30 source domains (videos S1–S30) and 4 held-out target domains (videos T1–T4). Individual object detectors were trained on each source domain.

• Baselines Used:
  • Fine-tuning source models on limited target data, which often led to overfitting.
  • Naive averaging (uniform weighting) of the outputs from all source models.
• Gating Network Methods:
  • Using all 30 source models with the gating network yielded increased mean Average Precision (mAP) on target domains. On T1, mAP improved from 87.9 (average method) to 90.4; on T3, from 87.0 to 94.1.
  • A "Top-5 selection" mechanism, applying the gating network to select and fuse the 5 most relevant models per input, achieved comparable or improved results: T1 mAP reached 91.0. This reduces computational demand without degrading accuracy.
• Statistical Analysis:
  • Performance gains from model fusion persist as the number of source domains increases, provided only reasonably accurate models are selected. Including poorly performing models was observed to harm accuracy on certain domains (e.g., T2).
  • The adaptive fusion mechanism outperformed both fine-tuning (which was susceptible to overfitting with small target sets) and naive averaging across all target domains.

These results affirm the practicality and efficacy of gating-based multi-source fusion for object detection in challenging, real-world environments.

5. Practical Applications and Implications

FusionGatedFIRNet addresses several key real-world needs:

• Video Surveillance Systems: Supports construction of robust, bias-adapted detectors for new camera installations by aggregating knowledge from multiple existing, domain-specific models without exposing raw images.
• Scalability in Multi-Camera Deployments: Automated fusion streamlines adaptation across diverse camera views, obviating manual model selection even as system scale increases.
• Privacy-Aware Learning: Transfer of learned model parameters, rather than raw sensor data, minimizes compliance and privacy concerns, relevant to both regulated public spaces and commercial environments.
• Broader Domain Applicability: The methodology extends to other heterogeneous-sensor scenarios—such as retail, transportation, or smart city analytics—where distributed data silos and domain bias predominate.
• Resource Optimization: The top-$k$ selection strategy enables practitioners to balance accuracy and computational cost, making deployment in resource-constrained environments feasible.

A plausible implication is that this architectural pattern could generalize to other tasks (beyond object detection) requiring effective, privacy-preserving, multi-source knowledge aggregation.

6. Considerations and Limitations

• Fusion Quality Dependence: The performance benefit depends on the quality of source models; indiscriminate fusion of low-accuracy models can degrade results.
• Limited Target Data: Though the gating network requires only limited target labels, its effectiveness may be bounded by the diversity and representativeness of this set.
• Resource Constraints: While top-$k$ fusion reduces computational burden, inference still requires passing data through multiple large expert networks, which may be nontrivial at edge or embedded deployments.

Continued research on dynamic model selection, incremental expert updating, and lightweight gating mechanisms may further enhance the usability of this methodology in diverse application domains.