---
title: PACGNet Architecture for Multimodal Detection
url: https://www.emergentmind.com/topics/pacgnet-architecture
type: topic
---

# PACGNet Architecture for Multimodal Detection

The Pyramidal Adaptive Cross-Gating Network (PACGNet) is an architecture for multimodal object detection, explicitly designed to address the limitations of conventional feature fusion methods in aerial imagery-based detection tasks. PACGNet preserves hierarchical feature structures and reduces cross-modal noise through a combination of bidirectional “horizontal” gating (Symmetrical Cross-Gating, SCG) and progressive “vertical” fusion (Pyramidal Feature-aware Multimodal Gating, PFMG) within a dual-stream deep backbone. This approach produces state-of-the-art results on challenging benchmarks, especially in the detection of small or fine-grained objects in registered RGB and infrared (IR) imagery [2512.18291].

## 1. Network Structure and Dataflow

PACGNet processes registered RGB and IR image pairs—each resized to $640 \times 640$—via a dual-stream YOLOv8 backbone. The backbone consists of two identical but independently parameterized branches (for RGB and IR), using standard C2f blocks. Each stream generates hierarchical feature maps at four scales: $P_2 \in \mathbb{R}^{C_2 \times \frac{H}{4} \times \frac{W}{4}}$, $P_3$, $P_4$, and $P_5$, with $C_2=128$, $C_3=256$, $C_4=512$, and $C_5=1024$ for the base model.

After each backbone stage except the final ($P_5$), a Symmetrical Cross-Gating (SCG) module fuses and refines features between the two modalities at corresponding resolutions. The SCG modules operate at $P_2$, $P_3$, and $P_4$.

Following the backbone, a series of PFMG modules perform progressive, top-down fusion of features at decreasing resolutions ($P_3$, $P_4$, $P_5$), with each level’s fusion guided by the fused features from the previous, higher-resolution stage. The final fused pyramid $\{P_{\text{fused}3}, P_{\text{fused}4}, P_{\text{fused}5}\}$ is processed by a standard PANet neck and a YOLOv8-oriented box detection head.

The main dataflow is as follows:

- $[RGB_{in}, IR_{in}] \rightarrow$ dual-stream backbone
- $\{P_2^{RGB}, P_2^{IR}\} \rightarrow SCG@P_2 \rightarrow \{P_2'^{RGB}, P_2'^{IR}\}$
- $... \rightarrow P_3^{RGB}, P_3^{IR} \rightarrow SCG@P_3 \rightarrow ...$
- $... \rightarrow P_4^{RGB}, P_4^{IR} \rightarrow SCG@P_4$
- $... \rightarrow P_5^{RGB}, P_5^{IR}$ (no SCG at $P_5$)
- $PFMG@P_3$ (guided by $P_2'$) $\rightarrow P_{\text{fused}3}$
- $PFMG@P_4$ (guided by $P_{\text{fused}3}$) $\rightarrow P_{\text{fused}4}$
- $PFMG@P_5$ (guided by $P_{\text{fused}4}$) $\rightarrow P_{\text{fused}5}$
- $PANet \rightarrow$ oriented-box detection head, outputs $[x, y, w, h, \theta, \text{class scores}]$

## 2. Symmetrical Cross-Gating (SCG) Module

At each of $P_2$, $P_3$, and $P_4$, the SCG module implements a symmetrical, bidirectional “horizontal” cross-gating function between modality feature maps $F_k^{RGB}$ and $F_k^{IR}$. Both streams are refined through intra-modal bottleneck layers before entering modality cross-gating.

SCG includes two gating routes per direction:
- **Spatial Gating**: Computes a spatial attention map from one modality and applies it to the other, e.g., $M_s^{IR \rightarrow RGB} = \sigma(W_s^{IR \rightarrow RGB} * Frgb)$, then $Frgb^{(s)} = Frgb \odot (1 + M_s^{IR \rightarrow RGB})$.
- **Channel Gating**: Projects the guidance modality through a bottleneck and 1x1 conv to yield a channel map, e.g., $g^{IR \rightarrow RGB} = \sigma(W_c^{IR \rightarrow RGB} * G^{IR \rightarrow RGB} + b)$, then $Frgb^{(c)} = g^{IR \rightarrow RGB} \odot G^{IR \rightarrow RGB}$.

The final output is obtained by residual addition and normalization:
$$\hat F^{RGB} = \text{BatchNorm}\left(F^{RGB} + Frgb^{(s)} + Frgb^{(c)}\right)$$

The operation is symmetric for $F^{IR}$, and each direction employs independent parameter sets. This ensures that cross-modal enhancement selectively absorbs complementary information while suppressing noise. The residual path prevents destructive gating and preserves modality-specific semantics.

## 3. Pyramidal Feature-aware Multimodal Gating (PFMG) Module

The PFMG module implements a progressive vertical fusion mechanism at each level $i \in \{3,4,5\}$, guided by the higher-resolution fused feature $P_{i-1}^{fused}$. The detailed steps include:

- **Hierarchical Spatial Gate**: Computes $M^{(i)} = \sigma(\text{Conv}_{3 \times 3, s=2}([P_{i-1}^{fused}; F_{i-1}^{RGB/IR}])) \in \mathbb{R}^{1 \times H_i \times W_i}$.
- **Modality Interaction & Weighting**: Concatenates and projects $F_i^{RGB}$ and $F_i^{IR}$, then computes a softmax score map $[\alpha, 1-\alpha]$ to yield a spatially adaptive blend.
- **Gated Fusion**: 
$$
F_i^{base} = \alpha \odot U^{RGB} + (1-\alpha) \odot U^{IR}
$$
$$
P_i^{fused} = F_i^{base} \odot M^{(i)} + F_i^{base}
$$

This process enforces the propagation of fine object detail down the feature pyramid, leveraging high-resolution spatial guidance to improve the preservation of object boundaries and small-object sensitivity.

## 4. Integration with Backbone and Detection Head

The dual-stream backbone is based on the YOLOv8 architecture, with C2f blocks in both RGB and IR branches (independent weights, identical topology). The SCG modules augment intra-scale feature interaction within the backbone, while PFMG modules replace traditional lateral connections typically seen in FPNs. After feature fusion, the resulting fused pyramids $P_{\text{fused}3}, P_{\text{fused}4}, P_{\text{fused}5}$ are fed into a standard PANet neck and then to an oriented-box detection head. Outputs consist of $[x, y, w, h, \theta, \text{class scores}]$.

## 5. Implementation Strategies and Training Configuration

- **Convolutional Kernels**:
  - SCG spatial gate: 1x1 conv + sigmoid
  - SCG channel gate: bottleneck 1x1 conv ($C_k \rightarrow C_k/2 \rightarrow C_k$) + sigmoid
  - PFMG spatial gate: 3x3 conv (stride=2) + sigmoid
  - PFMG interaction: 1x1 conv
- **Gate Initialization**: All gating conv weights set to zero, biases to zero, yielding initial gates $\approx 1$ post-sigmoid.
- **Optimization**: SGD, initial learning rate $lr_0 = 0.01$, final $lrf = 0.01$, momentum $0.937$, weight decay $5\times 10^{-4}$.
- **Batch Size & Augmentation**: Batch size $128$ distributed over $8\times$ RTX3090 GPUs; data augmentation via Mosaic, random flip, translate.
- **Loss Functions**: WIoU-v3 for bounding box regression, BCE for classification.
- **Epochs**: 3 epoch warmup, total 300 epochs.

## 6. Experimental Results and Comparative Evaluation

PACGNet was evaluated on DroneVehicle (5 classes, oriented boxes) and VEDAI (9 classes, small-object-heavy, oriented boxes) datasets. The principal metric was mAP50. Results are summarized:

| Model Variant               | DroneVehicle mAP50 | VEDAI mAP50 |
|-----------------------------|-------------------|-------------|
| Baseline (no SCG, PFMG)     | 80.1%             | 74.1%       |
| + PFMG only                 | 80.7%             | 76.7%       |
| + SCG only                  | 80.8%             | 76.6%       |
| SCG + PFMG (PACGNet)        | 81.7%             | 82.1%       |
| Prior SOTA (RGFNet)         | 81.4%             | 81.2%       |

PACGNet outperforms prior best methods, particularly on VEDAI, exhibiting an $8.0\%$ absolute gain in mAP50 over the baseline, indicating the synergistic effect of SCG and PFMG modules in enhancing cross-modal reinforcement and detail-aware gating for small object detection. The effect on DroneVehicle is positive but more modest, with non-additive gains confirming complementary roles for the two gating strategies.

## 7. Architectural Significance and Practical Implications

PACGNet’s integration of both cross-modal horizontal fusion (SCG) and vertical, detail-aware refinement (PFMG) addresses two prevalent challenges: cross-modal noise and the disruption of multi-scale feature hierarchies observed in conventional fusion schemes. The architecture maintains semantic integrity for each modality while promoting selective absorption of complementary features, and enforces high-fidelity propagation of fine-grained details across pyramid levels. The empirical outcomes on diverse benchmarks support the efficacy of these principles, especially in domains characterized by small object density and challenging multimodal registration [2512.18291].

*This suggests that progressive, attention-based gating at both intra- and inter-hierarchical levels may be an effective paradigm for multimodal feature fusion, motivating its application to other vision modalities and tasks characterized by scale sensitivity and cross-modal heterogeneity.*

Source: https://www.emergentmind.com/topics/pacgnet-architecture