---
title: 'ARPGNet: Appearance & Relation-aware Graph Fusion'
url: https://www.emergentmind.com/topics/appearance-and-relation-aware-parallel-graph-attention-fusion-network-arpgnet
type: topic
---

# ARPGNet: Appearance & Relation-aware Graph Fusion

The Appearance- and Relation-aware Parallel Graph attention fusion Network (ARPGNet) is an architecture developed for facial expression recognition (FER) that jointly models both facial appearance and inter-region relations with explicit cross-modality and temporal fusion. ARPGNet utilizes spatial facial appearance representations alongside a graph-based encoding of local relational cues, harnessed and fused with graph attention mechanisms which also incorporate local temporal context. This approach addresses deficiencies in conventional FER systems that focus exclusively on appearance features extracted by Convolutional Neural Networks (CNNs), neglecting the structured relationships between facial subregions and their temporal interplay [2511.22188].

## 1. Architecture Overview

ARPGNet consists of three principal modules: (1) a facial region relation graph branch, (2) an appearance representation backbone, and (3) a parallel graph attention fusion module. The overall workflow is as follows:

- **Facial Region Relation Graph:** Convolutional feature maps are subdivided into a $P \times P$ grid, constructing a graph where each node is a facial patch and edges encode local adjacency and bilateral symmetry. Graph Attention Networks (GATs) are then applied to aggregate regional and symmetric context per frame.
- **Appearance Backbone:** Features are extracted per frame by an InsightFace ResNet-50 CNN pre-trained on MS-Celeb-1M, then fine-tuned on FER datasets; the first fully connected (FC) layer serves as the appearance embedding.
- **Parallel Fusion:** Temporal sequences from both branches are enhanced and fused in a graph attention scheme that operates on a constructed fusion graph, incorporating a temporal response scope (TRS) constraint to promote local temporal modeling and interaction between appearance and relation features without recurrence.

This design enables ARPGNet to simultaneously learn and integrate complementary intra-frame (appearance, relation) and inter-frame (temporal) cues [2511.22188].

## 2. Facial Region Relation Graph Construction

After extracting convolutional feature maps (size $H=W=12$), adaptive average pooling divides the response into a $P \times P$ grid of patches, yielding $V=\{1,2,\ldots,P^2\}$ nodes per frame. The graph's fixed adjacency matrix $A \in \{0,1\}^{P^2 \times P^2}$ is defined by:

- Self-connection,
- 4-neighborhood (spatial adjacency),
- Mirror symmetry about the vertical facial midline.

Formally,
\[
a_{ij} =
\begin{cases}
1, & i = j, \\
1, & \text{$i,j$ are spatial grid neighbors}, \\
1, & \text{$i,j$ are symmetric about the midline}, \\
0, & \text{otherwise}.
\end{cases}
\]

Three stacked GAT layers (attention heads, LeakyReLU activations) update node features:
\[
e_{ij} = \mathrm{LeakyReLU}\left( \mathbf{a}^\top [h_i \parallel h_j] \right), \quad
\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k\in N_i}\exp(e_{ik})}, \quad
x'_i = \sigma\left( \sum_{j\in N_i} \alpha_{ij} h_j \right)
\]
with multi-head aggregation. Global average pooling yields per-frame relation embeddings $r_t \in \mathbb{R}^{C'}$ for temporal sequence $R = [r_1;\ldots;r_T]$ [2511.22188].

## 3. Appearance Feature Extraction Backbone

The appearance branch uses the InsightFace ResNet-50 backbone, structured as conv1 $\rightarrow$ Res2–Res5 $\rightarrow$ two FC layers. The first FC layer's output, $a_t \in \mathbb{R}^{C'}$, is used as the per-frame appearance feature. The network is fine-tuned on each target emotion dataset. The result is a temporal sequence $A = [a_1; \ldots; a_T] \in \mathbb{R}^{T \times C'}$ [2511.22188].

## 4. Parallel Graph Attention Fusion with Temporal Response Scope

To enable effective fusion and temporal modeling:

- **Positional Encoding:** Standard sine/cosine encodings are added to $A$ and $R$ before fusion,
  \[
  \text{PE}(pos,2k) = \sin\left( pos/10000^{2k/C'} \right),\;
  \text{PE}(pos,2k+1) = \cos\left( pos/10000^{2k/C'} \right)
  \]
- **Fusion Graph:** The fusion graph $G_{fusion}$ has $2T$ nodes—appearance ($0\ldots T-1$) and relation ($T\ldots 2T-1$)—with edges determined by the TRS hyperparameter:
  \[
  a_{ij}^{fusion} =
  \begin{cases}
    1, & |t_i-t_j| \le \mathrm{TRS} \\
    0, & \text{otherwise}
  \end{cases}
  \]
  This allows inter- and intra-modality attention within a $2\cdot TRS+1$ time window.
- **GAT Fusion:** The same multi-head GAT formulation is applied across this graph:
  \[
  \beta_{ij} = \frac{\exp(\mathrm{LeakyReLU}(\mathbf{a}^\top[Wx_i \parallel Wx_j]))}{\sum_{k\in N_i} \exp(\mathrm{LeakyReLU}(\mathbf{a}^\top[Wx_i \parallel Wx_k]))}
  \]
  producing enhanced appearance ($A'$) and relation ($R'$) sequences.

- **Aggregation and Classification:** At each timestep $t$, $z_t = [A'_t; R'_t] \in \mathbb{R}^{2C'}$, which are temporally pooled (mean) to form a video representation $z_{video}$ for MLP-based expression classification [2511.22188].

## 5. Temporal Sequence Modeling

ARPGNet integrates temporal modeling directly within the fusion graph by leveraging the TRS constraint. This enforces attention locality in time, allowing the network to model local spatiotemporal dynamics and fine-grained cross-modality effects without introducing explicit sequence models like RNNs or global Transformers. This parallel, locality-aware attention mechanism improves cross-modal temporal interaction and reduces modeling complexity [2511.22188].

## 6. Training Protocols, Data, and Implementation

- **Data Preprocessing:** Detection, alignment, and cropping to $96 \times 96$ are performed via OpenFace 2.0. For RML and AFEW, $T=16$ frames per video are sparsely sampled (random for training, uniform for evaluation); Aff-wild2 employs dilated sequences of 8 frames (dilation=3).
- **Optimization:** Adam optimizer ($\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-8}$). Learning rates: $10^{-4}$ (ResNet blocks), $10^{-3}$ (others). Dropout (0.25) follows all FC layers. Kaiming initialization is used; PReLU/LeakyReLU activations are adopted in ResNet/GAT, respectively.
- **Losses:** Standard multi-class cross-entropy for balanced datasets (RML, AFEW):
  \[
  \mathcal{L}_{\rm CE} = -\sum_{c=1}^C y_c \log p_c
  \]
  and focal loss ($\gamma=2$) for class-imbalanced Aff-wild2:
  \[
  \mathcal{L}_{\rm focal} = -\sum_{c=1}^C y_c (1-p_c)^\gamma \log p_c
  \]
  No additional regularization beyond dropout is used [2511.22188].
- **Implementation:** PyTorch platform on NVIDIA TITAN RTX GPUs. Batch sizes and epochs typically range $16$–$32$ and $30$–$50$, respectively.

## 7. Benchmark Results and Ablation Analysis

ARPGNet demonstrates consistently strong or superior performance across major FER datasets:

| Dataset     | State-of-the-art Baselines           | ARPGNet Performance         | ARPGNet w/pretraining    |
|-------------|--------------------------------------|-----------------------------|--------------------------|
| RML         | MulT 73.3%, C3D+audio 73.9%          | 76.53±3.05%                | —                        |
| AFEW        | HSE-NN 59.3%, MulT 55.87%            | 57.70%                     | 60.05%                   |
| Aff-wild2   | MulT 0.536 (𝑀), HSE-NN 0.521         | 0.547                      | 0.628                    |

Ablation studies on AFEW reveal that:
- Single-stream appearance or relation alone underperform (50.65% and 47.52%),
- Simple late fusion reaches 51.96%,
- GAT fusion without TRS yields 55.35%,
- Full model with TRS achieves 57.70%.

Grid-based relation graphs (adjacency+symmetry) surpass prior AU/lankdmark strategies: 57.70% (grid), 52.22% (RA-UWML), 54.31% (DDRGCN), 55.61% (Chang et al.). Patch size $6 \times 6$ outperforms $4 \times 4$, $12 \times 12$, and $3 \times 3$ configurations for the grid.

These results indicate both the complementary nature of appearance and relational cues, and the effectiveness of TRS-constrained graph attention fusion for spatiotemporal expression analysis [2511.22188].

Source: https://www.emergentmind.com/topics/appearance-and-relation-aware-parallel-graph-attention-fusion-network-arpgnet