Bi-Level Attention R-GCN

• The paper introduces BR-GCN with a dual-level attention mechanism that captures both intra-relation and inter-relation dependencies in heterogeneous graphs.
• It implements node-level masked self-attention and Transformer-style relation-level aggregation to generate expressive node embeddings for classification and link prediction.
• Empirical evaluations on datasets like AIFB and MUTAG demonstrate significant accuracy gains over R-GCN and GAT, underscoring its effectiveness and scalability.

Bi-Level Attention-Based Relational Graph Convolutional Networks (BR-GCN) are neural architectures that operate on directed, labeled graphs with large numbers of relation types by leveraging a hierarchical, or bi-level, attention mechanism. BR-GCN extends the principles of both Graph Attention Networks (GAT) and Transformer models to multi-relational and heterogeneous graph settings. Its design facilitates efficient and effective learning on highly multi-relational data, supporting both node classification and link prediction tasks with state-of-the-art performance (Iyer et al., 2024).

1. Model Architecture and Bi-Level Attention Structure

BR-GCN is structured as a multi-layer graph neural network in which each layer comprises two attention stages:

• Node-Level Attention (Intra-Relation): For each relation type $r$, node $i$ attends solely to its neighbors under relation $r$ via a masked, scaled dot-product self-attention. The result is a set of relation-specific node embeddings $h_i^r$.
• Relation-Level Attention (Inter-Relation): The set $\{h_i^r \mid r\in R_i\}$ is then aggregated for each node $i$ using a Transformer-style self-attention mechanism. This fuses the relation-specific embeddings into a final node representation $h_i$.

This hierarchical attention design generalizes GAT’s additive neighborhood attention and Transformer’s multiplicative attention, enabling BR-GCN to model both intra-relation (node-node) and inter-relation (relation-relation) dependencies in large-scale heterogeneous graphs.

2. Mathematical Formulation

Given a directed, labeled heterogeneous graph $G=(V, E, R)$ with node feature matrix $X \in \mathbb{R}^{|V| \times d}$:

• Node-Level Attention: For node $i$ and relation $i$0, compute:
  • $i$1 (query), $i$2 (key), and $i$3 (value).
  • Masked attention restricts attention to $i$4:

  $i$5

  $i$6 - Aggregate to obtain $i$7.

• Relation-Level Attention: Project each $i$8 to new query/key/value triples. Compute inter-relation attention:

  $i$9

  $r$0

Finally, sum over relations:

$r$1

By replacing standard R-GCN aggregation sums with these attention-weighted mechanisms, BR-GCN universally extends relational graph convolution with expressive bi-level attention.

3. Training Objectives and Implementation Considerations

BR-GCN supports both node-level and edge-level supervision:

• Node Classification: Typically two BR-GCN layers, followed by a softmax layer and cross-entropy optimization:

  $r$2

• Link Prediction: Embedding vectors from BR-GCN are passed to knowledge-graph embedding decoders (ComplEx, DistMult, TransE) with negative sampling and logistic loss:

  $r$3

Typical hyperparameters include 16 hidden units, dropout rates of 0.4–0.6, LeakyReLU slopes 0.2–0.8, and Adam optimizer. Efficient implementations utilize batching and sparse tensor representations in PyTorch Geometric or DGL.

4. Empirical Performance and Ablation Analysis

In benchmark evaluations, BR-GCN yields significant accuracy gains on both node classification and link prediction:

Dataset	BR-GCN Accuracy	R-GCN Accuracy	GAT Accuracy	Gain (vs. R-GCN)
AIFB	96.97%	95.83%	92.50%	+1.14%
MUTAG	81.13%	73.23%	66.18%	+7.90%
BGS	88.30%	83.10%	77.93%	+5.20%
AM	92.57%	89.29%	88.52%	+3.28%

On link prediction (FB15k, WN18), BR-GCN as encoder improves filtered MRR scores by 0.02–0.07 over R-GCN baselines, with further improvements when paired with ComplEx decoders.

Ablation studies show both node-level and relation-level attention contribute substantially: removing either results in an accuracy drop (node-only or relation-only variants underperform). Using only the most attended relations, as identified by relation-level attention, retains high task performance, indicating these scores capture edge importance effectively (Iyer et al., 2024).

5. Computational Complexity and Scalability

BR-GCN’s per-layer computational cost per node is dominated by:

• $r$4 for projection operations.
• $r$5 for masked self-attention, matching the scaling of GAT and R-GCN.

Memory usage is $r$6 due to per-relation projections and attention intermediates. The model supports efficient mini-batch training and scales linearly with the total number of edges and relation types. Sparse-matrix and batched implementations are fully supported.

6. Transferability, Modularity, and Extensions

The modular bi-level attention design allows the intra-relation (node-level) aggregator to be replaced with other GNN mechanisms (e.g., GraphSAGE, GIN) or augmented with multi-head attention. The relation-level attention weights yield interpretable importance scores, which support:

• Subgraph and meta-path selection strategies,
• Cross-architecture transfer: using BR-GCN’s attention scores to guide training or edge pruning in other GNNs,
• Integration with dynamic graph tasks, multi-hop reasoning, or cross-domain recommendation.

Future directions include exploring extensions to temporal graphs, leveraging hierarchical attention for multi-hop question answering, and cross-domain graph transfer learning (Iyer et al., 2024).

7. Comparison to Other Bi-Level Attention GNNs

Bi-Level Attention Graph Neural Networks (BA-GNN) employ a closely related hierarchical attention mechanism, but with additive node-level and multiplicative relation-level attentions. Both BA-GNN and BR-GCN demonstrate that the bi-level scheme achieves superior expressivity in modeling both entity and relation-level dependencies in heterogeneous graphs. BA-GNN reports consistent outperformance of R-GCN and other strong baselines, with empirical ablations underscoring the importance of both levels of attention (Iyer et al., 2023). In both frameworks, learned relation-level attention can be used to enhance transferability and graph compression for other GNN-based models.