---
title: 'PFGA: Partially Frozen Graph Attention'
url: https://www.emergentmind.com/topics/partially-frozen-graph-attention-pfga
type: topic
---

# PFGA: Partially Frozen Graph Attention

Partially Frozen Graph Attention (“PFGA”; *Editor’s term*) denotes, in the literature considered here, a family of graph-attention design strategies in which only part of the overall system remains trainable while attention behavior is stabilized, constrained, or selectively preserved. The most literal PFGA-like instance is an AASIST-style anti-spoofing architecture in which the entire Wav2Vec 2.0 XLS-R 300M encoder is frozen while the downstream graph/attention backend remains trainable [2507.11777]. Closely related but non-identical work controls attention by regularizing attention coefficients rather than freezing them [1811.00181], by applying dimension-wise gates and learnable temperature inside graph attention layers [2605.29803], or by freezing the query/key block after warm-up so that the attention kernel remains fixed while the value block continues to adapt [2605.06446]. This suggests that PFGA is best understood not as a single canonical operator, but as a design space centered on stabilizing neighborhood weighting, preserving reliable subspaces, and limiting harmful attention drift.

## 1. Terminological scope and conceptual boundaries

A central terminological caution is that PFGA is not the formal method name used by the cited papers. The label is exact only in an interpretive sense. In "Towards Scalable AASIST: Refining Graph Attention for Speech Deepfake Detection" [2507.11777], the model is partially frozen at the whole-model level because the SSL front end is frozen while the downstream AASIST modules remain trainable. In "FedFrozen: Two-Stage Federated Optimization via Attention Kernel Freezing" [2605.06446], the freezing target is the query/key block that determines the attention kernel, but the paper is not graph-specific. In "A Regularized Attention Mechanism for Graph Attention Networks" [1811.00181] and "Gated Graph Attention Networks with Learnable Temperature" [2605.29803], attention is controlled without literal freezing.

| Work | Mechanism | Relation to PFGA |
|---|---|---|
| [2507.11777] | Frozen XLS-R encoder, trainable graph/attention backend | PFGA-like at system level |
| [2605.06446] | Warm-up, then freeze Q/K and optimize V | Kernel-freezing blueprint |
| [1811.00181] | Attention regularization on \(\mathbf{A}^k\) | Soft-control comparator |
| [2605.29803] | Dimension-wise gating and learnable temperature | Soft selective suppression |

The conceptual commonality is control over attention flexibility. In the regularized GAT setting, all attention parameters remain trainable but the loss biases the learned coefficients. In the gated-temperature setting, unreliable coordinates are attenuated and attention sharpness is adjusted. In the federated kernel-freezing setting, the subspace responsible for attention geometry is fixed after warm-up. A plausible implication is that PFGA spans a continuum from hard parameter freezing to soft activation-level preservation.

## 2. Architectural decompositions underlying PFGA-style models

The clearest system-level PFGA formulation appears in the ASVspoof 5 anti-spoofing model of [2507.11777]. Its integration pipeline is
\[
\text{waveform} \rightarrow \text{Wav2Vec 2.0 XLS-R} \rightarrow \text{1024-d contextual embeddings} \rightarrow \text{MLP adapter} \rightarrow \text{AASIST backend}.
\]
The baseline backend is described as six ResNet blocks, split into spectral and temporal branches, branch-wise graph processing, a heterogeneous stacking graph-attention layer (HS-GAL), graph pooling, a second HS-GAL, max-graph-out fusion, and a linear classifier. In the refined system, the entire Wav2Vec 2.0 XLS-R 300M encoder is frozen throughout, and only the adapter, ResNet blocks, attention layers and classifier receive gradient updates. The original bespoke pair-wise graph-attention blocks are replaced with canonical multi-head self-attention, while cross-modal interaction uses separate linear projections for temporal and spectral queries and keys, with a shared value projection [2507.11777].

The more explicit parameter-level decomposition appears in FedFrozen [2605.06446], which writes the attention module as
\[
O(H;W^Q,W^K,W^V)=\varphi(HW^Q)\varphi(HW^K)^\top HW^V.
\]
Here \(\Phi=(W^Q,W^K)\) determines the attention kernel,
\[
A_\Phi(H)=\varphi(HW^Q)\varphi(HW^K)^\top,
\]
and \(\theta=W^V\) defines the value block,
\[
V(H;\theta)=H\theta.
\]
This yields
\[
O(H;\Phi,\theta)=A_\Phi(H)\,V(H;\theta).
\]
The paper’s direct claim is that freezing the query/key block fixes the attention kernel and its induced representation geometry, so that later value-block updates take place within a stable representation space [2605.06446].

A cautious graph-attention interpretation follows directly from the paper’s own transfer discussion. In graph Transformers, the correspondence is direct: \(W^Q,W^K\) determine node-node or edge-conditioned attention scores, and \(W^V\) transforms the node features being aggregated. In GAT-style layers of the form
\[
h_i' = \sum_{j\in \mathcal{N}(i)} \alpha_{ij} W h_j,
\]
the closest analogue is that the parameters determining \(\alpha_{ij}\) form the kernel block, while \(W\) plays the value-block role [2605.06446]. This suggests a PFGA decomposition into frozen neighborhood-weighting parameters and trainable message-transformation parameters.

## 3. Soft attention control as a conceptual neighbor of freezing

The regularized GAT model of [1811.00181] is important because it diagnoses a specific failure mode of vanilla GAT on unweighted graphs: attention often becomes nearly uniform over the closed neighborhood. For node \(v_i\), with transformed feature \(\tilde{\mathbf{x}}_i\), the attention score is
\[
e_{ij} = \left\langle \mathrm{Att}, \tilde{\mathbf{x}}_i \,\|\, \tilde{\mathbf{x}}_j \right\rangle,
\]
the normalized coefficient is
\[
a_{ij} = \frac{\exp(e_{ij})}{\sum_{k \in \mathcal{N}_c(v_i)} \exp(e_{ik})},
\]
and the update is
\[
\tilde{\mathbf{z}}_i = \sigma\!\left( \sum_{j\in\mathcal{N}_c(v_i)} a_{ij}\,\tilde{\mathbf{x}}_j \right).
\]
The paper argues that such near-uniform attention makes GAT vulnerable to heterogeneous rogue nodes and structural noise [1811.00181].

To counter this, it adds two regularizers to the original GAT loss:
\[
L_{\text{total}} = L_{\text{GAT}} + \lambda_1 L_{\text{excl}} - \lambda_2 L_{\text{nonunif}}.
\]
The exclusivity term,
\[
L_{\text{excl}} = \frac{1}{NK} \sum_{k=1}^{K}\sum_{j=1}^{N}\sum_{i=1}^{N} |A^k_{ij}|,
\]
penalizes excessive global influence, while the non-uniformity term,
\[
L_{\text{nonunif}} = \frac{1}{K}\sum_{k=1}^{K}\frac{1}{N}\sum_{i=1}^{N} \left(\|\mathbf{A}^k_i\|_0 - \deg(v_i)\right),
\]
encourages fewer participating neighbors. The discrepancy score
\[
d_i = \frac{\|A_{i,:} - U_i\|_1}{\deg(v_i)}
\]
quantifies deviation from uniform attention. The important boundary condition is that this method does not freeze attention coefficients or attention parameters; it shapes attention through soft penalties [1811.00181].

A second soft-control line appears in [2605.29803], where graph attention is modified by dimension-wise gating and learnable temperature. Standard neighborhood attention is written as
\[
\alpha_{ij}^{(l)} = \frac{ \exp(e_{ij}^{(l)}) }{ \sum_{r\in\mathcal N(i)\cup\{i\}} \exp(e_{ir}^{(l)}) }.
\]
Temperature introduces
\[
\alpha_{ij}^{(l)} = \frac{ \exp\!\left(e_{ij}^{(l)}/T^{(l)}\right) }{ \sum_{r\in\mathcal N(i)\cup\{i\}} \exp\!\left(e_{ir}^{(l)}/T^{(l)}\right) },
\qquad
T^{(l)}=\operatorname{softplus}(\theta^{(l)})+\epsilon,
\]
while the gate is
\[
g_i^{(l)} = \operatorname{sigmoid}\!\left( W_g^{(l)} h_i^{(l)} + b_g^{(l)} \right).
\]
In Post-Gate attention,
\[
h_i^{(l+1)} = \left( \sum_{j\in\mathcal N(i)\cup\{i\}} \alpha_{ij}^{(l)} W^{(l)} h_j^{(l)} \right) \odot g_i^{(l)},
\]
whereas Gate-First attention applies the gate before both scoring and aggregation:
\[
e_{ij}^{(l)} = \hat s^{(l)}\!\left( W^{(l)} h_i^{(l)} \odot g_i^{(l)},\; W^{(l)} h_j^{(l)} \odot g_j^{(l)} \right).
\]
Gate-First is the closest soft analogue to PFGA because coordinates with low gate values are prevented from strongly participating in both logits and propagated messages [2605.29803].

## 4. Optimization regimes and theoretical interpretations

FedFrozen formalizes a two-stage optimization schedule [2605.06446]. During warm-up, all parameters are trained for \(T_{\rm warm}\) rounds. After that, the kernel block is frozen at
\[
\bar{\Phi} := \Phi^{T_{\rm warm}},
\]
and only the value block is updated:
\[
\theta_{k,j+1}^t = \theta_{k,j}^t - \eta \nabla_\theta F_k^\lambda(\bar{\Phi},\theta_{k,j}^t).
\]
The associated profile objective is
\[
h_\lambda(\Phi):=\min_\theta f_\lambda(\Phi,\theta),
\]
and the regularized freezing bias is
\[
B_t^\lambda := h_\lambda(\Phi^t)-f_\lambda^\star.
\]
The paper interprets warm-up as inexact descent on this kernel-profile objective, and the frozen stage as restricted value-block optimization under a fixed attention kernel [2605.06446].

The core trade-off is explicit. Freezing too early creates freezing bias because the kernel is undertrained. Freezing too late allows client drift to corrupt the kernel block and leaves fewer rounds for stable value-block optimization. Proposition 1 gives the end-to-end decomposition
\[
f(w_{\tau,T})-f^\star \le B_\tau^{\lambda_0} + \rho_{\lambda_0}^{T-\tau}R_\tau^{\lambda_0} + CB_\theta,
\]
with \(B_\tau^{\lambda_0}\) the freezing bias and \(R_\tau^{\lambda_0}\) the value-block residual at freezing time [2605.06446]. In the paper’s linear-attention setting with \(\ell_2\)-regularization on the value block, fixing \(\Phi\) makes the problem linear in \(\theta\), and the second stage becomes smooth and strongly convex in \(\theta\).

The theoretical support for soft PFGA-like behavior is strengthened by [2605.29803]. Under global Gaussian noise,
\[
h_i = y_i\mu + \sigma \varepsilon_i,
\qquad
\varepsilon_i\sim \mathcal N(0,I_d),
\]
Theorem 1 states
\[
\sup_{T>0}\mathrm{SNR}(T)\ge \mathrm{SNR}(1),
\]
and in the high-noise limit \(\mathrm{SNR}(T)>\mathrm{SNR}(1)\) for sufficiently large \(T\). Under coordinate-missing noise,
\[
h_{i\ell} = r_{i\ell}\, y_i\mu_\ell + (1-r_{i\ell})\xi_{i\ell},
\]
with oracle gate \(g_{i\ell}=r_{i\ell}\), the class-separation logit gap is preserved,
\[
\Delta_{\hat e} = \Delta_e = 2(1-\rho)^2 \sum_{\ell=1}^d w_\ell|\mu_\ell|,
\]
while the gated variance is bounded independently of \(\tau\):
\[
\mathrm{Var}(\hat e_{ij}\mid y_i,y_j) \le 4\sum_{\ell=1}^d w_\ell^2\mu_\ell^2 =:C_{\mathrm g}.
\]
This is not freezing in a literal parameter sense, but it gives a formal rationale for preserving reliable coordinates and suppressing unreliable ones before they distort attention [2605.29803].

## 5. Empirical behavior across tasks and benchmarks

The most explicit PFGA-like performance evidence is reported on ASVspoof 5 in [2507.11777]. The freezing ablation is sharp: Baseline AASIST reaches **27.58% EER**, the trainable Wav2Vec front-end reaches **21.67% EER**, the frozen Wav2Vec front-end reaches **8.76% EER**, and the full proposed modifications reach **7.66% EER**. The graph-attention replacement itself is beneficial but smaller in effect: **8.76% EER** for baseline AASIST + frozen Wav2Vec, **8.43% EER** when MHA replaces bespoke graph attention, and **7.66% EER** for the full proposed modifications. The fusion ablation is similarly incremental: **8.43% EER** for frozen Wav2Vec + MHA + heuristic max fusion and **7.93% EER** when the max is replaced by learnable MHA fusion. The system is trained for **20 epochs** with **NAdam**, learning rate \(1 \times 10^{-4}\), **cosine annealing** with \(T_{max}=300\), global batch size **48**, on **4 × NVIDIA V100 GPUs**, with **~16 hours** total training time and **≤16 GB per GPU** [2507.11777].

In the robust GAT study of [1811.00181], experiments are performed on **Cora** and **Citeseer** for transductive semi-supervised node classification. Structured noise is introduced by sampling \(n_0\) validation nodes uniformly at random without replacement, deleting their existing edges, and adding \(m_0\) arbitrary edges chosen randomly from the entire graph. For Cora, \(n_0\) varies from \(10\) to \(60\) with \(m_0=100\); for Citeseer, \(n_0\) varies from \(40\) to \(80\) with \(m_0=500\). Each \(n_0\) is evaluated with **20 independent realizations**, and the paper reports average test accuracy. The main empirical claim is that the robust variant outperforms baseline GAT as the number of noisy nodes increases, while also producing larger discrepancy values \(d_i\), meaning less uniform and more selective attention [1811.00181].

The gated-temperature GAT results in [2605.29803] extend the argument to both homogeneous and heterophilic heterogeneous settings. On homogeneous benchmarks, examples include **Citeseer: \(69.21\) vs \(68.61\)** for Gated vs GAT, **OGBN-Products: \(80.62\) vs \(78.87\)** for Gated vs GAT, and **Cora: \(81.13\) vs \(80.53\)** for Gated\_v2 vs GATv2. On heterophilic heterogeneous benchmarks, examples include **Pokec: Gated\_v2 \(76.21\) vs GATv2 \(75.22\), GAT \(71.13\)** and **Mag-year: Gated\_temp\_v2 \(38.66\) vs GATv2 \(37.96\), GAT \(37.08\)**. Controlled noise studies show that learned temperature tends to increase as global Gaussian feature noise becomes stronger, while mean gate activation tends to decrease under coordinate-missing noise, especially in shallow layers [2605.29803].

FedFrozen provides non-graph empirical evidence for the kernel-freezing principle [2605.06446]. On heterogeneous federated training with ViT backbones, it is reported as stable and competitive, and its communication-cost ratios relative to FedAvg include **ViT-B/32: \(0.8712\)**, **T5-Small: \(0.8752\)**, **BART-Base: \(0.8780\)**, **ELECTRA-Base: \(0.8959\)**, and **BERT-Base: \(0.8964\)**. This suggests that freezing the kernel block can reduce communication by roughly 10% or more after freezing.

## 6. Misconceptions, limitations, and likely research directions

A frequent misconception is that PFGA necessarily means freezing part of the graph attention operator itself. The ASVspoof 5 model of [2507.11777] does not support that stricter interpretation. Its partially frozen status is at the system level: the entire SSL encoder is frozen, while the downstream graph/attention backend remains trainable. It is explicitly not partial layerwise freezing of Wav2Vec, not selective block tuning, and not staged unfreezing.

A second misconception is that regularization or gating is equivalent to freezing. The robust GAT of [1811.00181] keeps all attention parameters trainable and modifies the objective instead. The gated-temperature model of [2605.29803] modulates activations and logits through soft, node-dependent mechanisms. Both are highly relevant to PFGA, but neither is literal partial freezing.

The literature also contains reproducibility gaps. The regularized GAT paper does not state the exact multi-head output merge rule, does not explain how the \(\ell_0\)-based term is optimized, and omits several base GAT optimization details in the text [1811.00181]. The AASIST refinement does not report the number of attention heads, attention hidden size, exact adapter dimensions, dropout values, normalization type, or exact graph/node counts [2507.11777]. FedFrozen does not provide theory for standard nonlinear softmax attention in full Transformer form, graph attention networks, explicit personalization, or alternatives such as freezing only \(Q\), only \(K\), or other submodules [2605.06446].

These limitations delimit what can be claimed about PFGA as a formal research area. The strongest supported conclusion is narrower: attention control improves robustness or stability when the uncontrolled mechanism is vulnerable to uniform weighting, unreliable coordinates, noisy logits, or heterogeneous drift. This suggests that future PFGA formulations could profitably investigate explicit frozen-versus-trainable partitions of heads, channels, logits, or scoring parameters; warm-up-then-freeze schedules for neighborhood-weight generators; and hybrid designs that combine hard kernel stabilization with soft gating or temperature control. Such directions are inferential rather than directly established by the present papers, but they follow closely from the reported mechanisms and results [2605.06446].

Source: https://www.emergentmind.com/topics/partially-frozen-graph-attention-pfga