PFGA: Partially Frozen Graph Attention

• PFGA is a framework that partitions attention mechanisms into frozen components (kernel) and trainable components (value) to regulate neighborhood weighting.
• It utilizes methods like encoder freezing, regularization, gating, and temperature modulation to balance stability and adaptability in graph models.
• Empirical studies in tasks such as ASVspoof 5 and node classification demonstrate significant improvements in error rates and robustness.

Partially Frozen Graph Attention (“PFGA”; Editor’s term) denotes, in the literature considered here, a family of graph-attention design strategies in which only part of the overall system remains trainable while attention behavior is stabilized, constrained, or selectively preserved. The most literal PFGA-like instance is an AASIST-style anti-spoofing architecture in which the entire Wav2Vec 2.0 XLS-R 300M encoder is frozen while the downstream graph/attention backend remains trainable (Viakhirev et al., 15 Jul 2025). Closely related but non-identical work controls attention by regularizing attention coefficients rather than freezing them (Shanthamallu et al., 2018), by applying dimension-wise gates and learnable temperature inside graph attention layers (Ma et al., 28 May 2026), or by freezing the query/key block after warm-up so that the attention kernel remains fixed while the value block continues to adapt (Du et al., 7 May 2026). This suggests that PFGA is best understood not as a single canonical operator, but as a design space centered on stabilizing neighborhood weighting, preserving reliable subspaces, and limiting harmful attention drift.

1. Terminological scope and conceptual boundaries

A central terminological caution is that PFGA is not the formal method name used by the cited papers. The label is exact only in an interpretive sense. In "Towards Scalable AASIST: Refining Graph Attention for Speech Deepfake Detection" (Viakhirev et al., 15 Jul 2025), the model is partially frozen at the whole-model level because the SSL front end is frozen while the downstream AASIST modules remain trainable. In "FedFrozen: Two-Stage Federated Optimization via Attention Kernel Freezing" (Du et al., 7 May 2026), the freezing target is the query/key block that determines the attention kernel, but the paper is not graph-specific. In "A Regularized Attention Mechanism for Graph Attention Networks" (Shanthamallu et al., 2018) and "Gated Graph Attention Networks with Learnable Temperature" (Ma et al., 28 May 2026), attention is controlled without literal freezing.

Work	Mechanism	Relation to PFGA
(Viakhirev et al., 15 Jul 2025)	Frozen XLS-R encoder, trainable graph/attention backend	PFGA-like at system level
(Du et al., 7 May 2026)	Warm-up, then freeze Q/K and optimize V	Kernel-freezing blueprint
(Shanthamallu et al., 2018)	Attention regularization on $\mathbf{A}^k$	Soft-control comparator
(Ma et al., 28 May 2026)	Dimension-wise gating and learnable temperature	Soft selective suppression

The conceptual commonality is control over attention flexibility. In the regularized GAT setting, all attention parameters remain trainable but the loss biases the learned coefficients. In the gated-temperature setting, unreliable coordinates are attenuated and attention sharpness is adjusted. In the federated kernel-freezing setting, the subspace responsible for attention geometry is fixed after warm-up. A plausible implication is that PFGA spans a continuum from hard parameter freezing to soft activation-level preservation.

2. Architectural decompositions underlying PFGA-style models

The clearest system-level PFGA formulation appears in the ASVspoof 5 anti-spoofing model of (Viakhirev et al., 15 Jul 2025). Its integration pipeline is

$$\text{waveform} \rightarrow \text{Wav2Vec 2.0 XLS-R} \rightarrow \text{1024-d contextual embeddings} \rightarrow \text{MLP adapter} \rightarrow \text{AASIST backend}.$$

The baseline backend is described as six ResNet blocks, split into spectral and temporal branches, branch-wise graph processing, a heterogeneous stacking graph-attention layer (HS-GAL), graph pooling, a second HS-GAL, max-graph-out fusion, and a linear classifier. In the refined system, the entire Wav2Vec 2.0 XLS-R 300M encoder is frozen throughout, and only the adapter, ResNet blocks, attention layers and classifier receive gradient updates. The original bespoke pair-wise graph-attention blocks are replaced with canonical multi-head self-attention, while cross-modal interaction uses separate linear projections for temporal and spectral queries and keys, with a shared value projection (Viakhirev et al., 15 Jul 2025).

The more explicit parameter-level decomposition appears in FedFrozen (Du et al., 7 May 2026), which writes the attention module as

$$O(H;W^Q,W^K,W^V)=\varphi(HW^Q)\varphi(HW^K)^\top HW^V.$$

Here $\Phi=(W^Q,W^K)$ determines the attention kernel,

$A_\Phi(H)=\varphi(HW^Q)\varphi(HW^K)^\top,$

and $\theta=W^V$ defines the value block,

$V(H;\theta)=H\theta.$

This yields

$O(H;\Phi,\theta)=A_\Phi(H)\,V(H;\theta).$

The paper’s direct claim is that freezing the query/key block fixes the attention kernel and its induced representation geometry, so that later value-block updates take place within a stable representation space (Du et al., 7 May 2026).

A cautious graph-attention interpretation follows directly from the paper’s own transfer discussion. In graph Transformers, the correspondence is direct: $W^Q,W^K$ determine node-node or edge-conditioned attention scores, and $W^V$ transforms the node features being aggregated. In GAT-style layers of the form

$$\text{waveform} \rightarrow \text{Wav2Vec 2.0 XLS-R} \rightarrow \text{1024-d contextual embeddings} \rightarrow \text{MLP adapter} \rightarrow \text{AASIST backend}.$$0

the closest analogue is that the parameters determining $$\text{waveform} \rightarrow \text{Wav2Vec 2.0 XLS-R} \rightarrow \text{1024-d contextual embeddings} \rightarrow \text{MLP adapter} \rightarrow \text{AASIST backend}.$$1 form the kernel block, while $$\text{waveform} \rightarrow \text{Wav2Vec 2.0 XLS-R} \rightarrow \text{1024-d contextual embeddings} \rightarrow \text{MLP adapter} \rightarrow \text{AASIST backend}.$$2 plays the value-block role (Du et al., 7 May 2026). This suggests a PFGA decomposition into frozen neighborhood-weighting parameters and trainable message-transformation parameters.

3. Soft attention control as a conceptual neighbor of freezing

The regularized GAT model of (Shanthamallu et al., 2018) is important because it diagnoses a specific failure mode of vanilla GAT on unweighted graphs: attention often becomes nearly uniform over the closed neighborhood. For node $$\text{waveform} \rightarrow \text{Wav2Vec 2.0 XLS-R} \rightarrow \text{1024-d contextual embeddings} \rightarrow \text{MLP adapter} \rightarrow \text{AASIST backend}.$$3, with transformed feature $$\text{waveform} \rightarrow \text{Wav2Vec 2.0 XLS-R} \rightarrow \text{1024-d contextual embeddings} \rightarrow \text{MLP adapter} \rightarrow \text{AASIST backend}.$$4, the attention score is

$$\text{waveform} \rightarrow \text{Wav2Vec 2.0 XLS-R} \rightarrow \text{1024-d contextual embeddings} \rightarrow \text{MLP adapter} \rightarrow \text{AASIST backend}.$$5

the normalized coefficient is

$$\text{waveform} \rightarrow \text{Wav2Vec 2.0 XLS-R} \rightarrow \text{1024-d contextual embeddings} \rightarrow \text{MLP adapter} \rightarrow \text{AASIST backend}.$$6

and the update is

$$\text{waveform} \rightarrow \text{Wav2Vec 2.0 XLS-R} \rightarrow \text{1024-d contextual embeddings} \rightarrow \text{MLP adapter} \rightarrow \text{AASIST backend}.$$7

The paper argues that such near-uniform attention makes GAT vulnerable to heterogeneous rogue nodes and structural noise (Shanthamallu et al., 2018).

To counter this, it adds two regularizers to the original GAT loss: $$\text{waveform} \rightarrow \text{Wav2Vec 2.0 XLS-R} \rightarrow \text{1024-d contextual embeddings} \rightarrow \text{MLP adapter} \rightarrow \text{AASIST backend}.$$8 The exclusivity term,

$$\text{waveform} \rightarrow \text{Wav2Vec 2.0 XLS-R} \rightarrow \text{1024-d contextual embeddings} \rightarrow \text{MLP adapter} \rightarrow \text{AASIST backend}.$$9

penalizes excessive global influence, while the non-uniformity term,

$$O(H;W^Q,W^K,W^V)=\varphi(HW^Q)\varphi(HW^K)^\top HW^V.$$0

encourages fewer participating neighbors. The discrepancy score

$$O(H;W^Q,W^K,W^V)=\varphi(HW^Q)\varphi(HW^K)^\top HW^V.$$1

quantifies deviation from uniform attention. The important boundary condition is that this method does not freeze attention coefficients or attention parameters; it shapes attention through soft penalties (Shanthamallu et al., 2018).

A second soft-control line appears in (Ma et al., 28 May 2026), where graph attention is modified by dimension-wise gating and learnable temperature. Standard neighborhood attention is written as

$$O(H;W^Q,W^K,W^V)=\varphi(HW^Q)\varphi(HW^K)^\top HW^V.$$2

Temperature introduces

$$O(H;W^Q,W^K,W^V)=\varphi(HW^Q)\varphi(HW^K)^\top HW^V.$$3

while the gate is

$$O(H;W^Q,W^K,W^V)=\varphi(HW^Q)\varphi(HW^K)^\top HW^V.$$4

In Post-Gate attention,

$$O(H;W^Q,W^K,W^V)=\varphi(HW^Q)\varphi(HW^K)^\top HW^V.$$5

whereas Gate-First attention applies the gate before both scoring and aggregation: $$O(H;W^Q,W^K,W^V)=\varphi(HW^Q)\varphi(HW^K)^\top HW^V.$$6 Gate-First is the closest soft analogue to PFGA because coordinates with low gate values are prevented from strongly participating in both logits and propagated messages (Ma et al., 28 May 2026).

4. Optimization regimes and theoretical interpretations

FedFrozen formalizes a two-stage optimization schedule (Du et al., 7 May 2026). During warm-up, all parameters are trained for $$O(H;W^Q,W^K,W^V)=\varphi(HW^Q)\varphi(HW^K)^\top HW^V.$$7 rounds. After that, the kernel block is frozen at

$$O(H;W^Q,W^K,W^V)=\varphi(HW^Q)\varphi(HW^K)^\top HW^V.$$8

and only the value block is updated: $$O(H;W^Q,W^K,W^V)=\varphi(HW^Q)\varphi(HW^K)^\top HW^V.$$9 The associated profile objective is

$\Phi=(W^Q,W^K)$0

and the regularized freezing bias is

$\Phi=(W^Q,W^K)$1

The paper interprets warm-up as inexact descent on this kernel-profile objective, and the frozen stage as restricted value-block optimization under a fixed attention kernel (Du et al., 7 May 2026).

The core trade-off is explicit. Freezing too early creates freezing bias because the kernel is undertrained. Freezing too late allows client drift to corrupt the kernel block and leaves fewer rounds for stable value-block optimization. Proposition 1 gives the end-to-end decomposition

$\Phi=(W^Q,W^K)$2

with $\Phi=(W^Q,W^K)$3 the freezing bias and $\Phi=(W^Q,W^K)$4 the value-block residual at freezing time (Du et al., 7 May 2026). In the paper’s linear-attention setting with $\Phi=(W^Q,W^K)$5-regularization on the value block, fixing $\Phi=(W^Q,W^K)$6 makes the problem linear in $\Phi=(W^Q,W^K)$7, and the second stage becomes smooth and strongly convex in $\Phi=(W^Q,W^K)$8.

The theoretical support for soft PFGA-like behavior is strengthened by (Ma et al., 28 May 2026). Under global Gaussian noise,

$\Phi=(W^Q,W^K)$9

Theorem 1 states

$A_\Phi(H)=\varphi(HW^Q)\varphi(HW^K)^\top,$0

and in the high-noise limit $A_\Phi(H)=\varphi(HW^Q)\varphi(HW^K)^\top,$1 for sufficiently large $A_\Phi(H)=\varphi(HW^Q)\varphi(HW^K)^\top,$2. Under coordinate-missing noise,

$A_\Phi(H)=\varphi(HW^Q)\varphi(HW^K)^\top,$3

with oracle gate $A_\Phi(H)=\varphi(HW^Q)\varphi(HW^K)^\top,$4, the class-separation logit gap is preserved,

$A_\Phi(H)=\varphi(HW^Q)\varphi(HW^K)^\top,$5

while the gated variance is bounded independently of $A_\Phi(H)=\varphi(HW^Q)\varphi(HW^K)^\top,$6: $A_\Phi(H)=\varphi(HW^Q)\varphi(HW^K)^\top,$7 This is not freezing in a literal parameter sense, but it gives a formal rationale for preserving reliable coordinates and suppressing unreliable ones before they distort attention (Ma et al., 28 May 2026).

5. Empirical behavior across tasks and benchmarks

The most explicit PFGA-like performance evidence is reported on ASVspoof 5 in (Viakhirev et al., 15 Jul 2025). The freezing ablation is sharp: Baseline AASIST reaches 27.58% EER, the trainable Wav2Vec front-end reaches 21.67% EER, the frozen Wav2Vec front-end reaches 8.76% EER, and the full proposed modifications reach 7.66% EER. The graph-attention replacement itself is beneficial but smaller in effect: 8.76% EER for baseline AASIST + frozen Wav2Vec, 8.43% EER when MHA replaces bespoke graph attention, and 7.66% EER for the full proposed modifications. The fusion ablation is similarly incremental: 8.43% EER for frozen Wav2Vec + MHA + heuristic max fusion and 7.93% EER when the max is replaced by learnable MHA fusion. The system is trained for 20 epochs with NAdam, learning rate $A_\Phi(H)=\varphi(HW^Q)\varphi(HW^K)^\top,$8, cosine annealing with $A_\Phi(H)=\varphi(HW^Q)\varphi(HW^K)^\top,$9, global batch size 48, on 4 × NVIDIA V100 GPUs, with ~16 hours total training time and ≤16 GB per GPU (Viakhirev et al., 15 Jul 2025).

In the robust GAT study of (Shanthamallu et al., 2018), experiments are performed on Cora and Citeseer for transductive semi-supervised node classification. Structured noise is introduced by sampling $\theta=W^V$0 validation nodes uniformly at random without replacement, deleting their existing edges, and adding $\theta=W^V$1 arbitrary edges chosen randomly from the entire graph. For Cora, $\theta=W^V$2 varies from $\theta=W^V$3 to $\theta=W^V$4 with $\theta=W^V$5; for Citeseer, $\theta=W^V$6 varies from $\theta=W^V$7 to $\theta=W^V$8 with $\theta=W^V$9. Each $V(H;\theta)=H\theta.$0 is evaluated with 20 independent realizations, and the paper reports average test accuracy. The main empirical claim is that the robust variant outperforms baseline GAT as the number of noisy nodes increases, while also producing larger discrepancy values $V(H;\theta)=H\theta.$1, meaning less uniform and more selective attention (Shanthamallu et al., 2018).

The gated-temperature GAT results in (Ma et al., 28 May 2026) extend the argument to both homogeneous and heterophilic heterogeneous settings. On homogeneous benchmarks, examples include Citeseer: $V(H;\theta)=H\theta.$2 vs $V(H;\theta)=H\theta.$3 for Gated vs GAT, OGBN-Products: $V(H;\theta)=H\theta.$4 vs $V(H;\theta)=H\theta.$5 for Gated vs GAT, and Cora: $V(H;\theta)=H\theta.$6 vs $V(H;\theta)=H\theta.$7 for Gated_v2 vs GATv2. On heterophilic heterogeneous benchmarks, examples include Pokec: Gated_v2 $V(H;\theta)=H\theta.$8 vs GATv2 $V(H;\theta)=H\theta.$9, GAT $O(H;\Phi,\theta)=A_\Phi(H)\,V(H;\theta).$0 and Mag-year: Gated_temp_v2 $O(H;\Phi,\theta)=A_\Phi(H)\,V(H;\theta).$1 vs GATv2 $O(H;\Phi,\theta)=A_\Phi(H)\,V(H;\theta).$2, GAT $O(H;\Phi,\theta)=A_\Phi(H)\,V(H;\theta).$3. Controlled noise studies show that learned temperature tends to increase as global Gaussian feature noise becomes stronger, while mean gate activation tends to decrease under coordinate-missing noise, especially in shallow layers (Ma et al., 28 May 2026).

FedFrozen provides non-graph empirical evidence for the kernel-freezing principle (Du et al., 7 May 2026). On heterogeneous federated training with ViT backbones, it is reported as stable and competitive, and its communication-cost ratios relative to FedAvg include ViT-B/32: $O(H;\Phi,\theta)=A_\Phi(H)\,V(H;\theta).$4, T5-Small: $O(H;\Phi,\theta)=A_\Phi(H)\,V(H;\theta).$5, BART-Base: $O(H;\Phi,\theta)=A_\Phi(H)\,V(H;\theta).$6, ELECTRA-Base: $O(H;\Phi,\theta)=A_\Phi(H)\,V(H;\theta).$7, and BERT-Base: $O(H;\Phi,\theta)=A_\Phi(H)\,V(H;\theta).$8. This suggests that freezing the kernel block can reduce communication by roughly 10% or more after freezing.

6. Misconceptions, limitations, and likely research directions

A frequent misconception is that PFGA necessarily means freezing part of the graph attention operator itself. The ASVspoof 5 model of (Viakhirev et al., 15 Jul 2025) does not support that stricter interpretation. Its partially frozen status is at the system level: the entire SSL encoder is frozen, while the downstream graph/attention backend remains trainable. It is explicitly not partial layerwise freezing of Wav2Vec, not selective block tuning, and not staged unfreezing.

A second misconception is that regularization or gating is equivalent to freezing. The robust GAT of (Shanthamallu et al., 2018) keeps all attention parameters trainable and modifies the objective instead. The gated-temperature model of (Ma et al., 28 May 2026) modulates activations and logits through soft, node-dependent mechanisms. Both are highly relevant to PFGA, but neither is literal partial freezing.

The literature also contains reproducibility gaps. The regularized GAT paper does not state the exact multi-head output merge rule, does not explain how the $O(H;\Phi,\theta)=A_\Phi(H)\,V(H;\theta).$9-based term is optimized, and omits several base GAT optimization details in the text (Shanthamallu et al., 2018). The AASIST refinement does not report the number of attention heads, attention hidden size, exact adapter dimensions, dropout values, normalization type, or exact graph/node counts (Viakhirev et al., 15 Jul 2025). FedFrozen does not provide theory for standard nonlinear softmax attention in full Transformer form, graph attention networks, explicit personalization, or alternatives such as freezing only $W^Q,W^K$0, only $W^Q,W^K$1, or other submodules (Du et al., 7 May 2026).

These limitations delimit what can be claimed about PFGA as a formal research area. The strongest supported conclusion is narrower: attention control improves robustness or stability when the uncontrolled mechanism is vulnerable to uniform weighting, unreliable coordinates, noisy logits, or heterogeneous drift. This suggests that future PFGA formulations could profitably investigate explicit frozen-versus-trainable partitions of heads, channels, logits, or scoring parameters; warm-up-then-freeze schedules for neighborhood-weight generators; and hybrid designs that combine hard kernel stabilization with soft gating or temperature control. Such directions are inferential rather than directly established by the present papers, but they follow closely from the reported mechanisms and results (Du et al., 7 May 2026).