--- title: Brain-Informed Graph Transformer Readout url: https://www.emergentmind.com/topics/brain-informed-graph-transformer-readout-bigtr type: topic --- # Brain-Informed Graph Transformer Readout Brain-Informed Graph Transformer Readout (BIGTR) is a modular component within the Brain Tokenized Graph Transformer (Brain TokenGT) framework, developed for longitudinal brain functional connectome (FC) embedding. Its primary purpose is to provide interpretable, graph-level readout for multitype tokenized embeddings representing brain regions, spatial edges, and temporal edges derived from longitudinal fMRI data, specifically aimed at applications such as neurodegenerative disease diagnosis and prognosis [2307.00858]. ## 1. Role within the Brain TokenGT Pipeline BIGTR operates as a downstream module, ingesting embeddings produced by the Graph Invariant and Variant Embedding (GIVE) module. GIVE generates the following tokenized vectors: - Node embeddings $x_v$ for each brain region $v$ at each time point $t$. - Spatial-edge embeddings $x_{(u,v)}$ for edges within each FC at each time point. - Temporal-edge embeddings $x_{(v,v')}$ that connect a node $v$ at time $t$ with itself at time $t+1$. BIGTR's function is to process this heterogeneous collection of tokens, infuse them with spatio-temporal context using uniquely designed identifier embeddings, project them into a unified embedding space, introduce an explicit [graph] token, and read out a graph-level representation via a Transformer encoder. The final representation is used for classification tasks such as MCI vs. Control, MCI conversion prediction, and amyloid status discrimination [2307.00858]. ## 2. Token Construction and Augmentation Each input token is represented as a concatenation of its original embedding, a type identifier, and node identifiers. Let $h$ denote the dimensionality of the GIVE embeddings. The augmentation mechanism is as follows: - **Type Identifiers:** A learnable matrix $P \in \mathbb{R}^{3 \times d_p}$ contains three distinct vectors $P_v$ (for node tokens), $P_{e_s}$ (for spatial-edge tokens), $P_{e_t}$ (for temporal-edge tokens). - **Node Identifiers:** A fixed matrix $Q \in \mathbb{R}^{(M \cdot T) \times d_q}$ contains one-hot or orthonormal vectors for each brain region--time pair. Token augmentations: - For node token $x_v$, form $z_v = [x_v \;\|\, P_v \;\|\, Q_{(t,v)} \;\|\, Q_{(t,v)}] \in \mathbb{R}^{h + d_p + 2d_q}$. - For spatial-edge token $x_{(u,v)}$, form $z_{(u,v)} = [x_{(u,v)} \;\|\, P_{e_s} \;\|\, Q_{(t,u)} \;\|\, Q_{(t,v)}]$. - For temporal-edge token $x_{(v,v')}$ (linking $v_t\rightarrow v_{t+1}$), form $z_{(v,v')} = [x_{(v,v')} \;\|\, P_{e_t} \;\|\, Q_{(t,v)} \;\|\, Q_{(t+1,v)}]$. All such augmented tokens are stacked into $Z \in \mathbb{R}^{N_{\text{tokens}} \times (h + d_p + 2d_q)}$. A learnable projection $\omega \in \mathbb{R}^{(h + d_p + 2d_q) \times h'}$ is applied: $Z_{\text{proj}} = Z \cdot \omega$. A learnable "[graph]" token $X_{[\mathrm{graph}]} \in \mathbb{R}^{h'}$ is prepended to form the Transformer input $Z^{\mathrm{in}} = [X_{[\mathrm{graph}]};\, Z_{\text{proj}}] \in \mathbb{R}^{(1 + N_{\text{tokens}}) \times h'}$ [2307.00858, Eqs. (4), (5)]. ## 3. Transformer Encoder and Self-Attention Readout The Transformer encoder processes the input $Z^{\mathrm{in}}$ through $L$ layers and $H$ attention heads. For each layer $\ell = 1 \ldots L$ and head $k = 1 \ldots H$: \[ \begin{align*} Q_{(k)} &= Z^{(\ell-1)} W^Q_{(k)} \in \mathbb{R}^{(1 + N_{\text{tokens}}) \times d_k} \\ K_{(k)} &= Z^{(\ell-1)} W^K_{(k)} \\ V_{(k)} &= Z^{(\ell-1)} W^V_{(k)} \\ A_{(k)} &= \mathrm{softmax}\left(\frac{Q_{(k)} K_{(k)}^T}{\sqrt{d_k}}\right) \\ H_{(k)} &= A_{(k)} V_{(k)} \end{align*} \] Outputs across heads are concatenated and linearly transformed with $W^O$. Each layer applies Add & Norm, a position-wise feed-forward network (FFN), and another Add & Norm: \[ \begin{align*} Z' &= \mathrm{LayerNorm}\left(Z^{(\ell-1)} + H^{(\ell)}\right) \\ Z^{(\ell)} &= \mathrm{LayerNorm}(Z' + \mathrm{FFN}(Z')) \end{align*} \] After $L$ layers, the hidden state corresponding to the [graph] token, $h_{\mathrm{graph}} = Z^{(L)}[0]$, is adopted as the graph-level embedding. ## 4. Readout Function and Classification The final graph-level embedding $h_{\mathrm{graph}}$ undergoes a linear transformation and a sigmoid to produce the predicted probability for the class of interest: \[ \hat y = \sigma(W_{\mathrm{out}} h_{\mathrm{graph}} + b_{\mathrm{out}}) \] The model is trained using binary cross-entropy loss for downstream tasks [2307.00858]. ## 5. Spatio-Temporal Context Encoding Unlike standard Transformer architectures, BIGTR does not employ sinusoidal or learned position embeddings. Instead, spatio-temporal context is directly encoded through the non-trainable node identifier $Q_{(t,v)}$, providing explicit information on both the brain region and time. The type identifier $P$ conveys token category, ensuring the model can disambiguate between node, spatial-edge, and temporal-edge tokens. This explicit design encodes spatial, temporal, and type-specific token context without the need for position embeddings [2307.00858]. ## 6. Interpretability Features BIGTR supports interpretability through two avenues: - Token-level attention visualization is afforded by extracting the attention weights associated with the [graph] token. Specifically, for each head $k$, the row corresponding to the [graph] token in $A_{(k)}$ indicates which input tokens most strongly inform the graph-level representation. Averaging over heads and layers enables inspection of the ROIs and connections prioritized by the model on each instance. - Because tokens are augmented with explicit region, time, and type identifiers, attention diagnostics directly map to neurobiologically interpretable entities. This has enabled identification of salient regions such as parahippocampal edges in MCI conversion cases [2307.00858]. ## 7. Empirical Performance and Hyperparameterization In five-fold cross-validation (repeated five times), the full GIVE + BIGTR pipeline demonstrated superior performance compared to standard global pooling and ablated variants on longitudinal fMRI datasets: | Task | AUC (%) | Accuracy (%) | |----------------------------------------------|-------------|---------------| | MCI vs. Control (ADNI HC vs. MCI) | 90.48 (±4.99)| 84.62 (±8.43) | | MCI Conversion Prediction (OASIS-3) | 87.14 (±7.16)| 89.23 (±7.84) | | Amyloid + vs. – Classification (OASIS-3) | 94.60 (±4.96)| 87.11 (±7.88) | These results indicate a substantial gain attributed to the richer token-level readout enabled by BIGTR. Default hyperparameters include: $L=2$ Transformer layers, $H=8$ attention heads, $h'=128$ post-projection hidden size, $d_p=16$ type-ID dimension, $d_q=16$ node-ID dimension, feed-forward inner dimension of 512, and dropout of 0.1 on both attention and FFN modules [2307.00858]. Source: https://www.emergentmind.com/topics/brain-informed-graph-transformer-readout-bigtr