Center-based Transformer Architecture

• Center-based Transformer architectures are defined by using central tokens or hubs to aggregate information, reducing computational complexity.
• Techniques like CP-ViT's core–periphery masking, Centroid Transformer's soft clustering, and CenterFormer’s heatmap querying illustrate distinct methods of token abstraction.
• These models achieve efficiency gains of up to 50-70% fewer attention connections while maintaining competitive accuracy and interpretability.

Center-based Transformer architectures constitute a family of models in which attention and representation mechanisms leverage centers, hubs, or summary tokens as information integrators, departing from standard fully-connected self-attention. These architectures are motivated by both biological neural network organization and algorithmic efficiency demands. Notable exemplars include the Core–Periphery Vision Transformer (CP-ViT), CenterFormer for 3D perception, and the Centroid Transformer, each operationalizing “center-based” mechanisms via distinct but related topologies and algorithmic constructs (Yu et al., 2023, Zhou et al., 2022, Wu et al., 2021).

1. Principles and Motivation

Center-based Transformer architectures are predicated on the notion that information can be more efficiently, robustly, and interpretably processed by structuring token interactions around distinguished “center” roles. Biological neural systems, especially human brain functional graphs, often display a core–periphery topology: densely connected hubs (core) mediate long- and short-range communication among sparsely connected peripheral regions. Similar centrality concepts appear in clustering and abstraction methods for representation learning. These principles are instantiated within center-based Transformer mechanisms to achieve competitive accuracy and improved resource usage, while retaining the modeling strengths of attention-based computation (Yu et al., 2023, Wu et al., 2021).

2. Core–Periphery and Centroid Graph Construction

Models such as CP-ViT and the Centroid Transformer formalize the center concept through explicit graph or abstraction routines:

• Core–Periphery Graphs (CP-ViT): The input token set $V = \{1, ..., n\}$ is partitioned into a core subset $C \subset V$, $|C|=m$, and periphery $P = V \setminus C$. The adjacency matrix $A \in \{0,1\}^{n \times n}$ is stochastically populated: core–core edges appear with probability $p_{cc}$ (dense), core–periphery with $p_{cp}$ (moderate), and periphery–periphery with $p_{pp}$ (sparse, often zero). The resulting binary mask $M_{CP}$ is used to constrain self-attention (Yu et al., 2023).
• Centroid Abstraction (Centroid Transformer): Given $N$ input embeddings $X \in \mathbb{R}^{N \times d}$, $M < N$ centroid vectors $U \in \mathbb{R}^{M \times d}$ are initialized (via random sampling, mean-pooling, or domain-specific selection). These centroids evolve using amortized gradient steps on a soft $k$-means–like clustering loss, yielding information-aggregating centers over which subsequent (compressed) attention is performed (Wu et al., 2021).
• Centralized Query Selection (CenterFormer): In CenterFormer, a learned center heatmap head predicts object-centric center candidates from a BEV feature map. These highest-scoring positions guide extraction and subsequent transformer query initialization, focusing attention and regression on object-relevant regions (Zhou et al., 2022).

Model	Center Mechanism	Graph/Selection Principle
CP-ViT	Core–Periphery Masked Attention	Probability-based graph over tokens
Centroid Transformer	Centroid Abstraction	Amortized k-means, soft clustering
CenterFormer	Center Heatmap Queries	CNN-based center proposal

3. Modified Self-Attention and Abstraction Operations

Center-based architectures reorganize attention computations for computational and inductive efficiency:

• CP-Masked Self-Attention: Standard dense self-attention is replaced by a masked version: attention logits and weights are computed only for allowed ($M_{CP}[i,j]=1$) token pairs, typically dense for core–core, moderate for core–periphery, and sparse elsewhere. Row-wise softmax ensures each token attends only across its permissible neighborhood (Yu et al., 2023).
• Centroid Attention Update: Centroid attention reduces $N$ inputs to $M$ centroids via $T$ unrolled amortized gradient steps on a soft clustering objective:

$$u_j^{t+1} = u_j^t + \varepsilon \sum_{\ell=1}^L \sum_{i=1}^N \mathrm{sim}_\ell(x_i,u_j^t) \cdot V_\ell(x_i,u_j^t)$$

where $\mathrm{sim}_\ell(x_i,u_j)$ is a softmax similarity, $V_\ell$ is the chosen value function, and $T$ is typically 1–2 (Wu et al., 2021).

• Center-founded Decoder Attention (CenterFormer): Only object-relevant centers extracted by the heatmap contribute queries to the transformer decoder. Cross-attention is implemented efficiently via localized (multi-scale) or deformable windows, sharply reducing attention cost relative to full quadratic self-attention (Zhou et al., 2022).

4. Core/Center Selection, Redistribution, and Adaptivity

Core node or center selection is both a hyperparameter choice and, in several models, an adaptive, data-driven process:

• Gradient-based Saliency (CP-ViT): After each epoch, patch importance is measured as the normalized sum of pre-softmax logit gradients, redistributing the top-ranked patches to core nodes in descending degree order. This ensures core nodes preferentially attend to task-relevant image regions, guided by gradient saliency signals (Yu et al., 2023).
• Initialization and Adaptation (Centroid Transformer): Centroids can be initialized via mean-pooling, random sampling, or data-driven procedures like farthest-point sampling. The number of centroids $M$ and iteration count $T$ are considered per-task hyperparameters. KNN-based sparsification further reduces computational load by restricting attention to top-$k$ relevant tokens (Wu et al., 2021).
• CNN-based Proposals (CenterFormer): The heatmap-based proposal mechanism is supervised by a focal loss matching ground-truth object centers with Gaussian peaks, trained to be both class-specific and spatially precise (Zhou et al., 2022).

5. Architectural Integration and Layer Composition

Center-based techniques are integrated into various transformer backbones without radical architectural departure:

• CP-ViT: Each multi-head self-attention (MHSA) block in a standard ViT backbone (e.g., ViT-S/16) is replaced with the masked CP-MHSA; overall layer normalization, MLP sublayers, positional embeddings, and residual connections are preserved (Yu et al., 2023).
• Centroid Transformer: Standard Transformer layers are interleaved with centroid-attention stages, reducing sequence length at each abstraction level and thus propagating efficiency gains downstream. The $N \rightarrow M$ map can occur early or at intermediate depth in the network (Wu et al., 2021).
• CenterFormer: Center-based decoder is inserted atop a VoxelNet or similar 3D convolutional encoder. The entire transformer pipeline consists of lightweight (2–3 layer) decoder-only blocks, omitted encoder, and efficient cross-attention fields (Zhou et al., 2022).

6. Computational Complexity and Empirical Performance

Center-based transformers achieve substantial computational savings with competitive or improved accuracy, attributed to selective focus and information abstraction.

• Complexity:
  • CP-ViT: Reduces the number of active attention connections by up to 70% (INbreast) while achieving minor top-1 accuracy improvements (0.3–0.8%) over vanilla ViT. The fractional connection ratio is formally tracked as $\mathrm{CR} = \|M_{CP}\|_1 / n^2$ (Yu et al., 2023).
  • Centroid Transformer: Shrinks per-layer attention cost from $O(N^2 d)$ to $O(N M d + M^2 d)$, yielding over 50% complexity reduction for practical $M$ (Wu et al., 2021).
  • CenterFormer: Transformer cost is $O(9 \cdot 3 \cdot N)$ for cross-attention (constant window size/locality), avoiding the $O((HW)^2)$ scaling of BEV-wide dense attention (Zhou et al., 2022).
• Empirical Results:

Model	Task/Dataset	Accuracy/Metric	Compute Ratio/Properties
CP-ViT	INbreast/CIFAR/TinyIM	+0.3%–0.8% over ViT	Up to 70% fewer attention pairs
CentroidTrans	Gigaword summariz.	+1.7 ROUGE-1; 50% MACs	Encoder cost halved at same/↑ quality
CenterFormer	Waymo 3D detection	73.7 mAPH (val)	Fast convergence; lowest MACs

Inferentially, selective abstraction leads to better feature selection and interpretability, as visualized by patch overlaps with expert heatmaps in CP-ViT (Yu et al., 2023), and batch-size and memory improvements in centroid-based point cloud reconstruction (Wu et al., 2021).

7. Interpretability, Biological Analogy, and Limitations

Biological analogy provides both explanatory and inductive rationale for these architectures:

• Core Hubs as Integrators: CP-ViT mirrors functional MRI-derived brain network graphs, where information is efficiently integrated through hubs. Gradient-based core redistributions result in core tokens aligning with saliency and object-of-interest locations (Yu et al., 2023).
• Clustering-Induced Abstraction: Centroid attention’s connection to clustering and mean-field energy approximations provides a statistical-mechanical and information-theoretic lens (Wu et al., 2021).
• Limitations:
  • Hyperparameter sensitivity (e.g., core/centroid count, step size, initialization method).
  • Fixed number of centers may not be optimal; dynamic adaptation remains an open area.
  • Applicability in autoregressive and decoder-only settings is nontrivial for centroid-based reductions (Wu et al., 2021).
  • While CP-ViT and CenterFormer demonstrate improved interpretability, generalization to other modalities or tasks may depend on the affordances of the center selection mechanism.

• CP-ViT, “Core–Periphery Principle Guided Redesign of Self-Attention in Transformers” (Yu et al., 2023).
• CenterFormer, “CenterFormer: Center-based Transformer for 3D Object Detection” (Zhou et al., 2022).
• Centroid Transformer, “Centroid Transformers: Learning to Abstract with Attention” (Wu et al., 2021).