Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory

Abstract: Despite the widespread adoption of Vision Transformers (ViTs) and their success across numerous computer vision applications, the fundamental understanding of their dimensional and representational geometry remains relatively underexplored. To address this gap, we introduce Transformer Geometry Observatory (TGO), a systematic framework of experiments and analysis pipelines designed to investigate the representational geometry and dynamics of Vision Transformers. TGO-I, the first installment of the framework, focuses on the spectral geometry of ViT representations. Using a ViT-Small/16 model trained on ImageNet-100, we analyze Effective Rank, Stable Rank, Participation Ratio, Spectral Entropy, Spectral Flatness, Spectral Anisotropy, covariance structure, eigenspectra, and singular value spectra throughout training. Our results reveal a consistent increase in dimensional utilization, accompanied by decreasing anisotropy, increasing spectral entropy, increasing participation ratio, and progressively flatter eigenspectra. Contrary to the common intuition that training should concentrate information into a small number of dominant directions, we observe a progressive redistribution of variance across representational dimensions. This phenomenon is particularly pronounced in the final CLS token representation, which exhibits the highest effective dimensionality and lowest anisotropy within the network.

• The paper demonstrates that Vision Transformer representations evolve toward higher effective rank and redistributed variance.
• It employs eight spectral observables to analyze covariance matrices, capturing dynamic changes across layers and training epochs.
• Findings indicate that decorrelation in the CLS token enhances model interpretability and reduces feature redundancy.

Spectral Geometry Evolution in Vision Transformers: Insights from the Transformer Geometry Observatory TGO-I

Context and Motivation

Transformer-based architectures, especially Vision Transformers (ViTs), have demonstrated pronounced success in visual recognition tasks, yet the underlying representational geometry through the training process has remained under-characterized. The Transformer Geometry Observatory (TGO) framework addresses this gap by proposing a systematic methodology to instrument and analyze the geometric evolution of representations in ViT models. TGO-I, the initial component, focuses specifically on spectral geometry, leveraging an array of spectral observables to interrogate the statistical structure and dynamics of internal representations.

Methodological Approach

TGO-I centers its analysis on the ViT-Small/16 model trained on ImageNet-100. Representations were extracted from major architectural elements—Patch Embedding, Positional Embedding, Transformer blocks, and the final CLS token—across 100 epochs. The framework utilizes forward hooks for consistent activation extraction over a fixed subset of validation images, ensuring observational invariance. Analytical focus is placed on covariance matrices of these activations, from which eight spectral observables are derived: Effective Rank, Stable Rank, Participation Ratio, Spectral Entropy, Spectral Flatness, Spectral Anisotropy, Eigenspectra, and Singular Value Spectra.

These observables, based on eigendecomposition and normalization, quantify dimensional utilization, variance distribution, and concentration. Scalar and visual spectral measures permit detailed tracking of representational dynamics both across layers and as a function of training time.

Figure 1: Effective Rank evolution across 100 epochs for all layers, showing increasing dimensional utilization, especially pronounced for the CLS token.

Empirical Results

Dimensional Utilization and Rank Metrics

Throughout training, the Effective Rank rises consistently in all layers, with marked growth in deeper layers and the CLS token. Patch and Positional embeddings remain relatively stable, indicating limited geometric evolution prior to transformer processing. These trends are accompanied by increases in Stable Rank and Participation Ratio, signaling that variance is shared across many spectral directions rather than concentrating in a few dominant axes.

Spectral Anisotropy and Entropy Dynamics

Spectral Anisotropy, measuring concentration onto principal spectral directions, declines substantially, particularly in transformer blocks and the final CLS token. Simultaneously, Spectral Entropy, reflecting uniformity in the eigenvalue distribution, demonstrates persistent growth across training epochs.

Figure 2: Spectral Anisotropy declines across layers, reflecting diminished dominance of principal eigen-directions.

Figure 3: CLS Covariance at epoch 100 reveals weak off-diagonal correlations and strong decorrelation, consistent with maximal effective rank.

These transitions point to a progressive shift from concentrated to distributed representational geometry, characterized by flatter eigenspectra and weaker feature correlations.

Covariance and Spectral Visualization

Qualitative inspection of covariance matrices, eigenspectra, spectral decay, and singular value distributions further corroborate the quantitative findings. Over the course of training, covariance matrices evolve toward strong diagonal dominance with off-diagonal entries diminishing, indicative of increased decorrelation between features and less redundancy.

Figure 4: Progressive evolution of CLS representation (epochs 20, 50, 100): covariance structure becomes diagonal, eigenspectra flatten, spectral concentration decreases.

Singular value spectra similarly broaden, with meaningful contributions from larger fractions of the embedding space.

Theoretical Implications

Variance Redistribution

Contradicting conventional intuition, the findings indicate that training does not compress representational information into a small set of dominant directions; instead, variance is increasingly redistributed across more dimensions. This is most pronounced in the global CLS token, whose geometry becomes highly distributed and exhibits the lowest spectral anisotropy.

Hypotheses: Mechanisms Underlying Observed Geometry

Three explanations are posited:

1. Token Diversification: Representation dimensionality may increase as tokens become more distinctive, leading to broader variance allocation.
2. Semantic Expansion: The model could be encoding a larger set of semantically meaningful directions, reflected in rising spectral entropy and participation ratio.
3. Redundancy Reduction: The observed decorrelation may signify reduction in feature redundancy, resulting in more efficient use of the embedding space.

Each hypothesis is consistent with empirical trends but requires direct token-level or semantic analysis in future observatories.

Practical Consequences and Future Directions

The systematic redistribution of variance and decorrelation observed suggests implications for interpretability, compression, and potential architectural modifications—e.g., favoring distributed representations for global tokens. The methodology establishes a foundation for deeper mechanistic exploration, including studies of token dynamics, layer similarity, attention structure, and optimization geometry.

The TGO framework will extend into observatories addressing representation similarity (CKA, SVCCA, PWCCA), token-level dynamics (cosine similarity, token covariance), attention geometry (attention entropy, sparsity), and optimization geometry (gradient and Hessian structure). These expansions aim to triangulate the causal factors responsible for spectral evolution and provide actionable insights for future transformer design and analysis.

Conclusion

TGO-I introduces a robust, reproducible observational framework for analyzing spectral geometry in Vision Transformers. The main empirical outcome is that representations, especially the CLS token, evolve toward distributed, decorrelated spectral profiles with high effective rank and spectral entropy. These findings challenge the assumption of variance concentration post-training and highlight the necessity of multi-faceted observational approaches in unraveling deep learning dynamics. The modular and extensible nature of TGO sets the stage for future causal studies linking geometry, semantics, and optimization in transformer architectures.