Topology-Agnostic Transformers

• Topology-agnostic transformers are neural architectures that process data with variable, unknown, or non-Euclidean topologies using adaptive tokenization and encoding.
• They utilize novel attention mechanisms and positional encodings to facilitate robust inference across diverse domains such as graphs, manifolds, and sensor arrays.
• Their flexible design enables cross-domain transfer, reduces preprocessing demands, and achieves competitive performance on tasks from graph classification to motion retargeting.

Topology-agnostic transformers constitute a class of neural architectures designed to process data distributed over domains whose topology varies, is unknown, or is not expressible within the conventional Euclidean or sequential frameworks. These architectures generalize transformer models beyond fixed graphs or regular grids, instead enabling inference and representation learning that is robust to diverse, even unseen, topological structures. Such models have demonstrated efficacy on graphs, cell complexes, manifolds, time-varying sensor arrays, skeletal data, cellular automata, and more, through a combination of novel tokenization, positional encoding, attention mechanisms, and architectural abstractions.

1. Defining Topology-Agnosticism in Transformers

Topology-agnostic transformers are neural models that maintain predictive performance, transferability, and efficiency across domains with varying topology. Unlike standard transformers, which assume a sequential or grid-like token ordering, topology-agnostic models operate on arbitrary or variable topological structures without imposing task-specific rewiring or hard-coding domain information. Architectural ingredients typically include:

• Tokenizations or embeddings that do not depend on a fixed order or spatial arrangement.
• Positional encodings or geometric features that abstract away from domain-specific coordinates.
• Attention or routing mechanisms that adaptively incorporate topological, geometric, or incidence-based information.
• Losses and regularization that enforce invariance to topological variations.

Examples include the Riemannian mixture-of-experts projector for graphs (Jyothish et al., 9 Jul 2025), cross-attention set-abstraction over variable sensor arrays (Döner et al., 25 Oct 2025), spiral-curve ordering for manifolds (Maurin et al., 11 Jul 2025), cell complex attention (Ballester et al., 2024), and template-conditioned latent factorization in animation retargeting (Mourot et al., 2023).

2. Topology-Agnostic Embeddings and Input Representations

A core challenge is the projection of domain-specific inputs into a common latent space that is independent—or flexibly adaptive to—topology.

• Riemannian Mixture-of-Experts Projector: Each node is routed via a gating MLP to a mixture of constant-curvature manifolds (Euclidean, hyperbolic, spherical), with explicit exponential-map projection per manifold. The node’s embedding is a convex combination weighted by per-node routing scores, providing a geometry-aware, topology-adapted feature (Jyothish et al., 9 Jul 2025).
• Query-based Channel Unification: For data such as EEG, a cross-attention between learned queries and sensor channels maps arbitrary and variable-size channel sets onto a fixed-size latent set, removing dependency on sensor arrangement (Döner et al., 25 Oct 2025).
• Space-Filling Curves: For manifold domains lacking a global token order, a continuous space-filling curve (e.g., polar spiral on the 2-sphere) is sampled, and the sequence serves as input to the transformer. Positional encodings are derived from parametrization of the curve, preserving manifold connectivity without explicit adjacency encoding (Maurin et al., 11 Jul 2025).
• Topological Templates: For motion data, template vectors encoding joint positions and connectivity are concatenated with per-joint embeddings. This conditioning enables encoding/decoding of motions for skeletons with unseen or arbitrary topology (Mourot et al., 2023).
• Topology-Agnostic Tokenization: Flattening grid states (e.g., for cellular automata) into sequences combined with rotary positional embeddings allows GPT models to operate on arbitrary grid size and shape, retaining no hard-coded topological assumptions (Berkovich et al., 2024).

This design abstraction is summarized in the following table:

Methodology	Domain	Topology Handling Mechanism
Mixture-of-experts (Riemannian)	Graphs	Adaptive manifold projection, gating
Query cross-attention	Sensor arrays, EEG	Fixed latent set via learned queries
Space-filling curve	Manifolds	Curve-induced sequentialization
Template-based embedding	Motion, skeletons	Conditioning on neutral-pose template
Rotary PE + byte encoding	Cellular automata	Sequence agnostic to spatial order

3. Attention and Positional Encoding in Non-Euclidean and Variable Topologies

The self-attention mechanism must often be adapted for domains that lack canonical adjacency or invariant ordering. Major strategies include:

• Geometry/Incidence-aware Bias: The Cellular Transformer uses pairwise or general attention augmented with neighborhood or incidence bias matrices drawn from the algebraic topology of the cell complex (signed incidence, upper/lower adjacency), enabling joint processing of vertices, edges, and faces (Ballester et al., 2024).
• Spectral and Topological Encodings: Positional encodings are derived from barycentric subdivision Laplacians (BSPe), random walks (RWPe), or topological Slepians. These encode connectivity, neighborhood structure, and spectral characteristics in a topology-invariant manner, and improve the attention’s expressiveness in cell complexes, graphs, and molecules (Ballester et al., 2024).
• Learned/Parametric Positionality: Space-filling curve models parameterize PE via continuous coordinates along the curve (e.g., $t$ on the spiral), or via small MLPs mapping curve positions to PE vectors, effectively circumventing the lack of a canonical index (Maurin et al., 11 Jul 2025).
• Rotary and Implicit Positionality: Rotary positional embeddings (RoPE) encode relative position in a way that does not require explicit, absolute encoding of coordinates or sizes, aiding in agnosticism to sequence length and layout (Berkovich et al., 2024).

4. Training Objectives and Regularization

Topology-agnostic transformers are typically trained with domain-invariant or geometry-adapted objectives.

• Classification, Reconstruction, and Consistency Losses: Node classification in graph domains uses cross-entropy; masked patch reconstruction for EEG uses Smooth L1 loss; motion retargeting is trained with sequence MSE and bone-length temporal consistency to enforce plausible anatomical structure (Döner et al., 25 Oct 2025, Mourot et al., 2023).
• Regularizers for Topological and Geometric Consistency: Gating entropy regularizes mixture-assignment confidence; curvature penalties ensure expert parameters do not drift; orthogonality constraints are imposed on Grassmann or Stiefel manifold projections; specialization losses enforce diversity in learned queries (Jyothish et al., 9 Jul 2025, Döner et al., 25 Oct 2025).
• Topology-Agnostic Data Augmentation: Stochastic joint subsampling and reconditioning on different skeleton templates during training enforce topology-independence in latent representations (Mourot et al., 2023).

5. Empirical Results Across Domains

Topology-agnostic transformers achieve state-of-the-art or highly competitive results on a range of domains and datasets:

• Graphs and Cell Complexes: Riemannian mixture-of-experts projectors yield 1–3% absolute accuracy gain on benchmarks such as Cora, Citeseer, Airport, and PubMed, surpassing strong baselines including GCN, GAT, and manifold-aware GNNs (Jyothish et al., 9 Jul 2025).
• Cellular Complex Learning: The Cellular Transformer matches or exceeds leading GNNs and graph transformers on GCB, ZINC, and ogbg-molhiv, attaining 0.752 ± 0.010 accuracy on GCB, 0.080 MAE on ZINC, and competitive AUC on ogbg-molhiv, without special graph rewiring or virtual nodes (Ballester et al., 2024).
• EEG and Sensor Arrays: LUNA achieves 0.921 AUROC on TUAR and robust generalization across variable electrode configurations and unseen layouts, while reducing FLOPs by up to 300× and GPU memory by up to 10× compared to quadratic-complexity architectures (Döner et al., 25 Oct 2025).
• Cellular Automata Simulation: LifeGPT attains >99.9% single-step accuracy for Conway’s Game of Life on toroidal grids of unknown size/configuration, using rotary positional embeddings for topological agnosticism (Berkovich et al., 2024).
• Motion Retargeting: HuMoT’s autoencoder delivers 1.13 cm mean per-joint position error on seen skeletons, 3.09 cm on unseen; in cross-structural retargeting, it outperforms previous methods especially on topology transfer (Mourot et al., 2023).

6. Limitations and Open Research Questions

Despite significant advances, topology-agnostic transformers face several open problems:

• Computational Overhead: While attention mechanisms can be tailored for sparsity or linear complexity (e.g., LUNA's cross-attention, XCA, or set-abstraction), general $O(N^2)$ complexity remains a bottleneck for large or dense cell complexes (Ballester et al., 2024, Döner et al., 25 Oct 2025).
• Global Consistency and Manifold Gluing: Convex mixing of embeddings, as in Riemannian projectors, does not guarantee smooth manifold-wise transitions or “gluing,” which may be suboptimal for domains with highly heterogeneous or interacting topological patches (Jyothish et al., 9 Jul 2025).
• Adaptive and Learned Topology Parameters: Fixed expert sets, positional curves, or incidence structures may be insufficient for dynamically changing topologies or tasks requiring online adaptation. Learning expert geometries, manifold parametrizations, or curve orderings per node/sample remains a challenging direction (Maurin et al., 11 Jul 2025).
• Application Breadth: Most empirical evaluations focus on graph-based or strictly structured domains; broader adoption across domains such as biological tissues, dynamical fields, or generalized cellular automata remains emergent (Berkovich et al., 2024).
• Generalization Boundaries: While empirical evidence confirms cross-topology transfer, there are performance drops on highly out-of-distribution or sparsely sampled layouts (e.g., unseen 62-channel EEG configurations) (Döner et al., 25 Oct 2025). Understanding and mitigating such failures is ongoing research.

7. Connections, Broader Impact, and Future Directions

Topology-agnostic transformers bridge topological deep learning, geometric machine learning, and classical transformer architectures. Their ability to process data without explicit knowledge of the domain’s structure enables foundational models applicable to heterogeneous, multi-source, and dynamically changing domains (e.g., multimodal biosignals, universal scientific simulators, high-dimensional point clouds, arbitrary mesh/skeleton domains).

Broader implications include:

• Enabling foundation models that natively generalize across new sensor deployments, experimental configurations, or biological morphologies.
• Providing a unified modeling language for forward/inverse problems in CA, physical simulation, and complex systems.
• Facilitating transfer learning, data augmentation, and cross-domain representation mixing without topology-specific retraining or preprocessing.
• Suggesting further generalizations (e.g., adaptive spiral curves, online topology estimation, fully geometric self-attention, dynamic manifold learning).

In sum, topology-agnostic transformers represent a critical advance in the abstraction and universality of deep learning methods, marrying the expressive capacity of transformers with rigorous, geometric/topological flexibility (Jyothish et al., 9 Jul 2025, Döner et al., 25 Oct 2025, Ballester et al., 2024, Mourot et al., 2023, Maurin et al., 11 Jul 2025, Berkovich et al., 2024).