Geometric Expressivity Gap in Models

• Geometric expressivity gap is the rigorous separation in model capabilities defined by geometric invariants like polyhedral region counts, tropical mappings, and manifold curvature.
• It employs analyses from tropical geometry, polyhedral formulations, and symmetry-aware graph learning to reveal how architectural modifications alter expressivity.
• This concept illustrates that changes such as attention gating, equivariance, and quantum parameterization yield significant shifts in the geometric structures underlying model performance.

“Geometric expressivity gap” denotes a family of formally distinct but closely related separations between model classes once their representations are interpreted geometrically. In recent work, the gap is quantified through polyhedral partitions and linear-region counts in tropical models of transformers and Mixture-of-Experts, through symmetry-respecting distinguishability in geometric graph learning, through intrinsic curvature of statistical manifolds induced by attention, through quotient-manifold approximation rates in equivariant diffusion models, and through manifold dimension, margin, gap, and diameter in quantum and geodesic optimization settings (Su et al., 16 Apr 2026, Su et al., 3 Feb 2026, Joshi et al., 2023, Bathula et al., 16 Apr 2026, Perera et al., 20 May 2026, Shao et al., 3 Apr 2026, Franks et al., 2021). This suggests that the term functions as an umbrella notion for rigorous differences in what architectures can realize once “expressivity” is measured by geometry rather than by parameter count alone.

1. Formal meanings of the gap

The term is not used with a single universal definition. In transformer theory, geometric expressivity is the number of maximal linear regions induced by attention, multi-head aggregation, and feed-forward refinement; the gap is then a separation in polyhedral complexity and region counts between architectures such as single-head and multi-head self-attention (Su et al., 16 Apr 2026). In tropical analyses of MoE, the gap is the rigorously quantified difference in geometric and topological expressivity between dense networks and Top-$k$ routing, measured by region counts in ambient space and on low-dimensional manifolds (Su et al., 3 Feb 2026). In geometric GNN theory, the gap is the difference in which geometric graphs can be distinguished while respecting permutations, rotations, reflections, and translations, with GWL and its variants serving as upper bounds on model classes (Joshi et al., 2023). In gated attention, the gap is a separation between intrinsically flat Fisher–Rao manifolds realizable by ungated attention and non-flat, even positively curved, manifolds realizable by multiplicative gating (Bathula et al., 16 Apr 2026).

Setting	Compared objects	Gap quantity
Transformers	SHA, MHSA, shallow, deep	Newton-polytope complexity, maximal linear regions (Su et al., 16 Apr 2026)
MoE	Dense, Top-1, Top-$k$	$\binom{N}{k}$-scaled region counts, effective capacity (Su et al., 3 Feb 2026)
Geometric graph learning	Invariant, equivariant, low/high body-order	GWL/GSWL distinguishability classes (Joshi et al., 2023, Wang et al., 7 May 2026)
Attention geometry	Ungated, gated	Fisher–Rao curvature of representation manifolds (Bathula et al., 16 Apr 2026)
Equivariant diffusion	Non-equivariant, equivariant	Representation gap $R(\Omega,\Omega_f)$, intrinsic dimension (Perera et al., 20 May 2026)
Geodesic and quantum settings	Full, truncated, or ill-conditioned geometries	Effective dimension, margin, gap, diameter (Shao et al., 3 Apr 2026, Franks et al., 2021)

A common structural feature across these formulations is that expressivity is attached to a geometric object: a normal fan, a hypersimplex, a statistical manifold, a quotient manifold, or a reachable orbit manifold. The “gap” then records how changing heads, routing, equivariance, gating, or algebraic structure alters that object.

2. Tropical and polyhedral formulations

In the tropical analysis of transformers, self-attention is modeled as a structured vector-valued tropical rational map. In the zero-temperature limit, the attention partition of query space is algebraically equivalent to a Power Voronoi Diagram generated by the keys, so each attention region is a convex polyhedron where one key wins (Su et al., 16 Apr 2026). This turns “geometric expressivity” into the number of maximal linear regions of the resulting CPWL map. The first major gap then appears between single-head and multi-head attention. A single head has Newton polytope vertex count $V_{\text{single}} \le N$, whereas multi-head self-attention, via the Minkowski sum of head-wise Newton polytopes, has $V_{\text{multi}} = \mathcal{O}(N^H)$ in the standard regime $H \le d_{\text{model}}$. The same framework yields the asymptotically tight transformer scaling law

$$\mathcal{N}(\mathcal{T}) = \Theta\!\bigl(N^{d_{\text{model}}L}\bigr)$$

for the number of full-dimensional linear regions when $H \ge d_{\text{model}}$ and $d_{\text{model}}, d_{\text{ff}}, L$ are fixed (Su et al., 16 Apr 2026). Within this formalism, the gap is therefore combinatorial and geometric: more heads, larger embedding dimension, and greater depth increase the complexity of the induced polyhedral complex.

The same tropical logic produces a parallel result for sparsely routed architectures. In Top-$k$0 MoE, routing is algebraically identical to evaluation of the $k$1-th elementary symmetric tropical polynomial, and the routing partition is the normal fan of the hypersimplex $k$2 pulled back to input space (Su et al., 3 Feb 2026). This yields a single-layer capacity

$k$3

compared with $k$4 for a dense layer and $k$5 for Top-1 routing. On a data manifold $k$6 of intrinsic dimension $k$7, the corresponding effective capacity becomes

$k$8

whereas dense models collapse to $k$9 (Su et al., 3 Feb 2026). The paper terms this persistence of the binomial factor under manifold restriction “Combinatorial Resilience,” and the contrasting dense behavior “capacity collapse.”

A broader tropical background is provided by work on ReLU networks as tropical Puiseux rational maps, where linear regions are identified with polyhedra arising from numerator and denominator fans, exact region counts can be computed symbolically, and quantities such as Hoffman constants bound the minimal sampling radius needed to intersect all regions (Lezeau et al., 2024). This does not itself define a single gap, but it supplies the polyhedral toolkit used in later gap theorems.

3. Symmetry-aware discrimination gaps in geometric learning

In geometric graph learning, the gap is usually phrased as a separation in discrimination power. The Geometric Weisfeiler–Leman test assigns each node both an invariant color and an equivariant geometric object, and it upper-bounds the expressive power of $\binom{N}{k}$0-equivariant geometric GNNs (Joshi et al., 2023). Within this framework, a strict gap appears between invariant and equivariant message passing. IGWL and invariant geometric GNNs cannot distinguish 1-hop identical geometric graphs no matter how many layers are used, while GWL and equivariant architectures can propagate geometric orientation beyond one hop and therefore distinguish a strictly larger class. A second gap concerns local scalarization: $\binom{N}{k}$1 is at least as powerful as $\binom{N}{k}$2, and for $\binom{N}{k}$3 the hierarchy is strict, so 2-body, 3-body, 4-body, and higher-order scalarizations define genuinely different local geometric expressivities (Joshi et al., 2023). A third gap concerns tensor order: vector-only equivariant layers fail on high-fold rotational symmetries that higher-order spherical tensors can resolve.

Sparse geometric MPNNs introduce a related but distinct separation. For connected sparse geometric graphs, the paper proves that message-passing networks with rotation-equivariant intermediate features can generically separate pairs of non-isomorphic geometric graphs as long as the underlying graph is connected, while models restricted to invariant intermediate features require the stronger condition of generic global rigidity (Sverdlov et al., 2024). This converts sparsity into a precise geometric condition: equivariant intermediate transport suffices under connectedness, but invariant transport alone needs rigidity to avoid information loss.

In simplicial learning, GSWL extends SWL by inserting coordinates into the initial colors of simplices, producing a geometry-aware refinement procedure. Geometry-aware simplicial message passing is then upper-bounded by GSWL and, on any fixed finite family of embedded simplicial complexes, can be matched by suitable parameters. Combined with the Euler Characteristic Transform, this yields a geometric expressivity characterization for embedded complexes and exposes a strict gap between combinatorial simplicial models, which only see connectivity, and geometry-aware models, which can distinguish different embeddings of the same abstract complex (Wang et al., 7 May 2026).

A further symmetry-sensitive gap appears in the choice of geometric algebra for equivariant transformers. Euclidean GA is computationally cheap but only carries $\binom{N}{k}$4 natively and cannot represent absolute positions as first-class geometric objects; naive projective GA is degenerate and, without the join, cannot realize all multilinear maps nor use inner-product attention to encode distances between points; conformal GA, and the improved projective construction with join and CGA-based attention, recover richer $\binom{N}{k}$5-equivariant multilinear expressivity and distance-aware attention (Haan et al., 2023). Persistent homology yields a complementary gap: with suitable filtrations, PH matches the discriminative power of the WL hierarchy, is strictly more expressive than 1-WL, and empirically separates some graph pairs that defeat higher-order WL tests, thereby exposing a topological expressivity deficit in purely message-passing local refinements (Ballester et al., 2023).

4. Curvature, projective geometry, and manifold approximation

A more intrinsic geometric definition of the gap appears in attention layers once outputs are interpreted as mean parameters of Gaussian families endowed with the Fisher–Rao metric. Under this model, ungated attention outputs are affine combinations of fixed value vectors, so the induced statistical manifold has constant metric and zero Riemann curvature. Multiplicative gating breaks this affine restriction. The paper proves that ungated attention can realize only intrinsically flat manifolds, while gated attention can realize non-flat manifolds, including a patch of the unit sphere with Gaussian curvature $\binom{N}{k}$6, whereas the corresponding ungated construction has $\binom{N}{k}$7 (Bathula et al., 16 Apr 2026). It also identifies a structured regime in which curvature accumulates under composition, yielding a depth amplification effect with curvature scaling quadratically in depth.

MöbiusAttention advances a related claim from a different geometric starting point. Tokens and positions are lifted to complex vectors $\binom{N}{k}$8, queries are obtained by elementwise Möbius transformations

$\binom{N}{k}$9

and the resulting attention mechanism operates in complex projective geometry (Halacheva et al., 2024). The paper argues that standard attention is “predominantly linear,” whereas MöbiusAttention can realize circular, elliptic, hyperbolic, parabolic, and loxodromic geometries, and empirically observes head- and layer-level specialization into these geometry types. It is explicit, however, that the expressivity claim is qualitative and empirical rather than a formal universality or separation theorem (Halacheva et al., 2024).

A related manifold-based formalism is the representation gap

$R(\Omega,\Omega_f)$0

which measures how well a model’s prediction space approximates the true data manifold (Perera et al., 20 May 2026). In equivariant diffusion models, the asymptotic law

$R(\Omega,\Omega_f)$1

is governed by a single parameter, the intrinsic dimension of the task, where $R(\Omega,\Omega_f)$2 in the non-equivariant case and $R(\Omega,\Omega_f)$3 for $R(\Omega,\Omega_f)$4-equivariant models (Perera et al., 20 May 2026). This is not phrased as a gap between two fixed architectures, but it creates a rigorous geometric separation between models that exploit quotient-manifold structure and models that do not.

5. Quantum and geodesic variants of the gap

In quantum machine learning, expressivity is reinterpreted geometrically through the reachable manifold of a parameterized quantum circuit inside projective Hilbert space. The circuit generators define a Lie subalgebra $R(\Omega,\Omega_f)$5, and the Fubini–Study metric on the reachable manifold yields an effective geometric dimension

$R(\Omega,\Omega_f)$6

The paper proves that $R(\Omega,\Omega_f)$7, so expressivity is controlled by generator structure rather than raw parameter count, and establishes the scaling law

$R(\Omega,\Omega_f)$8

with an exponential barren-plateau regime when $R(\Omega,\Omega_f)$9 becomes large (Shao et al., 3 Apr 2026). The resulting geometric expressivity gap is a separation between highly expressive circuits that suffer concentration-of-measure-induced trainability collapse and structured Lie-truncated circuits that retain full metric rank while staying in a polynomial trainability regime.

A different barrier appears in geodesic optimization for scaling problems. Matrix scaling and operator scaling admit polynomial-time analyses based on diameter bounds and margin or gap parameters, but multidimensional array scaling, tensor scaling, and polynomial scaling do not. The paper constructs polynomial-size instances of 3-dimensional array scaling and 3-tensor scaling whose approximate solutions all have doubly exponential condition number, proves exponential lower bounds on the diameter of approximate solution sets, and shows that margin and gap are exponentially small for array scaling, tensor scaling, and polynomial scaling (Franks et al., 2021). In this setting, the “gap” is not an advantage of one architecture over another but a geometric barrier: current diameter-, margin-, and gap-based methods are not expressive enough as analyses to certify polynomial-time behavior for these problems.

6. Limitations, non-gaps, and open directions

The literature also emphasizes that not every geometric reformulation yields a strict separation. In transformer tropical geometry, the exact polyhedral theory is developed in the zero-temperature limit, but finite-temperature soft attention preserves the same topological partitions away from tie hyperplanes via exponentially tight bounds on function value, gradient, and curvature; accordingly, the framework states that there is no fundamental geometric expressivity gap between hard and soft attention at the level of polyhedral partition, only a difference between exact and approximate linearity (Su et al., 16 Apr 2026). Likewise, sliced ReLU attention shows that quasi-linear $V_{\text{single}} \le N$0 attention can match softmax attention on two strong notions of in-context expressivity, including contextual universal approximation; in that setting the conclusion is explicitly that there is no expressivity gap relative to softmax at the level of those theorems (Boufadène et al., 12 Dec 2025).

Several gap theorems are also qualified by idealizations. The transformer tropical results are worst-case, assume zero temperature for exact combinatorics, rely on generic-position conditions for maximal Minkowski complexity, and ignore LayerNorm or RMSNorm in the strict theory (Su et al., 16 Apr 2026). MoE effective-capacity results assume Top-$V_{\text{single}} \le N$1 routing, transversality, and the Manifold Hypothesis (Su et al., 3 Feb 2026). Geometry-aware simplicial message passing matches GSWL only on fixed finite families, while the ECT approximation theory on infinite classes requires bounded embeddings and stability assumptions (Wang et al., 7 May 2026). Persistent homology is highly filtration-dependent, and its strongest WL-comparison theorems are existential rather than constructive (Ballester et al., 2023). MöbiusAttention provides qualitative and empirical evidence for richer intra-layer geometry but explicitly lacks formal universal-approximation or strict-containment theorems (Halacheva et al., 2024).

Open directions follow naturally from these caveats. Transformer theory leaves open how much of the worst-case region complexity is realized in trained models and how normalization layers alter the geometry (Su et al., 16 Apr 2026). MoE theory points toward deep stacks, other routing polytopes, and direct topology measures such as Betti numbers rather than region counts (Su et al., 3 Feb 2026). Geometric graph learning continues to ask whether higher-order PH is strictly more expressive than $V_{\text{single}} \le N$2-WL for all $V_{\text{single}} \le N$3, and how learned filtrations compare with existential constructions (Ballester et al., 2023). Geodesic optimization seeks interior-point-like methods that do not rely on polynomial diameter bounds alone (Franks et al., 2021). The recurring pattern is that geometric expressivity gaps become sharpest when a model modification changes the geometry of partitions, orbits, or manifolds in a way that is both intrinsic and stable under the symmetries of the task.