Expressibility Gap in Computational Models

Updated 3 July 2026

Expressibility Gap is the quantifiable difference in expressive power across computational systems, defining limits in deep learning, quantum circuits, and formal models.
It is measured by comparing theoretical baselines with empirical results using metrics like negative log-likelihood, KL divergence, and covering numbers.
Understanding the expressibility gap guides model selection, architecture design, and optimization strategies to mitigate inherent expressive limitations.

The expressibility gap refers to the systematic, often quantifiable difference between the theoretical expressive power or practical performance of different computational models, architectures, or parameterizations across a range of domains, most notably in deep learning, quantum algorithms, probabilistic circuits, formal language theory, and cyber-physical systems. In essence, the expressibility gap measures the limitations imposed by a specific model class or implementation constraint relative to a chosen baseline (often an idealized or more expressive system), thereby delimiting the set of functions, distributions, or behaviors that can be exactly or approximately realized.

1. Formal Definitions and Characterizations

The expressibility gap is always tied to a precise notion of expressivity: the ability of a system to represent, realize, or approximate functions, languages, distributions, or physical configurations.

Neural/LLMs: In the context of transformer LLMs, the expressibility gap is defined between the set of formal languages recognizable by practical, fixed-precision, strictly-masked, soft-attention transformers and those definable in richer logical or automata-theoretic frameworks. Specifically, the work (Li et al., 29 May 2025) proves that such transformers can realize exactly the class of languages definable by the past-only fragment of linear temporal logic (PTL), which sits strictly below the class of all star-free (i.e., aperiodic) languages. The gap is precisely the set of regular languages outside PTL, for example, parity and mod-2 counting.
Probabilistic Circuits (PCs) vs. LLMs: The expressivity gap is operationalized as the systematic difference in negative log-likelihood (NLL) obtainable by the best PC and LLM for a given next-token prediction task (Zhao et al., 13 May 2026). It arises from two orthogonal bottlenecks:
1. Output bottleneck: Convex combinations in probability space (PCs) vs. unconstrained logit-space parameterizations (LLMs);
2. Context-encoding bottleneck: Rigid, vtree-aligned routing in structured-decomposable PCs vs. dynamically adaptable routing in LLM self-attention.
Quantum Circuits and Variational Quantum Algorithms (VQAs): Expressibility is often measured by closeness to Haar-random state ensembles via frame potentials, KL-divergence in fidelity distributions, or hypothesis-space covering numbers. The gap is quantified either as the difference between the covering numbers (upper and lower bounds) for specific ansatzes (Ghosh et al., 2023), or as the KL-divergence difference between circuit templates (Sim et al., 2019), or even as the inability to reach certain effective Hilbert subspaces without incurring exponential barren-plateau gradients (Hamid, 12 Apr 2026).
Latent Semantic Manifolds in LLMs: Expressibility gap is formulated geometrically as the normalized volume of points on the semantic manifold lying within small margin ε of the Voronoi partition boundary imposed by vocabulary discretization (Mabrok, 17 Mar 2026). The associated scaling law shows that any finite vocabulary incurs an irreducible distortion lower bounded by a rate-distortion theorem.
State-Space Models (SSMs): For DCD SSMs, the expressibility gap is algebraically characterized: a k-layer DCD SSM can express the state-tracking of a group if and only if the group is solvable with derived length ≤ k. Non-solvable groups and some practical optimization failures thus define expressivity and learnability gaps (Shakerinava et al., 2 Mar 2026).

2. Theoretical Frameworks and Expressibility Frontiers

Various mathematical frameworks yield precise borders for expressibility gaps:

Logic and Formal Languages: The equivalence "Transformers ≡ PTL" shows that the expressivity frontier for fixed-precision transformers is exactly the PTL-definable languages, corresponding to languages accepted by partially ordered DFAs and with R-trivial syntactic monoids (Li et al., 29 May 2025).
Quantum Circuits: The covering number of the hypothesis space for parameterized quantum circuits yields both upper and lower analytical bounds, and the difference between these (the expressibility gap) increases with ansatz depth and width (Ghosh et al., 2023). Frame-potential and fidelity-KL frameworks reveal template-dependent saturation floors and quantitative gaps between architectures (Sim et al., 2019, Kashif et al., 25 May 2026).
Latent Manifolds and Geometry: The expressibility gap in semantic manifolds is governed by the coarea formula and rate-distortion bounds, leading to linear scaling laws between the measure of the ambiguous set (low-margin region) and the margin parameter ε, and an intrinsic lower bound scaling inversely with vocabulary size and effective dimension (Mabrok, 17 Mar 2026).
Algebraic/Automata View of SSMs: The expressivity boundary for DCD SSMs is precisely the class of solvable groups up to derived length k, creating a strict algebraic gap between which sequential group actions can or cannot be tracked (Shakerinava et al., 2 Mar 2026).

3. Empirical Manifestations and Measurement Protocols

The expressibility gap is often directly measurable, both theoretically and empirically, via several metrics:

Domain	Expressibility Metric	Gap Manifestation
LLMs	Language capacity (PTL, FO[<])	Failure on non-PTL tasks
Prob. Circuits	NLL (held-out data), separation ranks	Persistent NLL gap
Quantum Circuits	KL-divergence(fid), covering number	KL difference, unreachable states
SSMs	Algebraic class (solvable group order)	Task failure for non-solvable G
Semantic Manifolds	Volume of ε-tubular margin region	Linear scaling, core irreducibility

For example, empirical training of transformers and LSTMs on synthetic formal languages demonstrates that only PTL-definable languages are perfectly learned and generalized by standard transformers, matching the formal frontier (Li et al., 29 May 2025). Similarly, in quantum circuits, benchmarking across ansatz templates and hardware transpilation regimes reveals up to 125% shifts in expressibility (KL divergence) under compilation, with profound consequences for NISQ device viability (Kashif et al., 25 May 2026, Somkuwar et al., 5 Jun 2026).

4. Causes and Structural Origins

Expressibility gaps arise from:

Architectural constraints: Fixed precision, masked attention, and lack of positional encodings in transformers (Li et al., 29 May 2025); strict vtree-alignment in PCs (Zhao et al., 13 May 2026); local diagonal or block-diagonal transitions in SSMs (Shakerinava et al., 2 Mar 2026).
Parameterization bottlenecks: Convex-hull restrictions in probability-space parameterization (PCs vs. LLMs) (Zhao et al., 13 May 2026); covering-number upper and lower bounds in quantum ansatz selection (Ghosh et al., 2023).
Combinatorial rigidity: In logic, syntactic and semantic closure properties define which modal operators are first-order expressible or axiomatizable, with starkly different status for submodel vs. extension modalities (Poliakov et al., 4 May 2026).
Hardware and compilation effects: Transpilation to hardware, routing overheads, and coherence times in NISQ computing can shift both effective expressibility and trainability gaps in nontrivial ways, decoupling theoretical template behavior from practical performance (Kashif et al., 25 May 2026, Somkuwar et al., 5 Jun 2026).

5. Consequences, Implications, and Mitigation Strategies

The existence of a nonzero expressibility gap imposes hard limitations:

Model Selection and Task Matching: Knowing the expressivity boundary prevents wasteful attempts to use inexpressive architectures on tasks outside their reach (e.g. PTL-inexpressible languages with standard transformers (Li et al., 29 May 2025), non-solvable group tracking with diagonal SSMs (Shakerinava et al., 2 Mar 2026), or dense asset correlation in NISQ finance with hardware-efficient VQNNs (Somkuwar et al., 5 Jun 2026)).
Quantum Algorithms: Both excessively low and excessively high expressibility harm VQE performance—too low and the ground state is inaccessible, too high and trainability collapses (barren plateaus). Analytical lower and upper bounds on the covering number allow identification of the "best expressive region" (Ghosh et al., 2023, Hamid, 12 Apr 2026).
Architectural Recommendations: Dynamic or decomposable architectures can reduce the gap (mixtures of vtrees in PCs, block-dense matrices in SSMs, adaptive softening of quantum circuit cutoffs) (Zhao et al., 13 May 2026, Shakerinava et al., 2 Mar 2026, Hamid, 12 Apr 2026). However, optimization and learnability gaps may persist.
Evaluation and Benchmarking: Hardware-level evaluation of expressibility is essential, as transpilation can induce substantial, ansatz-dependent gaps relative to logical designs (Kashif et al., 25 May 2026).
Geometric and Statistical Decomposition: The persistent "hard core" of boundary-proximal states in LLM manifolds defines an irreducible component of perplexity and semantic ambiguity, setting a lower bound to achievable accuracy with finite vocabularies (Mabrok, 17 Mar 2026).

6. Open Problems and Future Directions

Open challenges in understanding and bridging expressibility gaps include:

Parameterization Innovation: Developing hybrid approaches that interpolate between probability-space and logit-space parameterizations in tractable generative models (Zhao et al., 13 May 2026).
Adaptive Structure Learning: End-to-end learning of mixtures or dynamic routing patterns to avoid the rigidity of single tree/topology decompositions (Zhao et al., 13 May 2026, Shakerinava et al., 2 Mar 2026).
Scalable Optimization: Closing the gap between expressivity and practical learnability, particularly in regimes where models are in principle sufficient but optimization landscapes are challenging (Shakerinava et al., 2 Mar 2026).
Interplay with Other Gaps: Integrating expressibility analysis with trainability (gradient variance), hardware resilience (coherence time), and statistical generalization remains an active area, particularly in quantum and deep learning contexts (Hamid, 12 Apr 2026, Kashif et al., 25 May 2026, Somkuwar et al., 5 Jun 2026).
Formal Logic and Semantic Compression: Understanding the limitations of expressive power in rich modal logics and their practical impact on inference and automated reasoning (Poliakov et al., 4 May 2026).

7. Historical Trends and Cross-Domain Perspectives

Expressibility gaps have been quantified in motion expressivity for robots versus biological organisms, highlighting a stagnation in the mechanical (external) repertoire despite exponential growth in computational (internal) states—a pattern echoed in digital-physical system design and moving platforms (LaViers, 2018).

A common theme is that architectural, physical, or logical constraints can sharply limit the expressive reach of a system far below what high-level design goals or parameter growth would naively suggest. Quantitative frameworks for measuring and characterizing expressibility gaps, as exemplified in the cited works, are therefore central for principled algorithm and hardware design, evaluation, and deployment across a range of computational sciences.