---
title: Expressibility Gap in Computational Models
url: https://www.emergentmind.com/topics/expressibility-gap
type: topic
---

# Expressibility Gap in Computational Models

The expressibility gap refers to the systematic, often quantifiable difference between the theoretical expressive power or practical performance of different computational models, architectures, or parameterizations across a range of domains, most notably in deep learning, quantum algorithms, probabilistic circuits, formal language theory, and cyber-physical systems. In essence, the expressibility gap measures the limitations imposed by a specific model class or implementation constraint relative to a chosen baseline (often an idealized or more expressive system), thereby delimiting the set of functions, distributions, or behaviors that can be exactly or approximately realized.

## 1. Formal Definitions and Characterizations

The expressibility gap is always tied to a precise notion of expressivity: the ability of a system to represent, realize, or approximate functions, languages, distributions, or physical configurations.

- **Neural/Language Models:** In the context of transformer language models, the expressibility gap is defined between the set of formal languages recognizable by practical, fixed-precision, strictly-masked, soft-attention transformers and those definable in richer logical or automata-theoretic frameworks. Specifically, the work [2505.23623] proves that such transformers can realize exactly the class of languages definable by the past-only fragment of linear temporal logic (PTL), which sits strictly below the class of all star-free (i.e., aperiodic) languages. The gap is precisely the set of regular languages outside PTL, for example, parity and mod-2 counting.

- **Probabilistic Circuits (PCs) vs. Large Language Models (LLMs):** The expressivity gap is operationalized as the systematic difference in negative log-likelihood (NLL) obtainable by the best PC and LLM for a given next-token prediction task [2605.12940]. It arises from two orthogonal bottlenecks:
  1. **Output bottleneck:** Convex combinations in probability space (PCs) vs. unconstrained logit-space parameterizations (LLMs);
  2. **Context-encoding bottleneck:** Rigid, vtree-aligned routing in structured-decomposable PCs vs. dynamically adaptable routing in LLM self-attention.

- **Quantum Circuits and Variational Quantum Algorithms (VQAs):** Expressibility is often measured by closeness to Haar-random state ensembles via frame potentials, KL-divergence in fidelity distributions, or hypothesis-space covering numbers. The gap is quantified either as the difference between the covering numbers (upper and lower bounds) for specific ansatzes [2311.01330], or as the KL-divergence difference between circuit templates [1905.10876], or even as the inability to reach certain effective Hilbert subspaces without incurring exponential barren-plateau gradients [2604.10607].

- **Latent Semantic Manifolds in LLMs:** Expressibility gap is formulated geometrically as the normalized volume of points on the semantic manifold lying within small margin ε of the Voronoi partition boundary imposed by vocabulary discretization [2603.22301]. The associated scaling law shows that any finite vocabulary incurs an irreducible distortion lower bounded by a rate-distortion theorem.

- **State-Space Models (SSMs):** For DCD SSMs, the expressibility gap is algebraically characterized: a k-layer DCD SSM can express the state-tracking of a group if and only if the group is solvable with derived length ≤ k. Non-solvable groups and some practical optimization failures thus define expressivity and learnability gaps [2603.01959].

## 2. Theoretical Frameworks and Expressibility Frontiers

Various mathematical frameworks yield precise borders for expressibility gaps:

- **Logic and Formal Languages:** The equivalence "Transformers ≡ PTL" shows that the expressivity frontier for fixed-precision transformers is exactly the PTL-definable languages, corresponding to languages accepted by partially ordered DFAs and with R-trivial syntactic monoids [2505.23623].

- **Quantum Circuits:** The covering number of the hypothesis space for parameterized quantum circuits yields both upper and lower analytical bounds, and the difference between these (the expressibility gap) increases with ansatz depth and width [2311.01330]. Frame-potential and fidelity-KL frameworks reveal template-dependent saturation floors and quantitative gaps between architectures [1905.10876, 2605.25552].

- **Latent Manifolds and Geometry:** The expressibility gap in semantic manifolds is governed by the coarea formula and rate-distortion bounds, leading to linear scaling laws between the measure of the ambiguous set (low-margin region) and the margin parameter ε, and an intrinsic lower bound scaling inversely with vocabulary size and effective dimension [2603.22301].

- **Algebraic/Automata View of SSMs:** The expressivity boundary for DCD SSMs is precisely the class of solvable groups up to derived length k, creating a strict algebraic gap between which sequential group actions can or cannot be tracked [2603.01959].

## 3. Empirical Manifestations and Measurement Protocols

The expressibility gap is often directly measurable, both theoretically and empirically, via several metrics:

| Domain         | Expressibility Metric                    | Gap Manifestation                  |
|----------------|-----------------------------------------|------------------------------------|
| LLMs           | Language capacity (PTL, FO[<])          | Failure on non-PTL tasks           |
| Prob. Circuits | NLL (held-out data), separation ranks   | Persistent NLL gap                 |
| Quantum Circuits | KL-divergence(fid), covering number   | KL difference, unreachable states  |
| SSMs           | Algebraic class (solvable group order)  | Task failure for non-solvable G    |
| Semantic Manifolds | Volume of ε-tubular margin region  | Linear scaling, core irreducibility|

For example, empirical training of transformers and LSTMs on synthetic formal languages demonstrates that only PTL-definable languages are perfectly learned and generalized by standard transformers, matching the formal frontier [2505.23623]. Similarly, in quantum circuits, benchmarking across ansatz templates and hardware transpilation regimes reveals up to 125% shifts in expressibility (KL divergence) under compilation, with profound consequences for NISQ device viability [2605.25552, 2606.07727].

## 4. Causes and Structural Origins

Expressibility gaps arise from:

- **Architectural constraints:** Fixed precision, masked attention, and lack of positional encodings in transformers [2505.23623]; strict vtree-alignment in PCs [2605.12940]; local diagonal or block-diagonal transitions in SSMs [2603.01959].

- **Parameterization bottlenecks:** Convex-hull restrictions in probability-space parameterization (PCs vs. LLMs) [2605.12940]; covering-number upper and lower bounds in quantum ansatz selection [2311.01330].

- **Combinatorial rigidity:** In logic, syntactic and semantic closure properties define which modal operators are first-order expressible or axiomatizable, with starkly different status for submodel vs. extension modalities [2605.02725].

- **Hardware and compilation effects:** Transpilation to hardware, routing overheads, and coherence times in NISQ computing can shift both effective expressibility and trainability gaps in nontrivial ways, decoupling theoretical template behavior from practical performance [2605.25552, 2606.07727].

## 5. Consequences, Implications, and Mitigation Strategies

The existence of a nonzero expressibility gap imposes hard limitations:

- **Model Selection and Task Matching:** Knowing the expressivity boundary prevents wasteful attempts to use inexpressive architectures on tasks outside their reach (e.g. PTL-inexpressible languages with standard transformers [2505.23623], non-solvable group tracking with diagonal SSMs [2603.01959], or dense asset correlation in NISQ finance with hardware-efficient VQNNs [2606.07727]).

- **Quantum Algorithms:** Both excessively low and excessively high expressibility harm VQE performance—too low and the ground state is inaccessible, too high and trainability collapses (barren plateaus). Analytical lower and upper bounds on the covering number allow identification of the "best expressive region" [2311.01330, 2604.10607].

- **Architectural Recommendations:** Dynamic or decomposable architectures can reduce the gap (mixtures of vtrees in PCs, block-dense matrices in SSMs, adaptive softening of quantum circuit cutoffs) [2605.12940, 2603.01959, 2604.10607]. However, optimization and learnability gaps may persist.

- **Evaluation and Benchmarking:** Hardware-level evaluation of expressibility is essential, as transpilation can induce substantial, ansatz-dependent gaps relative to logical designs [2605.25552].

- **Geometric and Statistical Decomposition:** The persistent "hard core" of boundary-proximal states in LLM manifolds defines an irreducible component of perplexity and semantic ambiguity, setting a lower bound to achievable accuracy with finite vocabularies [2603.22301].

## 6. Open Problems and Future Directions

Open challenges in understanding and bridging expressibility gaps include:

- **Parameterization Innovation:** Developing hybrid approaches that interpolate between probability-space and logit-space parameterizations in tractable generative models [2605.12940].

- **Adaptive Structure Learning:** End-to-end learning of mixtures or dynamic routing patterns to avoid the rigidity of single tree/topology decompositions [2605.12940, 2603.01959].

- **Scalable Optimization:** Closing the gap between expressivity and practical learnability, particularly in regimes where models are in principle sufficient but optimization landscapes are challenging [2603.01959].

- **Interplay with Other Gaps:** Integrating expressibility analysis with trainability (gradient variance), hardware resilience (coherence time), and statistical generalization remains an active area, particularly in quantum and deep learning contexts [2604.10607, 2605.25552, 2606.07727].

- **Formal Logic and Semantic Compression:** Understanding the limitations of expressive power in rich modal logics and their practical impact on inference and automated reasoning [2605.02725].

## 7. Historical Trends and Cross-Domain Perspectives

Expressibility gaps have been quantified in motion expressivity for robots versus biological organisms, highlighting a stagnation in the mechanical (external) repertoire despite exponential growth in computational (internal) states—a pattern echoed in digital-physical system design and moving platforms [1807.02016].

A common theme is that architectural, physical, or logical constraints can sharply limit the expressive reach of a system far below what high-level design goals or parameter growth would naively suggest. Quantitative frameworks for measuring and characterizing expressibility gaps, as exemplified in the cited works, are therefore central for principled algorithm and hardware design, evaluation, and deployment across a range of computational sciences.

Source: https://www.emergentmind.com/topics/expressibility-gap