Optimizer-Dependent Expressivity

• Optimizer-dependent expressivity is the phenomenon where an optimizer’s choice and dynamics directly determine the range and efficiency of solutions beyond a model’s formal expressivity.
• It highlights that different optimizer structures, from classical to quantum variational ones, exploit inherent problem features to overcome traditional computational barriers.
• Empirical studies reveal that optimizer design not only influences convergence speed but also encodes inductive biases that enhance generalization and system-level performance.

Optimizer-Dependent Expressivity refers to the phenomenon whereby the choice, structure, and dynamics of an optimization algorithm directly influence the range and kind of solutions that a model can attain—beyond the nominal expressive capacity defined by the model’s architecture or formal language alone. Originally explored in the context of computational complexity and logic, this concept now spans mathematical logic, variational algorithms (including quantum and classical machine learning), neural network generalization, large-scale deep learning, and compiler design. At its core, optimizer-dependent expressivity highlights that both the efficiency and nature of solutions are determined by the interplay between an optimization algorithm and the formal or parametric description of a problem.

1. Syntactic Logic, Descriptive Complexity, and Optimization

The fundamental distinction between decision and optimization problem expressibility in logical systems demonstrates a primary facet of optimizer-dependent expressivity (0904.4331). While all decision problems decidable in polynomial time (P) can be expressed as existential second-order (ESO) universal Horn sentences (in the presence of a successor relation), the same is not true for their optimization counterparts. For optimization problems, even quantifier-free Horn formulas (Σ₀) can express NP-hard problems, such as MaxHorn2Sat, which are tractable in their decision form.

This dichotomy reflects that the logical "simplicity" of a formula is not a guarantee of computational tractability in the context of optimization: syntax alone does not dictate solvability. Instead, optimizers that can exploit deeper structure in a problem—such as Lagrangian duality or complementary slackness—can shortcut typical combinatorial search, sometimes verifying optimality in a single call to a decision oracle. Thus, the effective expressivity of a logical system is not static, but hinges on the optimization strategy and the problem’s structural features.

2. Optimizer-Dependent Expressivity in Program Optimization

In compiler design and program transformation, optimizer-dependent expressivity manifests through the methods by which code optimizations are performed (Tate et al., 2010). Traditional, sequentially ordered optimization passes can restrict expressivity, as the phase-ordering problem causes some potential optimizations to be irretrievably lost due to early, destructive rewrites.

The equality saturation paradigm eliminates this bottleneck by encoding many optimization opportunities non-destructively within a single intermediate representation—the equivalence PEG (E–PEG). By saturating the IR with discovered equalities, all possible program variants coexist, unlocking a far greater space of optimized possibilities. A global profitability heuristic, such as a Pseudo-Boolean solver, selects the most profitable version only after this space is fully explored, offering an order-independent and globally expressive framework. This demonstrates that optimizer structure directly affects not just the breadth but also the quality and depth of optimizations achievable.

3. Deep Neural Networks, Landscape Geometry, and Learning Dynamics

Neural network expressivity is influenced by both architecture and the nature of the optimizer. In deep convolutional neural networks (CNNs), expressivity depends not just on depth and width, but also on the associated optimization dynamics (Nguyen et al., 2017). When a “wide” layer exists—where the number of neurons exceeds the number of samples—the feature matrix at that layer can be linearly independent, enabling the network to "memorize" arbitrary labels. Overparameterization leads to a well-behaved (benign) loss landscape: almost all critical points are global minima, and standard gradient-based optimizers will converge reliably.

Optimizer-dependent expressivity also surfaces in the dynamic evolution of neural network representations. Hilbert space analysis reveals that activation function selection, batch size, and regularization strongly impact the propagation and preservation of information across layers, ultimately determining if a network can reach the "edge of chaos" where expressivity is maximal (Zhang et al., 2019). Optimizers that maintain gradient flow and efficiently escape saddle points, or that employ batch sizes tuned to avoid "washing out" hidden states, facilitate high expressivity.

4. Quantum Variational Algorithms: Ansatze, Generalization, and Effective Degrees of Freedom

In variational quantum algorithms (VQAs) and quantum neural networks (QNNs), expressivity is linked to the structure of parameterized circuits ("ansatze"), the variety and arrangement of gates, and the influence of classical optimizers. The covering number, derived from statistical learning theory, quantifies hypothesis space complexity and bounds generalization error—a high covering number confers representational flexibility but also risks untrainability due to barren plateaux (Du et al., 2021).

Recent work underscores optimizer-dependent expressivity by demonstrating that generalization in QNNs is governed not just by the model’s expressive power (number of trainable parameters, data uploading rounds), but also by classical optimizer hyperparameters such as learning rate and iteration count (Zhu et al., 27 Jan 2025). Uniform stability theory connects higher parameter count or aggressive learning rates to larger generalization error unless compensated by larger datasets or careful tuning.

A more refined quantifier, the effective rank κ, captures the number of independent parameter directions that actually influence circuit output, measured via the Fisher information matrix (Yao, 18 Jun 2025). κ both bounds expressivity and signals trainability, with automated design approaches (such as reinforcement learning with self-attention agents) using κ as a reward to guide architectural search—tying circuit design, input selection, and optimizer choice into a joint framework.

5. Optimizer Structure as a Source of Inductive Bias

Optimizers encode inductive biases by how they structure updates in parameter space. Methods employing different preconditioners or coordinate adaptation (e.g., diagonal in AdamW, non-diagonal in Shampoo) alter the path taken through the non-convex loss landscape, favoring solutions with different qualitative characteristics (Pascanu et al., 16 Jul 2025). For example, optimizers with non-diagonal preconditioning can lead to more localized or lower-dimensional learned representations, reduce interference in continual learning, or bias toward flatter minima associated with better generalization.

These effects demonstrate that optimizers are not purely mechanical actors but play an active, qualitative role in shaping solution properties. Optimizer design thus becomes a lever not only for improving convergence but also for encoding specific desiderata—such as sparsity, robustness, or compositionality—into the resultant model.

6. Empirical Illustrations and Applied Impacts

Empirical results consistently reveal that optimizer choices control more than just speed of convergence. Fitness Dependent Optimizers (FDO), by incorporating swarm-inspired exploration and exploitation, achieve greater expressivity in multilayer perceptrons than gradient descent, robustly avoiding local minima and attaining higher classification accuracy (Abbas et al., 2022). In deep pipeline training, optimizer-dependent weight prediction strategies adapt the forward-pass weights to anticipated parameter updates, ensuring staleness-free learning and high throughput regardless of optimizer specifics (Guan et al., 2023).

In transformer LLMs, theoretical limits on expressivity (as captured by temporal logic characterizations) manifest in empirical performance: standard optimizers (e.g., Adam) reliably uncover all learnable representations within the model's capacity, but cannot overcome architecture-imposed barriers even when optimization hyperparameters are tuned across wide regimes (Li et al., 29 May 2025).

7. Integration: Theory, Practice, and Future Directions

Optimizer-dependent expressivity bridges foundational theory and practical system design. It establishes that the set of reachable solutions and the tractability of optimization cannot be attributed to architecture or logical expressivity alone but are deeply dependent on the chosen optimizer, the schedule of parameter adaptation, and the capacity to exploit structural problem features.

This insight motivates research on the explicit design of optimizers to encode inductive biases, dynamic adaptation of model architecture based on optimization trajectory (Verbockhaven et al., 30 May 2024), and automated circuit search in quantum and classical learning. It also underscores the need for simultaneous consideration of architecture, dataset, and optimizer dynamics in both theoretical paper and practical deployment. Optimizer-dependent expressivity thus serves as a central and unifying principle in contemporary research at the intersection of logic, algorithms, machine learning, and quantum information science.