Optimizer-Induced Bias in Machine Learning

• Optimizer-induced bias is the phenomenon where an algorithm's design and geometry systematically influence which global or near-global optimum is selected.
• In overparameterized and nonconvex models, methods like mirror descent and steepest descent implicitly promote minimum norm or maximum-margin solutions that affect generalization.
• Understanding this bias enables practitioners to fine-tune algorithmic choices and constraints to enhance fairness, robustness, and interpretability in practical learning systems.

Optimizer-induced bias refers to the phenomenon where the choice or design of the optimization algorithm, its geometric structure, and its auxiliary parameters systematically influences which solution is selected from the set of global (or near-global) optima in machine learning and statistical estimation problems. In the modern context of over-parameterized and nonconvex models, this bias exerts a profound effect not only on convergence behavior but on generalization, interpretability, fairness, and even the functional expressivity of learned models.

1. The Role of Optimization Geometry in Inducing Bias

Optimization geometry primarily determines the implicit bias of iterative optimizers in both regression and classification. For underdetermined linear regression, mirror descent with a strongly convex potential function ψ selects, among all solutions satisfying $w^\top x_n = y_n$, the minimizer of the induced Bregman divergence $D_\psi(w, w_0)$ from the initialization. For steepest descent with respect to a norm, the algorithm chooses the solution closest to initialization in that norm, regardless of step-size or momentum under suitable conditions (Gunasekar et al., 2018):

$$w_\infty = \arg \min_{w \in G} D_\psi(w, w_0), \quad D_\psi(w, w_0) = \psi(w) - \psi(w_0) - \langle \nabla \psi(w_0), w - w_0 \rangle$$

Similarly, in separable classification with strictly monotone losses (e.g., logistic regression), the optimizer “selects” the direction of the maximum-margin (with respect to the norm or geometry) separator:

$$\lim_{t\to\infty} \frac{w_t}{\|w_t\|} = \arg\max_{\|w\|\leq 1} \min_n x_n^\top w$$

In both cases, the result is that different algorithmic choices (mirror descent, steepest descent, natural gradient) effectively “program” a unique, geometry-dependent solution by biasing the outcome through their induced metric, often with complete insensitivity to generic hyperparameters such as step-size (as long as these remain within certain regimes).

2. Constraints and the Emergence of Bias

The explicit constraints imposed in optimization can themselves be a direct source of statistical bias. In pairwise comparison models such as the Bradley–Terry–Luce model, the standard MLE under a box constraint, $\|\theta\|_\infty \leq B$, exhibits a systematic boundary-induced clipping effect: items at the upper boundary are consistently underestimated (and those at the lower boundary overestimated), manifesting as an $O(1/\sqrt{dk})$ bias where $d$ is the number of items and $k$ the number of comparisons per pair. By slightly enlarging the constraint set—“stretching” the bounding box to $\|\theta\|_\infty \leq A$ with $A > B$—one can reduce the worst-case bias to $O((\log d + \log k)/(dk))$ while retaining minimax-optimal MSE (Wang et al., 2019). This demonstrates that constraint tuning is both a source and a mitigation strategy for optimizer-induced bias.

3. Implicit Bias, Overparameterization, and Generalization

A central theoretical insight is that optimizer-induced bias offers an explanation for generalization in the absence of explicit regularization. Overparameterized models possess infinitely many global minima; optimizers such as gradient descent drive iterates toward those with minimum norm or maximum margin in the induced geometry. In linear classification and regression, this accounts for the surprising generalization properties observed in modern deep learning systems (Gunasekar et al., 2018). However, the sufficiency and universality of this explanation is limited (Dauber et al., 2020): for stochastic convex optimization, there exist problem instances with plateaus or degenerate solutions where no (distribution-independent) implicit regularizer can explain which element is chosen, and even distribution-dependent regularizers may leave the solution set too large or statistically complex to ensure generalization.

4. Algorithmic Choices, Nonconvexity, and Solution Properties

In highly nonconvex settings—such as deep neural networks—optimizers do more than determine convergence rates; they shape the qualitative nature of the learned solutions. The preconditioner P in a gradient update $$\theta_{t+1} = \theta_t - \eta P \nabla L(\theta_t)$$ not only accelerates or slows convergence along certain directions but also encodes inductive biases. For example, standard SGD encourages solutions close to the origin (small norm), while sophisticated preconditioners (like Shampoo) produce more localized, less redundant representations (Pascanu et al., 16 Jul 2025). Reparameterizations (such as Power-propagation, θ = φ^α) can be interpreted as preconditioners enforcing sparsity or other desired properties, revealing that optimizer design directly controls the accessible solution space, beyond what is determined by model architecture alone.

5. Practical Implications and Algorithm Design

Understanding optimizer-induced bias is critical for both the design and deployment of learning systems:

• Careful selection of optimization geometry (potential function/norm) enables practitioners to “program” preferences for smoother, more robust, or sparser solutions—often with minimal or no need for explicit regularization (Gunasekar et al., 2018, Jacobs et al., 3 Jun 2025).
• Tuning of explicit constraints (such as the bounding box in MLEs) can resolve trade-offs between accuracy and fairness in estimation, directly addressing boundary- or constraint-induced bias (Wang et al., 2019).
• In adversarially robust learning, aligning the implicit bias of the optimizer (e.g., choosing $\ell_1$-biased algorithms for robust classification against $\ell_\infty$ perturbations) leads to improved robust generalization and smaller adversarial gaps (Tsilivis et al., 7 Jun 2024).
• In evaluation, bilevel, and sample-based optimization, unaccounted bias can emerge due to entropy or information trade-offs, risk aversion, or optimistic sample selection (Dentcheva et al., 2021, Celis et al., 2023). Algorithmic interventions, including multi-objective optimization (for bias versus accuracy) or information-criterion-based corrections, provide rigorous mitigation strategies.

6. Broader Perspectives and Future Directions

While convexity-based analysis has guided the initial understanding of optimizer-induced bias, its relevance extends decisively to the nonlinear, overparameterized regime. The optimizer is now recognized as a critical lever—on a par with data and architecture—in determining generalization, fairness, and effective expressivity. The emerging view is not merely to analyze but to leverage optimizer-induced bias as a tool for encoding application-specific desiderata, such as sparsity, low interference (for continual learning), or tailored robustness properties (Pascanu et al., 16 Jul 2025). Ongoing research aims to build new optimizers with explicit intent to induce particular properties—sometimes by interpolating between classes of geometric biases (e.g., between L₂- and L₁-like regularity via hybrid mirror steps (Jacobs et al., 3 Jun 2025)), sometimes by co-designing optimization with architecture for robust or transferable representations (Li et al., 8 Oct 2024).

7. Concluding Summary

Optimizer-induced bias arises from the geometric and operational structure imposed by optimization algorithms. This bias determines which solution is selected among many possible optima, independently of explicit regularization and often insensitive to generic hyperparameters. Its theoretical foundation is now well established in both convex and (to an increasing degree) nonconvex settings, with practical impact documented in numerous applications including regression, classification, adversarial robustness, fairness-aware estimation, and real-world evaluation processes. A nuanced understanding and intentional harnessing of this phenomenon is increasingly recognized as essential for designing learning algorithms whose solutions are not only accurate, but also exhibit desired properties of simplicity, robustness, sparsity, and fairness.