Deep Learning is Not So Mysterious or Different (2503.02113v1)

Abstract: Deep neural networks are often seen as different from other model classes by defying conventional notions of generalization. Popular examples of anomalous generalization behaviour include benign overfitting, double descent, and the success of overparametrization. We argue that these phenomena are not distinct to neural networks, or particularly mysterious. Moreover, this generalization behaviour can be intuitively understood, and rigorously characterized using long-standing generalization frameworks such as PAC-Bayes and countable hypothesis bounds. We present soft inductive biases as a key unifying principle in explaining these phenomena: rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data. This principle can be encoded in many model classes, and thus deep learning is not as mysterious or different from other model classes as it might seem. However, we also highlight how deep learning is relatively distinct in other ways, such as its ability for representation learning, phenomena such as mode connectivity, and its relative universality.

• The paper demonstrates that deep learning models generalize effectively by integrating soft inductive biases within classical statistical frameworks.
• It reveals that phenomena like benign overfitting and double descent are not unique to deep learning but occur in various overparameterized models.
• The study highlights deep learning’s unique strengths in representation learning and universality while grounding its success in established generalization principles.

Introduction

The paper "Deep Learning is Not So Mysterious or Different" (2503.02113) challenges the perception that deep learning represents a radical departure from classical statistical learning theory. It argues that phenomena often considered unique to deep neural networks, such as benign overfitting and double descent, are not exclusive and can be understood through established generalization frameworks and the unifying concept of soft inductive biases. Instead of viewing deep learning as mysterious, the paper suggests its generalization behavior aligns with principles applicable across various model classes, while also acknowledging its distinct characteristics like representation learning and universality.

1. Demystifying Generalization Phenomena in Deep Learning

Deep learning models often exhibit behaviors that seem counterintuitive under classical learning theory, contributing to their perceived mystery. Key examples include:

• Benign Overfitting: This occurs when models, especially highly overparameterized ones (more parameters than data points), achieve near-perfect accuracy on the training set (even fitting noise) yet still generalize well to unseen data. For instance, a deep network might interpolate noisy image labels during training but maintain low error on a test set. This contrasts with the traditional bias-variance trade-off, which predicts poor generalization from overfitting.
• Double Descent: Classical theory suggests test error follows a U-shaped curve as model complexity increases – first decreasing, then increasing due to overfitting. However, in many deep learning settings (and other models), as complexity continues to increase beyond the interpolation threshold, the test error can decrease again. This non-monotonic behavior challenges the simple view that ever-increasing complexity beyond a certain point is always detrimental to generalization.

The paper argues that these phenomena are not unique signatures of deep learning but rather outcomes observable in other sufficiently complex models when viewed through appropriate theoretical lenses.

2. Unifying Principles: Soft Inductive Biases and Generalization Frameworks

The paper proposes soft inductive biases as a core principle explaining generalization across different machine learning models, including deep learning.

• Soft vs. Hard Inductive Biases:
  • Hard Biases: Impose strict limitations on the hypothesis space (e.g., restricting a model to only linear functions).
  • Soft Biases: Do not rigidly limit the hypothesis space but introduce a preference for certain solutions, typically simpler ones, while allowing complex solutions if supported by the data. This is often achieved via regularization (like L1/L2 penalties), Bayesian priors, or implicitly through optimization algorithms (like SGD).

Soft biases allow models to be flexible. They can leverage rich, expressive hypothesis spaces (necessary for complex tasks) but are guided towards solutions that avoid merely memorizing training noise, thus promoting better generalization. This aligns with Occam's Razor – favoring the simplest explanation consistent with the data.

Established generalization frameworks provide tools to formalize these ideas:

• PAC-Bayes Framework: This framework bounds the generalization error by considering distributions over hypotheses rather than single hypotheses. A key element is the Kullback-Leibler (KL) divergence, $D_{KL}(Q||P)$, between a posterior distribution $Q$ (learned from data) and a prior distribution $P$. A typical bound might look like: With high probability, $$L_D(Q) \le L_S(Q) + \sqrt{\frac{D_{KL}(Q||P) + \ln(m/\delta)}{2m}}$$, where $L_D$ is the true risk, $L_S$ is the empirical risk, $m$ is the sample size, and $\delta$ is the confidence parameter.
  • Implementation: Choosing priors (e.g., Gaussian over weights) and posteriors, estimating or bounding the KL divergence, and potentially optimizing the bound directly are key considerations.
  • Insights: PAC-Bayes can explain benign overfitting by showing that even complex models can generalize if the posterior $Q$ remains "close" to the prior $P$ (low KL divergence), effectively finding a simple solution within a large space. Soft biases encourage this closeness.
• Countable Hypothesis Bounds: These bounds apply when the hypothesis space is countable or can be discretized. They rely on the union bound over hypotheses.
  • Implementation: Requires discretization of continuous parameter spaces (e.g., weight quantization). The bound's usefulness depends on the effective size of the hypothesis space, which might be much smaller than the nominal size due to regularization or optimization biases.
  • Insights: Can help understand overparameterization by suggesting the learning algorithm effectively searches a smaller, well-behaved subset of the vast parameter space.

These frameworks demonstrate that controlling explicit model complexity (like VC dimension) isn't the only path to generalization. Implicit biases from optimization and explicit soft biases from regularization play crucial roles, particularly in overparameterized regimes common in deep learning.

3. Shared Generalization Behaviors Across Model Classes

The paper emphasizes that phenomena like benign overfitting and double descent are not exclusive to deep learning. They appear in other model classes when appropriate conditions (like overparameterization and suitable biases) are met.

• Benign Overfitting Examples:
  • Linear Models with $\ell_1$ Regularization (LASSO): In high-dimensional settings ($p > n$), LASSO can perform accurate prediction by inducing sparsity, a soft bias. The model effectively selects a small subset of features, achieving good generalization despite the potential to perfectly fit the training data.

  • import numpy as np from sklearn.linear_model import Lasso from sklearn.model_selection import train_test_split from sklearn.metrics import mean_squared_error # Generate synthetic data (more features than samples) n_samples = 100 n_features = 200 X = np.random.randn(n_samples, n_features) # True relationship depends only on a few features true_beta = np.zeros(n_features) true_beta[:10] = np.random.randn(10) y = X @ true_beta + 0.5 * np.random.randn(n_samples) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) # Fit Lasso model (alpha controls regularization strength) lasso = Lasso(alpha=0.1) lasso.fit(X_train, y_train) # Evaluate - LASSO finds sparse solution and generalizes y_pred = lasso.predict(X_test) mse = mean_squared_error(y_test, y_pred) num_coeffs = np.sum(lasso.coef_ != 0) print(f"LASSO MSE: {mse:.4f}, Non-zero coefficients: {num_coeffs}")

  • Kernel Methods: Kernel machines (e.g., SVMs, Gaussian Processes) using expressive kernels (like Gaussian kernels) can interpolate training data perfectly. Regularization (e.g., maximizing the margin in SVMs) acts as a soft bias, enabling good generalization even when the effective number of features (in the kernel-induced space) is very large.
• Double Descent Examples:
  • Random Forests: Test error can exhibit double descent as the number of trees increases. Initially, error decreases. It might increase slightly as trees become highly correlated (overfitting regime), but then decrease again as adding vastly more diverse trees pushes the model into the interpolation regime where ensemble averaging smooths out predictions effectively.

  • import numpy as np from sklearn.ensemble import RandomForestRegressor from sklearn.model_selection import train_test_split from sklearn.metrics import mean_squared_error import matplotlib.pyplot as plt # Generate synthetic data n_samples = 100 n_features = 20 X = np.random.rand(n_samples, n_features) y = np.sum(X[:, :5], axis=1) + 0.2 * np.random.randn(n_samples) # True function depends on first 5 features X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42) n_estimators_range = np.unique(np.logspace(0, 3, 20).astype(int)) test_errors = [] for n_estimators in n_estimators_range: # Control complexity: max_features limits diversity, max_depth limits individual tree complexity rf = RandomForestRegressor(n_estimators=n_estimators, max_features=0.5, # Use subset of features per split max_depth=None, # Allow deep trees random_state=42, n_jobs=-1) rf.fit(X_train, y_train) y_pred = rf.predict(X_test) test_errors.append(mean_squared_error(y_test, y_pred)) plt.figure(figsize=(8, 5)) plt.plot(n_estimators_range, test_errors, marker='o') plt.xscale('log') plt.xlabel("Number of Trees (Model Complexity)") plt.ylabel("Test MSE") plt.title("Potential Double Descent in Random Forest") plt.grid(True) plt.show() # Note: Observing clear double descent often requires careful tuning of dataset and RF parameters.

  • Polynomial Regression: Increasing the degree of a polynomial fit can show double descent. Test error decreases, increases (overfitting), and may decrease again at very high degrees where the model interpolates the data smoothly due to implicit regularization or specific properties of the basis functions.

These examples underscore that the underlying principles governing generalization (complexity, bias, data properties) operate across different algorithmic frameworks.

4. Distinctive Features of Deep Learning

While arguing for shared principles, the paper acknowledges features that make deep learning particularly effective and somewhat distinct in practice:

• Representation Learning: Deep networks excel at automatically learning hierarchical features from raw data (e.g., pixels, text tokens). This eliminates the need for extensive manual feature engineering required by many classical methods and is key to their success on complex perceptual tasks.
• Mode Connectivity: The loss landscapes of large neural networks often exhibit mode connectivity, meaning distinct solutions (local minima) found by training can be connected by paths of low loss. This suggests the optimization landscape might be less problematic than initially thought and allows for techniques like model averaging or ensembling along these paths.
• Universality and In-Context Learning: Deep learning models, especially large ones like Transformers, show increasing universality. They can act as general-purpose function approximators applicable to diverse tasks, sometimes even learning new tasks "in-context" without explicit fine-tuning. This universality may stem from biases shared between neural architectures and natural data (e.g., towards low Kolmogorov complexity solutions).

These characteristics contribute significantly to the practical power and scalability of deep learning but do not necessarily imply fundamentally different principles of generalization compared to other flexible, high-capacity models.

5. Practical Implications and Future Directions

Understanding deep learning through the lens of soft inductive biases and existing generalization theory has several practical implications:

• Leverage Existing Theory: Practitioners can apply insights from statistical learning theory (e.g., regularization, Bayesian methods) to design and train deep learning models more effectively.
• Design Better Biases: Research can focus on developing novel soft inductive biases (new regularizers, architectural choices, data augmentation strategies) tailored to specific deep learning tasks and architectures to improve robustness and generalization.
• Improve Theoretical Tools: Further work is needed to make theoretical bounds (like PAC-Bayes) tighter and more practical for complex deep learning models, potentially involving better estimation techniques or data-dependent priors.
• Investigate Unique Aspects: While shared principles exist, the practical implications of unique features like representation learning hierarchies and mode connectivity warrant continued investigation to fully understand their contribution to deep learning's success.

Acknowledging both the common ground and the distinctions helps bridge the gap between classical theory and deep learning practice.

6. Conclusion

"Deep Learning is Not So Mysterious or Different" (2503.02113) provides a valuable perspective by contextualizing deep learning's generalization behavior within broader statistical learning principles. By highlighting the role of soft inductive biases and demonstrating the presence of phenomena like benign overfitting and double descent in other model classes, the paper demystifies aspects of deep learning often perceived as unique. While acknowledging deep learning's distinctive strengths in representation learning and universality, it encourages leveraging established theoretical frameworks to better understand, analyze, and improve these powerful models. This unified view fosters a more principled approach to developing robust and generalizable AI systems.