Learning Without Training

Abstract: Machine learning is at the heart of managing the real-world problems associated with massive data. With the success of neural networks on such large-scale problems, more research in machine learning is being conducted now than ever before. This dissertation focuses on three different projects rooted in mathematical theory for machine learning applications. The first project deals with supervised learning and manifold learning. In theory, one of the main problems in supervised learning is that of function approximation: that is, given some data set $\mathcal{D}={(x_j,f(x_j))}_{j=1}^M$, can one build a model $F\approx f$? We introduce a method which aims to remedy several of the theoretical shortcomings of the current paradigm for supervised learning. The second project deals with transfer learning, which is the study of how an approximation process or model learned on one domain can be leveraged to improve the approximation on another domain. We study such liftings of functions when the data is assumed to be known only on a part of the whole domain. We are interested in determining subsets of the target data space on which the lifting can be defined, and how the local smoothness of the function and its lifting are related. The third project is concerned with the classification task in machine learning, particularly in the active learning paradigm. Classification has often been treated as an approximation problem as well, but we propose an alternative approach leveraging techniques originally introduced for signal separation problems. We introduce theory to unify signal separation with classification and a new algorithm which yields competitive accuracy to other recent active learning algorithms while providing results much faster.

• The paper presents a unified approximation-theoretic framework that bypasses traditional ERM by constructing direct data-driven approximations.
• It introduces a novel constructive method on unknown manifolds using spherical harmonics to yield localized error bounds without iterative optimization.
• The work extends its methodology to transfer learning and classification via support estimation, achieving competitive accuracy with reduced labeling costs.

Learning Without Training: A Theoretical and Algorithmic Paradigm Shift in Machine Learning

Introduction

The dissertation "Learning Without Training" (2602.17985) introduces a unified approximation-theoretic framework for a spectrum of machine learning problems—supervised learning, transfer learning, and classification. The author posits that contemporary ML practice, while empirically successful, is stunted by its reliance on existential (non-constructive) approximation guarantees and empirical risk minimization (ERM) procedures that obscure the connection between mathematical smoothness, data geometry, and algorithmic performance. Through a sequence of mathematically rigorous constructions, the work proposes and analyzes algorithms that “learn without training”—circumventing optimization by constructing direct, explicit approximations from data.

Critique of Current Supervised Learning Paradigm

A central thesis of the dissertation is a comprehensive critique of ERM as an operationalization of supervised learning. Standard approaches select a hypothesis space $V_n$, perform empirical risk minimization relative to a global loss functional (e.g., MSE), and fit the finite training data—typically via iterative optimization. Theoretical justifications often rest on universal approximation and degree of approximation results (see, e.g., Barron and Sobolev space rates), but these results are existential and assume knowledge of the data domain and the smoothness of $f$. Moreover, performing optimization in high-dimensional, noisy settings faces issues of convergence, instability, and sensitivity to initialization, and global loss minimization is insensitive to local artifacts of the target function.

Figure 1: A depiction of the standard supervised learning paradigm. The universe of discourse $\mathcal{X}$ is assumed to contain a target function $f$ and hypothesis spaces $V_n$ are judiciously chosen based on the algorithm of choice. $P^\#$ denotes the empirical risk minimizer, $\tilde{P}$ denotes the minimizer of the generalization error, and $P^*$ denotes the best approximation.

The author demonstrates that explicit, data-dependent construction can yield practical rates of approximation that are fundamentally more informative than global, existential bounds. For example, degree-of-approximation results relying on unavailable information such as best-approximation coefficients can mislead practical model design.

Constructive Approximation on Manifolds Without Optimization

The dissertation’s first technical core is a constructive approximation method on unknown manifolds. Building from multivariate trigonometric and spherical harmonics theory, the method projects (possibly noisy) data from an unknown $q$-dimensional submanifold of $\mathbb{R}^Q$ (or equivalently, $\mathbb{S}^Q$) onto the sphere, then forms a kernel-based interpolant directly from samples without optimization or manifold learning:

$$F_{n}(\mathcal{D}; x) = \frac{1}{M}\sum_{j=1}^M z_j \Phi_{n,q}(x \cdot y_j)$$

where $\Phi_{n,q}$ is a highly localized, polynomial kernel based on spherical harmonics and $n$ is a degree parameter. The key theoretical result gives explicit, high-probability, non-asymptotic error bounds in terms of the data manifold's dimension and the local smoothness of $f$:

$$\|F_{n}(\mathcal{D}; \cdot) - f\|_\mathbb{X} \lesssim ( {\|z\|} + \|f\|_{W_\gamma} ) \left( \frac{\log M}{M} \right)^{\gamma/(q+2\gamma)}$$

for $f \in W_\gamma(\mathbb{X})$. Notably, only the dimension $q$ is required as prior information; no manifold learning or tangent space estimation is performed. The method automatically produces locally adaptive error in regions of diverse smoothness.

Figure 2: A depiction of a new machine learning paradigm, where one constructs an approximation $\sigma_n$ in the space $V_n$ directly from the data. This is done in such a way that one can also measure a direct reconstruction error from the approximation to the target function.

Numerical experiments showcase sharp localization of errors to function singularities and superior percentiles of pointwise error compared to RBF and Nadaraya-Watson estimators, even when global RMS errors are comparable (see Figure 3).

Figure 3: Error comparison between our method, the Nadaraya-Watson estimator, and an interpolatory RBF network. (Left) Comparison of absolute errors between the methods with the target function plotted on the right y-axis for benefit of the viewer. The error from the RBF method is scaled by $10^{-3}$.

Localized Transfer Learning via Joint Data Spaces

The second major contribution extends the approximation-theoretic paradigm to a principled treatment of transfer learning. Here, transfer is modeled as the explicit lifting of a function from a source (base) manifold to a target manifold, potentially via known correspondences (e.g., landmark points, operator SVDs). A joint kernel is constructed respecting the geometry and spectral properties of both spaces:

$$\Phi_n(H, \Xi_1, \Xi_2; x_1, x_2) = \sum_{j,k} H\left(\frac{\ell_{j,k}}{n}\right) A_{j,k} \phi_{1,j}(x_1) \phi_{2,k}(x_2)$$

where $(\lambda_{i,j}, \phi_{i,j})$ are spectral triples and $A_{j,k}$ are connection coefficients. The main theorem specifies on which regions of the target space the lifted function is defined from data available only in a region of the source space, and quantifies how source smoothness $(\gamma)$ and geometric compatibilities control target space smoothness $(\gamma-Q+q_2)$.

This framework is illustrated through a detailed analysis of Jacobi polynomial expansions and their transplants, linking ML transfer learning to inverse problems such as limited angle tomography and Radon inversion.

Classification as Measure Support Estimation: The MASC Algorithm

The final part of the dissertation reconceptualizes classification as the problem of nonparametric support estimation for a mixture measure $\mu = \sum_k a_k \mu_k$, where each $\mu_k$ represents a (possibly overlapping) class-conditional distribution. Rather than approximating conditional expectations, the method seeks to partition the data into $K_\eta$ clusters via multiscale, kernel-based density estimation:

$F_n(x) = \frac{1}{M}\sum_{j=1}^M \Psi_n(x, x_j)$

with $\Psi_n(x,y) = \Phi_n(\rho(x, y))^2$ and localized trigonometric $\Phi_n$. Theoretical results guarantee that for proper threshold $\Theta$ and kernel scale $n$, the set

$$\mathcal{G}_n(\Theta) = \{ x : F_n(x) \geq \Theta \cdot \max F_n(x_j) \}$$

recovers a tight neighborhood of the true support $\mathbb{X}$, and, under a quantitative fine-structure property, yields clusters of minimal separation $\eta$ with vanishing overlap. This leads to competitive or superior F-scores in the limit. The accompanying Multiscale Active Super-resolution Classification (MASC) algorithm actively queries minimal points to label clusters in a multiscale fashion, followed by nearest-neighbor extension.

Empirical results (Figures 15–25) on synthetic data, document classification, and hyperspectral imaging demonstrate that MASC achieves competitive accuracy with markedly reduced labeling cost compared to existing active learning algorithms (LAND, LEND), and provides robust support recovery even in the presence of significant overlap.

Figure 4: This figure illustrates the result of applying MASC to a synthetic circle and ellipse data set. On the left are true labels of the given data, and on the right is the estimation attained by MASC.

Figure 5: Plots indicating the accuracy of MASC, LAND, and LEND for different query budgets, for both Salinas (left) and Indian Pines (right).

Numerical and Implementation Details

The constructive methods support fast and matrix-based implementations (kernel evaluations via Clenshaw’s algorithm) and scale linearly in the number of samples, with polynomial hyperparameters controlling locality and adaptiveness. Hyperparameters such as $n$ (kernel degree), $q$ (manifold dimension), and density thresholds are shown to directly trade off bias, variance, and computational cost. Select experiments highlight the substantial practical impact, both for regression and classification, and the strong out-of-sample extension properties.

Implications and Theoretical Advances

This work rigorously bridges modern harmonic analysis and approximation theory with practical algorithmic designs for learning on high-dimensional, structured, or unknown domains. It provides a systematic inversion of the standard narrative: rather than using learning theory to motivate approximation, it uses constructive approximation theory to derive novel, efficient, and robust machine learning algorithms—thereby achieving learning without training.

Several claims in the dissertation run counter to prevailing practice:

• Existence-based universal approximation is insufficient: Constructive, data-driven approximation yields explicit rates and local adaptivity that are not accessible via generic universality theorems.
• Optimization (training) is not inevitable: For wide classes of high-dimensional regression and classification, explicit kernel-based constructions can attain optimal rates without iterative optimization, regularization, or model selection.
• Separation of manifold learning and function approximation is not necessary: The presented methods bypass the need for manifold estimation or eigen-decomposition, requiring only the manifold’s dimension.
• Classification need not rely on function approximation: Direct support recovery and measure partitioning can outperform standard likelihood- or regression-based approaches, especially in the presence of non-disjoint class supports.

Future Directions and Outlook

The dissertation opens multiple research avenues, including:

• Further generalizations to arbitrary compact metric spaces and settings with unknown or variable regularity.
• Operator approximation via representations induced by these encodings (see manifold operator learning).
• Joint feature and function learning, extending the approach to automatic or adaptive feature discovery.
• Practical deployment in large-scale or resource-constrained scenarios, enabling efficient on-device learning via direct approximation.

The approach delineated fundamentally reconfigures the theoretical and practical interface between approximation theory and machine learning, offering promising new methodologies for interpretable, robust, and efficient AI systems.