Nested Learning Framework

• Nested Learning is a hierarchical paradigm in which models, objectives, or representations are structured in nested levels, from sub-networks to optimization routines.
• It enables explicit control over computation, memory, and supervision, leading to robust generalization, resource-aware adaptation, and improved interpretability across various applications.
• By incorporating architectural, optimization, and representational nesting, NL facilitates continuous learning, efficient hyperparameter tuning, and scalable deep network design.

Nested Learning (NL) comprises a family of machine learning paradigms and architectures in which models, objectives, or representations are hierarchically structured into multiple nested levels. Each level may correspond to a subnetwork, optimization problem, parameter subset, or task granularity, with the property that higher-capacity or finer-grained levels contain, refine, or subsume the lower-level ones. This enables explicit control over representation, computation, memory, or supervision hierarchy, facilitating robust generalization, adaptive computation, continual learning, and interpretability across diverse domains including deep architectures, subspace learning, dictionary learning, nested optimization, multi-instance learning, game theory, and resource-aware model design.

1. Formal Definitions and Theoretical Foundations

A canonical NL system consists of ordered levels $\ell=1,\ldots,K$, with each level solving an optimization problem or representing a sub-model. The general abstract definition is

$$\theta^{(\ell)*} = \arg\min_{\theta^{(\ell)}} \mathcal{L}^{(\ell)}\left(\theta^{(\ell)}; C^{(\ell)}\right),$$

where each $C^{(\ell)}$ denotes the $\ell$-level context flow (e.g., tokens, gradients, features, subpopulations), and the level's solution may serve as context or initialization for other levels, subject to a dependency ordering. NL systems can be strictly hierarchical (nesting by containment or sequence) or admit more general Directed Acyclic Graph (DAG) relationships, with both static (Behrouz et al., 31 Dec 2025) and dynamic variants (Jafari et al., 18 Nov 2025).

The nested structure can be instantiated at the architectural, optimization, or representational level:

• Architectural Nesting: Subnetworks or modules at increasing capacity (e.g., subspaces, masks, cells) are nested by parameter/pruning inclusion or encoding nested bottlenecks (Rauba et al., 22 Sep 2025, Kim et al., 2017, Achddou et al., 2020).
• Optimization Nesting: Multi-level or bilevel problems, with parameter blocks or subproblems subject to inner–outer dependency (e.g., hyperparameter optimization, meta-learning, adversarial games) (Lorraine, 2024, Behrouz et al., 31 Dec 2025).
• Representation Nesting: Hierarchical dictionaries, feature trees, or label taxonomies, with paths or embeddings associated to coarser-to-finer granularity (Li et al., 2012, Achddou et al., 2020).

Mathematically, flag manifolds and partition trees formalize nesting within subspace and similarity-based contexts (Szwagier et al., 9 Feb 2025, Mertikopoulos et al., 2024). Theoretical analysis encompasses convergence, expressivity increases, and regret bounds for both static and dynamically evolving NL hierarchies (Jafari et al., 18 Nov 2025).

2. Nested Learning in Deep Architectures

Parameter and Structural Nesting

• Nested Sparse Networks (NestedNet): Parameter masks of increasing density are imposed on a base network to induce nested sub-networks $$\mathcal{W}^{(1)} \subseteq \mathcal{W}^{(2)} \subseteq \cdots \subseteq \mathcal{W}^{(n)}$$, enabling joint optimization and resource-aware selection at inference (Kim et al., 2017). Channel/layer scheduling generalizes this to block or layer granularity.
• Nested Subspace Networks (NSNs): Linear layers are factorized such that lower-rank subspaces are strictly contained within higher-rank ones (${\rm Im}(W_{r}) \subseteq {\rm Im}(W_{r+1})$), allowing continuous capacity selection with explicit uncertainty balancing during training (Rauba et al., 22 Sep 2025).

Hierarchical Feature Bottlenecks

• Multi-Granular Nested Information Bottlenecks: Representation learning is enforced via $$f_1(X)\in\mathbb{R}^{d_1} \subset f_2(X)\in\mathbb{R}^{d_2} \subset \cdots$$ with networks outputting predictions at each level. Skip connections allow finer levels to directly draw additional information from inputs beyond coarser embeddings (Achddou et al., 2020). This architecture increases adversarial and distortion robustness.

3. Nested Multiple Instance, Collaborative, and Memory Architectures

Multiple Instance Learning and Attention

• Nested Multiple Instance Learning with Attention (NMIA): Inputs are structured as bags of bags (e.g., regions of instances). Instance-level, inner-bag, and outer-bag representations are aggregated via nested attention mechanisms. NMIA models both bag-of-bags labels and provides interpretability at each hierarchy (Fuster et al., 2021).

Collaborative and Multi-Expert Mechanisms

• Nested Collaborative Learning (NCL): Multiple experts are collaboratively trained, each optimizing over all classes (global) and over a set of dynamically mined hard categories (partial/local). The nested structure appears both in individual learning losses and in online distillation among experts, with robust feature learning claims for long-tailed distributions (Li et al., 2022).

Nested Recurrent Memory

• Nested LSTMs (NLSTM): RNNs with memory cells recursively computed by inner LSTMs instead of stacking layers, producing a deeper temporal hierarchy where inner memories capture long-term dependencies and outer cells manage rapid adaptation (Moniz et al., 2018). Outperforms stacked LSTM baselines on long-range sequence tasks.

Dynamic Nested Hierarchies

• Self-Evolving Lifelong Learning Models: Dynamic Nested Hierarchies (DNH) allow for online adjustment in the number of levels, dependency structure, and update frequencies—implemented through meta-optimization and Hebbian initialization—resulting in continual learning with sublinear regret and improved context adaptation in distribution-shifting environments (Jafari et al., 18 Nov 2025). These models support context compression and autonomous complexity modulation.

4. Nested Optimization and Subspace Learning

Bilevel and Multi-Level Optimization

• Scalable Nested Optimization: Bilevel learning is formalized as optimizing outer parameters (e.g., hyperparameters $\lambda$) subject to inner parameter minimization; hypergradients are computed via the implicit function theorem or unrolled differentiation (Lorraine, 2024). Hypernetworks, Neumann-series approximations, and complex momentum methods scale these to large parameter spaces and adversarial games.
• Flag-Trick for Subspace Learning: Classical Grassmannian subspace optimization problems are reformulated on flag manifolds, ensuring that subspaces of increasing dimension ($q_1<q_2<\cdots$) are nested ($S_{q_1}\subset S_{q_2}$), which is crucial for consistent multi-scale representations in tasks such as PCA, LDA, and spectral clustering (Szwagier et al., 9 Feb 2025).

Game Theory and Similarity-Nested Dynamics

• Nested Replicator Dynamics & Nested Logit Choice: Evolutionary game learning on action sets with hierarchical similarity partitions employs nested partitioned pairwise imitation (NPPI) protocols and corresponding dynamics, with rationality properties preserved even when the standard monotonicity postulates fail (Mertikopoulos et al., 2024). The nested logit rule, with a multi-level softmax, highlights NL’s connections to exponential weights and FTRL strategies.

5. Hierarchical and Generative Models

Dictionary Learning and Topic Trees

• Nested Dictionary Learning (nDL): Tree-based dictionary models jointly learn hierarchical representations for imagery (via nested Dirichlet processes) and text (via node-specific topic multinomials), where each image and its constituent parts traverse paths through the learned tree (Li et al., 2012). This framework enables unsupervised discovery of shared/global versus specialized/local visual and semantic structure, as well as flexible image annotation and retrieval.

6. Applications, Robustness, and Limitations

NL paradigms have been demonstrated in:

• Resource-aware adaptation: Single models that serve variable compute budgets or real-time constraints without retraining (Kim et al., 2017, Rauba et al., 22 Sep 2025).
• Hierarchical classification: Coarse-to-fine and multi-granular outputs, improved generalization, and calibration (Achddou et al., 2020, Kim et al., 2017).
• Robustness and interpretability: Improved calibration on out-of-distribution data, adversarial robustness, and hierarchical latent-importance attributions (Achddou et al., 2020, Fuster et al., 2021).
• Continual and lifelong learning: Explicit spectrum of memory and context compression, dynamic expansion/pruning in response to distribution shift, and sublinear regret in lifelong adaptation (Jafari et al., 18 Nov 2025, Behrouz et al., 31 Dec 2025).
• Scalable optimization: Efficient handling of high-dimensional or multi-level hyperparameter/GAN optimization (Lorraine, 2024).
• Unsupervised hierarchical annotation: Joint image and text structure in multi-modal data (Li et al., 2012).

Limitations observed include increased training complexity when optimizing many hierarchical levels, potential tuning overhead for dynamic/multi-level update frequencies, computational cost for deep-memory optimizers, as well as open questions regarding optimal nesting depth and full theoretical convergence of deeply nested probabilistic objective functions (Behrouz et al., 31 Dec 2025, Jafari et al., 18 Nov 2025, Achddou et al., 2020).

7. Outlook and Generalizations

Ongoing research extends Nested Learning in several directions:

• Automated search for optimal nesting and update schemes via meta-learning and AutoML.
• Bridging neuroscience and nested frequency bands to model continuum memory systems (Behrouz et al., 31 Dec 2025).
• Adaptation of NL concepts to new data modalities (e.g., audio, temporal logs, structural biology) and large foundation models (Rauba et al., 22 Sep 2025).
• Theoretical characterization of nesting benefits in deep architectures, regularization, and transferability.

Nested Learning thus provides a unifying framework to rethink model, optimizer, and objective design through the lens of explicit multi-level structure, achieving interpretable, flexible, and robust learning across a wide range of modern machine learning domains.