A Survey of Weight Space Learning: Understanding, Representation, and Generation

Abstract: Neural network weights are typically viewed as the end product of training, while most deep learning research focuses on data, features, and architectures. However, recent advances show that the set of all possible weight values (weight space) itself contains rich structure: pretrained models form organized distributions, exhibit symmetries, and can be embedded, compared, or even generated. Understanding such structures has tremendous impact on how neural networks are analyzed and compared, and on how knowledge is transferred across models, beyond individual training instances. This emerging research direction, which we refer to as Weight Space Learning (WSL), treats neural weights as a meaningful domain for analysis and modeling. This survey provides the first unified taxonomy of WSL. We categorize existing methods into three core dimensions: Weight Space Understanding (WSU), which studies the geometry and symmetries of weights; Weight Space Representation (WSR), which learns embeddings over model weights; and Weight Space Generation (WSG), which synthesizes new weights through hypernetworks or generative models. We further show how these developments enable practical applications, including model retrieval, continual and federated learning, neural architecture search, and data-free reconstruction. By consolidating fragmented progress under a coherent framework, this survey highlights weight space as a learnable, structured domain with growing impact across model analysis, transferring, and weight generation. We release an accompanying resource at https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning.

• The paper introduces a taxonomy of Weight Space Learning, detailing its three dimensions: understanding, representation, and generation.
• Methodologies exploit symmetry and manifold geometry to enhance optimization, model identifiability, and cross-model comparison.
• Generative techniques, including hypernetworks and diffusion models, enable rapid weight synthesis for adaptable, robust model merging.

Survey of Weight Space Learning: Understanding, Representation, and Generation

Introduction

The paper "A Survey of Weight Space Learning: Understanding, Representation, and Generation" (2603.10090) offers a comprehensive taxonomy and synthesis of Weight Space Learning (WSL), positioning neural network weights as a structured, learnable data modality. Historically, model weights have been considered mere byproducts of training, with predominant research focusing on input data, features, and architectures. However, recent advances show that the collective space of trained model weights possesses intrinsic organizational structure—exhibiting distributions, symmetries, and manifold properties—that enables deeper algorithmic and theoretical understanding, efficient model comparison, and direct knowledge transfer across models.

The survey delineates WSL into three main dimensions: Weight Space Understanding (WSU), Weight Space Representation (WSR), and Weight Space Generation (WSG). Each dimension is analyzed through foundational principles, algorithmic developments, representative works, and practical implications, forming a unified framework for both the theoretical and empirical study of neural weight space.

Weight Space Understanding

The WSU dimension interrogates the intrinsic geometry and topology of weight space, abstracted from particular datasets or training regimes. It underscores that weight spaces are not flat, unstructured domains; symmetries such as neuron permutations and scaling invariance result in large equivalence classes of functionally identical parameterizations. These symmetries induce both invariance (transformed weights yield identical functions) and equivariance (parameter transformations produce predictable functional changes). Recognizing and formalizing these symmetries yields theoretical advances in model identifiability, landscape connectivity, and optimization degeneracy, and motivates symmetry-invariant representations and algorithms.

WSU-driven methods include:

• Lossless compression exploiting permutation and scaling invariance: Approaches remove redundant parameters by aligning functionally equivalent parts of the network [sourek2021lossless, ganev2021universal].
• Symmetry-invariant and equivariant optimization: Algorithms such as Path-SGD, G-SGD, and projection-based weight normalization navigate oblique manifolds and operate directly on quotient spaces [neyshabur2015path, meng2018gsgd, huang2020projection].
• Weight space augmentation: Techniques like MixUp and equivariant augmentation interpolate or transform weights within identified symmetry classes to generate semantically consistent model instances [shamsian2023data, shamsian2024improved, navon2024equivariant].

WSU forms the theoretical substrate for subsequent WSR and WSG developments, moving model-centric analysis from parameter-centric to manifold-centric perspectives.

Weight Space Representation

The WSR dimension focuses on encoding weights into compact latent representations amenable to downstream reasoning, retrieval, and comparison. The representation function $\phi$ maps weight space to low-dimensional embeddings that capture structural regularities and functional semantics.

WSR methodologies are divided into two paradigms:

• Model-based representations: These include symmetry-agnostic encoders (statistics-based or high-dimensional regressors), symmetry-aware functionals (explicitly respecting group actions like permutation and scaling), and increasingly, graph-based metanetworks that leverage GNNs to encode computational dependencies and architectural symmetries [navon2023equivariant, zhou2023permutation, lim2024graph, kalogeropoulos2024scale].
• Model-free representations: Probing-based behavioral learning infers network embeddings from functional outputs on reference inputs, bypassing raw weight access and providing architecture-agnostic, symmetry-respecting descriptors [kahana2025deep, herrmann2024learning, horwitz2025learning].

WSR enables high-fidelity tasks such as model-level property prediction, cross-architecture retrieval, and latent-space model editing. Embedding-based frameworks transform model evaluation from data-centric to parameter-centric, supporting large-scale automated analysis with minimal human intervention.

Weight Space Generation

The WSG dimension extends the paradigm to the synthesis of model weights, leveraging hypernetworks and generative models to instantiate, adapt, or reconstruct neural weights.

WSG approaches include:

• Hypernetworks: Auxiliary networks generate weights conditioned on prompts (task, data, architectural description), trained end-to-end via downstream loss signals. Hypernetworks facilitate rapid adaptation, parameter efficiency, and modular or hierarchical weight synthesis [ha2017hypernetworks, krueger2017bayesian, zhang2018graph, ruiz2024hyperdreambooth].
• Generative models: Techniques such as VAE, GAN, autoregressive models, and diffusion-based denoising learn explicit distributions over weight manifolds. These models support diverse, architecture-agnostic generation and enable weight space exploration and functional interpolation beyond seen checkpoints [schurholt2021self, peebles2022learning, erkocc2023hyperdiffusion, jin2024conditional, wu2024difflora].

WSG unlocks practical applications including conditional weight generation (domain adaptation, continual learning, federated personalization), real-time optimization (fast adaptation via forward synthesis), robust model merging (alignment and latent-space fusion), initialization (knowledge-aware weight sampling), and data generation (INRs or neural radiance fields synthesized from weight space).

Practical Applications and Benchmarks

WSL methodologies have been successfully deployed across domains such as implicit neural representations (INR), model unification, continual/meta/federated learning, and neural architecture search. By representing and manipulating models directly in weight space, WSL supports scalable retrieval, lifelong adaptation, and efficient architecture evaluation without retraining.

Empirical progress is facilitated by large-scale benchmark "model zoos," including diverse collections of MLPs, CNNs, RNNs, and Transformer-based models, which enable rigorous evaluation and comparative analysis of weight space methods. Model zoos are crucial for the development and scalability of WSL, offering both breadth and depth of pretrained weight samples across varied architectures and domains.

Numerical Findings and Claims

• Model-level accuracy prediction: Weight embeddings achieve statistically significant performance in zero-shot accuracy regression and model retrieval tasks [unterthiner2020predicting, eilertsen2020classifying].
• Zero-shot weight synthesis: Diffusion- and hypernetwork-based generative models can sample weights for unseen architectures and tasks, achieving comparable or superior performance to traditional fine-tuning in continual/meta/federated settings [peebles2022learning, ruiz2024hyperdreambooth, jin2024conditional].
• Efficient model merging: Latent alignment and symmetry-invariant merging algorithms preserve task performance and generalization even across independently trained models [ainsworth2023git, navon2024equivariant].

Several bold assertions are substantiated: pretrained weights encode task-independent structural regularities; weight space can be treated as a geometric manifold supporting representation learning and generative modeling; the geometry of weight space (e.g., mode connectivity, symmetry-induced subspaces) determines optimization dynamics.

Implications and Future Directions

WSL redefines the algorithmic landscape, transforming the manipulation of neural networks from data- or architecture-centric to weight-centric. Theoretical advances in symmetry, manifold geometry, and structural invariances enable scalable architecture-agnostic algorithms, enhance interpretability, and support robust adaptation. Practical implications include accelerated training, efficient model deployment, and new vistas in data-free learning, continual adaptation, and distributed training.

Future directions include:

• Scaling weight space operations to extremely large models (LLMs, vision transformers), via modular and hierarchical processing, efficient compression, and fine-tuning module generation.
• Developing universal, architecture-agnostic representation and generation models.
• Formalizing robustness and safety for weight space operations, including adversarial risk detection, defense mechanisms, and controllable weight synthesis.

WSL is poised for cross-disciplinary integration, uniting geometric deep learning, differential geometry, representation theory, and generative modeling to form a foundational substrate for next-generation model-centric machine learning.

Conclusion

This survey synthesizes and systematizes Weight Space Learning as an emergent research paradigm. By elevating neural network weights to a first-class, structured learning domain, WSL opens the path for principled analysis, scalable representation, and generative synthesis across architectures and tasks. As pretrained models proliferate, WSL is expected to become a foundational perspective for model analysis, creation, adaptation, and robust deployment in AI research and application.