Weight Space Learning (WSL)

• Weight Space Learning is a framework that conceptualizes neural network weights as structured data, enabling direct analysis, representation, and generation.
• It encompasses three core dimensions—understanding, representation, and generation—applied to tasks like model retrieval, continual learning, and neural architecture search.
• WSL leverages geometry-based methods, hypernetworks, and optimization-derived representations to address challenges such as symmetry, high dimensionality, and model diversity.

Weight Space Learning (WSL) treats neural network weights themselves as a structured, learnable domain rather than as an opaque end product of training. In this view, a trained model is a point in a high-dimensional weight space, model populations become datasets, and the central problems are to understand, represent, and generate weights directly. A recent survey formalizes the field into three core dimensions—Weight Space Understanding (WSU), Weight Space Representation (WSR), and Weight Space Generation (WSG)—and places applications such as model retrieval, continual and federated learning, neural architecture search, and data-free reconstruction under a single framework (Han et al., 10 Mar 2026).

1. Formal scope and problem formulations

At the most general level, WSL studies learning over the parameter space $\Theta$ of neural networks. If a network is parameterized by weights $\theta \in \Theta$ and realizes a function $f_\theta : \mathcal{X} \to \mathcal{Y}$, then conventional learning optimizes $\theta$ on data in $\mathcal{X}$, whereas WSL studies the structure of $\Theta$ itself, learns mappings $\phi : \Theta \to \mathbb{R}^d$, or learns generators $G_\phi : \mathcal{Z} \to \Theta$ (Han et al., 10 Mar 2026). In the terminology of model-zoo-based work, a model zoo is a dataset of trained networks, and WSL learns from a population of trained models rather than from raw images, texts, or activations (Falk et al., 14 Apr 2025).

A second, complementary formulation appears in neural fields and implicit neural representations. There, each data instance $\mathbf{x}_i$ is represented by the weights of a fitted coordinate network. In the standalone INR setting, one writes

$$\boldsymbol{\theta}_i = \arg\min_{\boldsymbol{\theta}} \mathcal{L}_{\text{recon}}\Big(f(\mathbf{p}\mid\boldsymbol{\theta}),\,\mathbf{x}_i(\mathbf{p})\Big),$$

while in LoRA-based variants one instead optimizes per-instance adaptation weights on top of a frozen base model (Yang et al., 1 Dec 2025). This recasts WSL from “learning over populations of pretrained models” to “using the fit weights themselves as the representation of each sample.”

A third formulation is hypernetwork-based. In the HyperINR framework, a shared hypernetwork $\theta \in \Theta$0 maps an instance-specific latent code $\theta \in \Theta$1 to INR weights $\theta \in \Theta$2, and the reconstructed sample is obtained by evaluating the INR $\theta \in \Theta$3 on coordinates (Qiu et al., 30 Jan 2026). This introduces a low-dimensional latent weight space $\theta \in \Theta$4 whose geometry is intended to preserve the semantics of the original data manifold.

2. Geometry, symmetries, and identifiability

A defining difficulty of WSL is that raw parameter coordinates are not canonical. The survey literature emphasizes permutation, scaling, sign, and related symmetries, so that many different parameter vectors can realize the same function. Formally, if a symmetry group $\theta \in \Theta$5 acts on $\theta \in \Theta$6, then the quotient space $\theta \in \Theta$7 captures equivalence classes $\theta \in \Theta$8 rather than individual coordinates, and symmetry-aware distances take the form $\theta \in \Theta$9 rather than raw $f_\theta : \mathcal{X} \to \mathcal{Y}$0 distance (Han et al., 10 Mar 2026). The neural-field literature states the same issue more directly: unstructured weight space is “chaotic” and ambiguous because of symmetries, permutations, and scaling, so meaningful WSL requires constraints that align axes across instances (Yang et al., 1 Dec 2025).

For INRs this ambiguity is not merely a theoretical nuisance. Weight-space augmentation work makes the one-to-many relation explicit by introducing a conditional distribution $f_\theta : \mathcal{X} \to \mathcal{Y}$1 over neural views $f_\theta : \mathcal{X} \to \mathcal{Y}$2 representing the same object $f_\theta : \mathcal{X} \to \mathcal{Y}$3; a typical INR dataset that stores one fitted network per object therefore under-samples the support of valid representations and induces severe overfitting in downstream weight-space models (Shamsian et al., 2024). This observation is a practical counterpart to quotient-space arguments: meaningful learning over weights requires either architectural equivariance, explicit alignment, or augmentation along symmetry orbits.

A geometric approach makes these ideas operational. In the geodesic framework, the weight space $f_\theta : \mathcal{X} \to \mathcal{Y}$4 is equipped with a task-induced metric

$f_\theta : \mathcal{X} \to \mathcal{Y}$5

where $f_\theta : \mathcal{X} \to \mathcal{Y}$6 is the Jacobian of network outputs with respect to weights (Raghavan et al., 2021). Geodesics in this metric are minimum-functional-change paths: they traverse parameter space while preserving task performance as much as possible. This viewpoint unifies mode connectivity, sparsification, and catastrophic-forgetting mitigation as path-planning problems in a task-weighted parameter manifold.

A related but probabilistic geometric formulation appears in sequential learning. There, weights are treated as the latent state of a Gaussian state-space model with transition

$f_\theta : \mathcal{X} \to \mathcal{Y}$7

and likelihood proportional to $f_\theta : \mathcal{X} \to \mathcal{Y}$8; Laplace approximations then produce Gaussian posteriors over weights, with diagonal-plus-low-rank generalized Gauss–Newton precision updates (Sliwa et al., 2024). In both settings, geometry is not ancillary: it determines which directions in weight space are stable, connected, or semantically meaningful.

3. Representation learning paradigms

WSR methods differ primarily in how they turn a full network into an embedding or descriptor. Some methods encode weights directly; some operate on probes of network behavior; some obtain weight representations by optimization rather than by an explicit encoder.

Paradigm	Core idea	Representative papers
Direct weight encoders	Tokenize or structurally process weights, then learn latent embeddings and reconstructions	(Schürholt et al., 2024, Falk et al., 14 Apr 2025, Falk et al., 2 Oct 2025)
Probe-based representations	Represent a model by its outputs on learned probes	(Kahana et al., 2024)
Optimization-derived representations	Use fitted INR weights, LoRA weights, or latent hypernetwork codes as the representation	(Yang et al., 1 Dec 2025, Qiu et al., 30 Jan 2026)

A scalable direct-encoding line is exemplified by SANE, which tokenizes weights into fixed-size sequences, applies a transformer encoder–decoder, and trains with reconstruction plus contrastive objectives. Its central contribution is sequential processing of subsets of neural network weights, which allows larger neural networks to be embedded as sets of tokens and enables both discriminative and generative tasks in a single framework (Schürholt et al., 2024). The heterogeneous-zoo extension replaces per-layer preprocessing with masked per-token loss normalization, a runtime normalization scheme that is architecture-agnostic, dataset-agnostic, and padding-aware, thereby allowing a single encoder–decoder to train on inhomogeneous populations of models (Falk et al., 14 Apr 2025). The same general line is pushed further by training a weight-space backbone directly on arbitrary Hugging Face models; that work argues that curated model zoos are not indispensable and that high-quality weight-space representations can be learned from highly heterogeneous, largely undocumented public model repositories (Falk et al., 2 Oct 2025).

Probe-based WSL takes a different route. Instead of encoding raw weights, it represents a model by its outputs on learned synthetic inputs. The ProbeGen formulation adds a shared generator module with a deep linear architecture to produce structured probes, reducing overfitting relative to direct latent optimization of probes. On standard WSL benchmarks, ProbeGen is reported to require between 30 to 1000 times fewer FLOPs than other top approaches while outperforming prior weight-space baselines (Kahana et al., 2024). This replaces permutation-sensitive parameter coordinates by a function-space signature, but the learned object is still a weight-space representation of the model.

Optimization-derived representations are especially prominent in INRs and neural fields. In HyperINR, latent codes $f_\theta : \mathcal{X} \to \mathcal{Y}$9 are optimized jointly with a hypernetwork so that the induced INR weights reconstruct each sample; the Implicit Function Theorem is then used to show that, under full-rank Hessian conditions, there exists a unique smooth map from data space to latent weight representation space (Qiu et al., 30 Jan 2026). In modulated neural fields, per-instance multiplicative LoRA weights $\theta$0 are optimized on top of a frozen base network, and the resulting fitted weights serve directly as the representation. The key empirical claim is that a base model, low-rank multiplicative adaptation, and symmetry breaking can induce a semantically organized manifold in weight space without any encoder network at all (Yang et al., 1 Dec 2025).

4. Generation and synthesis in weight space

Weight Space Generation encompasses hypernetworks, autoencoders, diffusion models, autoregressive models, and flow-based methods that synthesize new parameters rather than only analyzing existing ones (Han et al., 10 Mar 2026). In practice, many WSG systems learn a latent distribution over model weights and decode samples into executable networks.

SANE established that tokenized autoencoding can support generation as well as analysis: the model can sequentially generate unseen neural network models, provide useful zero-shot initializations, and support transfer to larger ResNet architectures (Schürholt et al., 2024). The heterogeneous-zoo study sharpens the role of diversity: when SANE was trained on a combined CIFAR10+CIFAR100 zoo, the mean zero-shot accuracy of generated ResNet-18 models across CIFAR10, CIFAR100, TinyImageNet, SVHN, and EuroSAT rose to $\theta$1, compared with $\theta$2 for CIFAR10-only training and $\theta$3 for CIFAR100-only training, and the authors emphasize that the benefit is not only more models but also more diverse models (Falk et al., 14 Apr 2025). Training directly on public hubs extends the same point: a backbone trained on arbitrary Hugging Face models often outperforms backbones trained on laboratory-generated zoos and can generalize to unseen data modalities (Falk et al., 2 Oct 2025).

A particularly strong generative result appears in neural fields. There, diffusion is trained not on full MLP weights but on low-rank multiplicative LoRA codes on top of a shared base neural field. On FFHQ, the best variant, mLoRA-Asym, attains FD $\theta$4, MMD-G $\theta$5, and MMD-P $\theta$6; on multi-class ShapeNet it reaches FD $\theta$7, MMD-G $\theta$8, MMD-P $\theta$9, COV $\mathcal{X}$0, and $\mathcal{X}$1-NNA $\mathcal{X}$2 (Yang et al., 1 Dec 2025). The same representation also improves reconstruction, with mLoRA-Asym reaching PSNR $\mathcal{X}$3 on FFHQ and CD-A $\mathcal{X}$4, CD-M $\mathcal{X}$5 on ShapeNet. The broader implication is that generative quality is tightly coupled to the geometry induced by the weight parameterization.

Exploratory work in 3D generation studies weight space more directly as a geometric object. In a large 3D shape generator, interpolation in conditioning space induces trajectories in effective weight space that display a sharp phase transition in global connectivity, while PCA-based low-dimensional reparameterizations yield controlled local geometry changes with very limited data (Plattner et al., 26 Mar 2025). This does not yet amount to a standardized generative pipeline, but it supports the view that submanifolds of weight space can modulate topology and fine-grained geometry separately.

5. Adaptation, control, and sequential decision making

A major branch of WSL studies how parameter-space structure can be used to control adaptation without treating weights as static objects. In continual learning, Laplace–Gaussian filtering and smoothing formalize the weights $\mathcal{X}$6 of a neural network as the latent state of a sequential Bayesian model. Each task yields a Gaussian posterior over weights, updated through structured curvature approximations, and a backward smoothing pass can improve task-specific models without re-accessing any data (Sliwa et al., 2024). In this setting, weight space is the substrate for stability–plasticity trade-offs rather than only a representation domain.

The geodesic framework gives a different control mechanism. By equipping parameter space with the Jacobian-induced metric $\mathcal{X}$7, it constructs minimum-length paths that preserve functional performance while moving toward a secondary objective. The same machinery is used for three applications: network sparsification, mitigating catastrophic forgetting by constructing networks with high performance on a series of objectives, and finding high-accuracy paths connecting distinct local optima of deep networks (Raghavan et al., 2021). Here WSL appears as geometry-driven navigation of the loss landscape.

In few-shot class-incremental learning, explicit operations in classifier-head weight space can replace prediction-space ensembles. The Tripartite Weight-Space Ensemble interpolates the base, immediately previous, and current models in weight space, especially for the classification heads, via class-dependent convex combinations. On miniImageNet it reaches an average accuracy of $\mathcal{X}$8 and a last-session accuracy of $\mathcal{X}$9, outperforming variants without weight-space ensembling (Lee et al., 4 Jun 2025). The resulting view is that catastrophic forgetting can be constrained by keeping the effective classifier inside a learned simplex in weight space.

Robotic adaptation extends the same logic to large multimodal policies. WIZARD is a weight-space meta-learning framework for Vision-Language-Action models that predicts task-specific LoRA parameters for a frozen policy from only a language instruction and a short demonstration video, in a single forward pass and without target-task action labels or test-time optimization. On LIBERO, it improves performance by up to $\Theta$0 on unseen dataset collections and up to $\Theta$1 on unseen tasks, and on a Franka Emika Panda it improves over a real-domain adapted baseline (Bianchi et al., 5 Jun 2026). In this regime, WSL means learning a map from task evidence directly into policy-adaptation weights.

6. Infrastructure, evaluation, and open questions

WSL depends critically on the availability and structure of model populations. An early empirical study framed trained classifiers as points in neural weight space, trained deep meta-classifiers to identify footprints of training setup in those weights, and released the Neural Weight Space dataset with 320K weight snapshots from 16K individually trained deep neural networks (Eilertsen et al., 2020). More recent infrastructure makes the structure of model populations itself a research variable. A model zoo on phase transitions introduces 12 large-scale zoos that systematically cover known phases and vary over model architecture, size, and datasets across computer vision, NLP, and scientific machine learning, with loss landscape metrics such as train loss, Hessian measures, mode connectivity, and CKA computed for every model (Schürholt et al., 25 Apr 2025). This makes phase information an explicit notion of diversity for WSL rather than an accidental by-product.

Evaluation has also become more nuanced. A common misconception is that if a metadata factor is encoded in embeddings, then a predictor must be using it. Weight Space Correlation Analysis distinguishes these cases by measuring alignment between the classification heads of a primary task and auxiliary metadata tasks. In the medical-imaging study, embeddings contained substantial metadata, but the sPTB classifier’s weight vectors were highly correlated with clinically relevant factors such as birth weight while being decoupled from clinically irrelevant acquisition factors such as scanner (Wong et al., 15 Dec 2025). This establishes that “encodability” and “utilization” are distinct weight-space questions.

Another recurring issue is generalization. Deep weight spaces suffer from substantial overfitting when the dataset contains too little diversity in valid representations. For INR classification, weight-space augmentations and a MixUp method adapted to weight space improve performance similarly to having up to 10 times more data, and in self-supervised contrastive learning they yield substantial 5–10% gains in downstream classification (Shamsian et al., 2024). This supports a broader conclusion that WSL datasets should approximate the diversity of valid neural views rather than storing one canonical model per underlying object.

Several open problems remain central. The survey identifies high dimensionality and complexity, symmetry-induced redundancy, scalability to large models, evaluation difficulties, safety and robustness, and incomplete theoretical foundations as persistent challenges (Han et al., 10 Mar 2026). More specialized studies add practical limitations: heterogeneous zoos require normalization schemes that do not depend on aligned layer indices (Falk et al., 14 Apr 2025); public model hubs remove the need for curated zoos but introduce undocumented datasets, unknown hyperparameters, and selection bias (Falk et al., 2 Oct 2025). A consistent theme across the literature is that WSL becomes effective only when parameterization, symmetry handling, and data curation are treated as first-class methodological objects.