---
title: Neural Weight Space Dataset
url: https://www.emergentmind.com/topics/neural-weight-space-dataset
type: topic
---

# Neural Weight Space Dataset

Searching arXiv for relevant papers on neural weight space datasets and related model zoo resources.
arXiv search query: "neural weight space dataset model zoo weights as data"
A neural weight space dataset is a collection in which trained neural network parameters, checkpoints, or structured weight objects are treated as data points rather than merely as by-products of optimization. Within Weight Space Learning (WSL), this reframing supports three broad activities: understanding the geometry and symmetries of weight space, learning representations over weights, and generating new weights through hypernetworks or generative models [2603.10090]. In practice, such datasets range from controlled model zoos of many independently trained networks, to per-instance implicit neural representation (INR) weights, to paired resources such as text-weight corpora or phase-annotated checkpoint collections. The resulting objects can encode training setup, dataset provenance, optimization history, semantic content, or task structure, depending on how the dataset is constructed [2002.05688].

## 1. Definition and conceptual scope

A neural weight space dataset treats a model weight vector, a checkpoint trajectory, or a structured parameter object as an observational unit. In the formulation used by Eilertsen et al., each trained model is interpreted as a point in a high-dimensional neural weight space, and the dataset is built by sampling many training variations and collecting their weights [2002.05688]. The WSL survey formalizes this more broadly by treating neural weights as a meaningful domain for analysis and modeling, with datasets serving as the substrate for Weight Space Understanding, Weight Space Representation, and Weight Space Generation [2603.10090].

Two distinct dataset paradigms recur in the literature. The first is the **model zoo** paradigm, in which many networks are trained under controlled or partially controlled protocols and saved together with metadata, checkpoints, or landscape statistics. The second is the **instance-as-network** paradigm, common in INRs, where each data sample is first encoded as its own neural network and the collection of resulting parameters becomes the dataset. HyperDiffusion exemplifies the latter: for each sample \(S_i\), a separate MLP \(f(\mathbf{x}, \theta_i)\) is trained, the optimized weights \(\theta_i \in \mathbb{R}^h\) are flattened into 1D vectors, and diffusion is then trained directly on this MLP weight space [2303.17015].

This distinction matters because the semantics of a weight-space data point depend on the generating process. In model zoos, a data point usually corresponds to a model trained on a task under a sampled configuration. In INR datasets, a data point often corresponds directly to one underlying object, image, shape, or animation. This suggests that “neural weight space dataset” is not a single format but a family of dataset constructions whose common feature is that the primary modality is the learned parameterization itself.

## 2. Canonical construction patterns

Most neural weight space datasets are built through one of four recurring procedures.

The first is **large-scale sampling over training configurations**. NWS trains 16,000 CNNs by randomly sampling datasets, architectures, optimizers, learning rates, batch sizes, augmentation, activation functions, and initialization, and exports 20 weight snapshots per model, yielding **320,000 weight vectors** [2002.05688]. MLDS instead fixes architecture within each sub-dataset and varies random initialization and training data, ultimately collecting over **180,000 trained neural networks** across several RNN-based sub-datasets [2104.10555].

The second is **checkpoint trajectory capture**. Instead of storing only terminal models, some datasets preserve the temporal path through weight space. NWS always includes initial and final weights among its 20 snapshots per training run [2002.05688]. In a smaller but explicitly trajectory-oriented setting, a model zoo for monitoring training progress stores epoch-wise weight vectors \(W_i = \{W_i^1, W_i^2, ..., W_i^k\}\) for 250 MLPs and 250 CNNs, enabling analysis of smoothness, separability, and ordering in weight space [2006.10424].

The third is **per-sample network fitting**. HyperDiffusion constructs its dataset by “per-sample MLP overfitting”: each sample obtains its own MLP with fixed architecture—3 hidden layers, 128 neurons, ReLU, and positional encoding—and the optimized parameters are flattened into vectors of dimension \(h \sim 36{,}000\) [2303.17015]. Work on deep weight spaces later identifies a major limitation of this recipe: typical INR datasets use only a single neural view per object, even though one object admits many valid INRs, thereby reducing diversity and encouraging overfitting [2402.04081].

The fourth is **paired or annotated dataset construction**. Text2Weight introduces an open-source dataset of **36,000 pairs**, with **12,000 pairs** each from CIFAR-100, Caltech-256, and TinyImageNet; each entry contains a natural language task description, a flattened **16,384-dimensional** weight vector for a two-layer adapter, and task metadata [2508.13633]. A different annotation strategy appears in the phase-transition model zoos, which introduce **12 large-scale zoos**, around **2,500 unique trained models**, and **60,000+ checkpoints**, each tagged with performance metrics, loss landscape metrics, and phase assignments [2504.18072].

## 3. Representative datasets and resources

The literature now includes several recurring reference datasets and dataset families.

| Resource | Content | Distinctive property |
|---|---|---|
| NWS | 320,000 snapshots from 16,000 CNN trainings | Randomized hyperparameters and trajectory snapshots |
| MLDS | 180,000+ trained RNNs | Controlled architecture and training-data variation |
| Phase transition model zoos | 12 zoos, ~2,500 models, 60,000+ checkpoints | Explicit phase coverage and landscape annotation |
| HyperDiffusion weight set | Flattened per-sample INR MLP weights | Dataset built from object-specific MLPs |
| Text2Weight dataset | 36,000 text-weight pairs | Natural-language conditioning of weights |
| weights2weights collection | Over 60,000 customized diffusion models | Fine-tuned LoRA weights as semantic dataset |

The **NWS dataset** was introduced specifically to promote research on the neural weight space. It contains randomized small CNNs with 6–10 layers and between approximately 20,000 and 390,000 trainable weights per model, trained on MNIST, CIFAR-10, SVHN, STL-10, and Fashion-MNIST [2002.05688]. Because all trainable parameters are vectorized layer by layer, including biases and batch normalization parameters, NWS became a reference benchmark for meta-classification and weight-space analysis.

**MLDS** emphasizes controlled comparison. In MLDS-DS1 and MLDS-DS2, all networks in a given sub-dataset share an identical architecture—4 GRU layers, 4 linear layers, and a single output layer—with **4,364** total parameters, while DS3 uses a larger LSTM-based architecture with **136,846** parameters [2104.10555]. The paper shows that networks trained on identical data cluster in weight space, whereas even small changes to the training data induce meaningful divergence.

The **phase transition model zoos** generalize the model-zoo idea beyond ordinary hyperparameter sweeps by explicitly targeting known phases of neural network training. Their design spans computer vision, NLP, and scientific machine learning, with architectures including ResNet-18, ResNet-50, Vision Transformers, GPT-2, and PINNs, and provides metrics such as Hessian sharpness, mode connectivity, and CKA similarity for every model [2504.18072].

The **Text2Weight** dataset is a paired dataset rather than a conventional zoo. Each example contains a list of template-generated class descriptions such as “A photo of otter,” a CLIP-derived task embedding, and the flattened parameters of a trainable projection head with \(W_1 \in \mathbb{R}^{512 \times 16}\) and \(W_2 \in \mathbb{R}^{16 \times 512}\), yielding \(\theta \in \mathbb{R}^{16{,}384}\) [2508.13633]. The train/test split uses 80% seen tasks and 20% unseen tasks, with **no class overlap** between train and test subtasks per base dataset.

The **weights2weights** collection occupies another point in the design space. It consists of over **60,000 models**, described more specifically as \(N \approx 65{,}000\) identity-specific weight vectors in a roughly **100,000-dimensional** space, produced by DreamBooth fine-tuning with rank-1 LoRA on a latent diffusion base model [2406.09413]. Here the dataset is neither a generic zoo nor an INR set; it is a population of customized diffusion adapters whose structure supports sampling, editing, and inversion in a PCA-defined subspace.

## 4. Representation, annotation, and formal structure

Neural weight space datasets require explicit choices about how weights are serialized, normalized, and labeled. In NWS, vectorization proceeds layer by layer. For convolutional layers,
\[
\theta_i = \theta_{i-1} \shortparallel \mathrm{vec}(\mathbf{h}_{i,1}) \shortparallel \cdots \shortparallel \mathrm{vec}(\mathbf{h}_{i,K}) \shortparallel \mathbf{b}_i,
\]
and for fully connected layers,
\[
\theta_i = \theta_{i-1} \shortparallel \mathrm{vec}(\mathbf{W}_i) \shortparallel \mathbf{b}_i,
\]
with the final model weight vector \(\theta = \theta_L\) [2002.05688]. HyperDiffusion follows a similar flattening strategy for per-sample MLPs, but also reports a dataset-construction detail intended to smooth the induced weight-space distribution: every sample-specific MLP is optimized from the weights of a single reference MLP rather than from random initialization [2303.17015].

Some datasets preserve internal tensor structure instead of relying only on flat vectors. WeightFlow constructs neural network weights into a graph in which each neuron in linear layers becomes a node and node features are concatenations of incoming weights and biases [2508.00451]. This is not a conventional benchmark release, but it defines a representation strategy for datasets of time-indexed network weights and anchor models.

Annotations are equally important. NWS labels each weight snapshot with training setup variables such as optimizer, dataset, activation, initialization, and architecture choice, enabling mappings of the form \(g: \theta \rightarrow \phi_c\) for meta-classification [2002.05688]. The phase-transition zoos attach train/validation/test performance, Hessian-derived sharpness, mode connectivity, and CKA-based representation similarity, then assign categorical phases using those metrics [2504.18072]. Text2Weight stores dataset source, subtask class list, split membership, training configuration, and fused CLIP text embeddings alongside the weights [2508.13633].

At the theoretical level, recent work argues that the semantics encoded by such datasets need not be accidental. In a HyperINR setting, the Implicit Function Theorem is used to justify a local unique and differentiable mapping from a data sample \(\bm{X}\) to a latent code \(\bm{z}^*\), and from there to generated INR weights \(\bm{w}_j^*\), under a full-rank Hessian condition [2601.23181]. This suggests a principled link between data manifolds and weight-space datasets in settings where each instance is mapped into weights through a shared hypernetwork.

## 5. What neural weight space datasets reveal

These datasets support several kinds of empirical claims about weight space.

First, they reveal that weights encode training provenance. NWS shows that meta-classifiers can infer many hyperparameters from trained weights alone, and that even small local subsets of consecutive weights can be informative [2002.05688]. MLDS similarly finds clustering of models with identical training data and meaningful divergence with even small changes to the training data, including “backdoor” variants [2104.10555].

Second, they reveal trajectory structure. In the model-zoo study on training monitoring, DNN models evolve on unique, smooth trajectories in weight space, trajectories can be separated by single-linkage clustering, and epoch order can be partially recovered from distances along the path [2006.10424]. This suggests that checkpoint-based weight datasets can function as records of optimization dynamics rather than merely archives of endpoints.

Third, they reveal that dataset design strongly affects generalization in weight-space models. The augmentation studies on INRs argue that overfitting in deep weight space models largely stems from the lack of diversity in DWS datasets: a given object admits many neural views, but typical datasets store only one [2402.04081]. Adding multiple neural views per object mitigates overfitting and improves both “internal generalization” to new views of training objects and “external generalization” to novel objects; alignment-based MixUp in weight space can approximate the effect of substantially more data [2311.08851].

Fourth, they reveal semantic regularities in specialized collections. In weights2weights, principal component coefficients of LoRA adapters are approximately independent and Gaussian-distributed, and linear directions in the resulting subspace correspond to semantic edits such as adding a beard [2406.09413]. In Text2Weight, a paired text-weight dataset enables learning \(p(\theta \mid c)\), linking natural language task descriptions directly to classifier-head parameters [2508.13633].

## 6. Applications, limitations, and current directions

The applications of neural weight space datasets now span analysis, representation learning, and generation. ProbeGen uses collections of model weights to predict properties such as dataset origin or generalization error through learned probes passed through the target models, and reports between **30 and 1000 times fewer FLOPs** than other top approaches [2410.10811]. HyperDiffusion trains diffusion directly in the space of flattened INR weights and reports evaluation metrics including MMD, COV, 1-NNA, and FPD on ShapeNet and DeformingThings4D [2303.17015]. DeepWeightFlow uses datasets of fully trained, canonicalized models to generate complete neural network weights through Flow Matching, with Git Re-Basin or TransFusion used to address permutation symmetries for MLPs, ResNets, ViTs, and BERT [2601.05052].

Several limitations recur across the literature. **Dataset creation overhead** is central in INR-based collections because each sample requires a full optimization run; HyperDiffusion quantifies this as **6 min/shape × data size** [2303.17015]. **Lack of diversity** is another major issue in deep weight space datasets derived from INRs, where one neural view per object is often insufficient [2402.04081]. **Permutation symmetry and canonicalization** remain structural complications for generative modeling and comparison, motivating alignment procedures such as Git Re-Basin, weight matching for MixUp, and TransFusion [2311.08851]. **Heterogeneity** is a distinct challenge once the field moves beyond curated zoos: training on public repositories such as Hugging Face requires tokenization and normalization schemes that can handle missing metadata, architecture variation, and sparse documentation [2510.02096].

Current directions therefore move along two axes. One is toward **more structured curation**, exemplified by phase-aware model zoos with explicit coverage criteria and loss-landscape annotations [2504.18072]. The other is toward **less curated but larger-scale aggregation**, exemplified by weight-space backbones trained on arbitrary models downloaded from public model hubs, where curated zoos are no longer treated as indispensable [2510.02096]. Taken together, these developments indicate that a neural weight space dataset is increasingly understood not merely as a static archive of checkpoints, but as a research object whose construction protocol, symmetry handling, annotation scheme, and diversity profile determine what can be learned from neural weights as a modality [2603.10090].

Source: https://www.emergentmind.com/topics/neural-weight-space-dataset