Neural Weight Space Dataset

• Neural weight space datasets are collections of trained model parameters treated as individual data points for analyzing training dynamics and geometry.
• They are constructed using varied protocols such as model zoos, checkpoint trajectories, per-sample network fitting, and paired annotation to capture diverse training regimes.
• These datasets reveal training provenance, smooth optimization trajectories, and semantic regularities that enhance understanding of generalization and generative modeling.

Searching arXiv for relevant papers on neural weight space datasets and related model zoo resources. arXiv search query: "neural weight space dataset model zoo weights as data" A neural weight space dataset is a collection in which trained neural network parameters, checkpoints, or structured weight objects are treated as data points rather than merely as by-products of optimization. Within Weight Space Learning (WSL), this reframing supports three broad activities: understanding the geometry and symmetries of weight space, learning representations over weights, and generating new weights through hypernetworks or generative models (Han et al., 10 Mar 2026). In practice, such datasets range from controlled model zoos of many independently trained networks, to per-instance implicit neural representation (INR) weights, to paired resources such as text-weight corpora or phase-annotated checkpoint collections. The resulting objects can encode training setup, dataset provenance, optimization history, semantic content, or task structure, depending on how the dataset is constructed (Eilertsen et al., 2020).

1. Definition and conceptual scope

A neural weight space dataset treats a model weight vector, a checkpoint trajectory, or a structured parameter object as an observational unit. In the formulation used by Eilertsen et al., each trained model is interpreted as a point in a high-dimensional neural weight space, and the dataset is built by sampling many training variations and collecting their weights (Eilertsen et al., 2020). The WSL survey formalizes this more broadly by treating neural weights as a meaningful domain for analysis and modeling, with datasets serving as the substrate for Weight Space Understanding, Weight Space Representation, and Weight Space Generation (Han et al., 10 Mar 2026).

Two distinct dataset paradigms recur in the literature. The first is the model zoo paradigm, in which many networks are trained under controlled or partially controlled protocols and saved together with metadata, checkpoints, or landscape statistics. The second is the instance-as-network paradigm, common in INRs, where each data sample is first encoded as its own neural network and the collection of resulting parameters becomes the dataset. HyperDiffusion exemplifies the latter: for each sample $S_i$, a separate MLP $f(\mathbf{x}, \theta_i)$ is trained, the optimized weights $\theta_i \in \mathbb{R}^h$ are flattened into 1D vectors, and diffusion is then trained directly on this MLP weight space (Erkoç et al., 2023).

This distinction matters because the semantics of a weight-space data point depend on the generating process. In model zoos, a data point usually corresponds to a model trained on a task under a sampled configuration. In INR datasets, a data point often corresponds directly to one underlying object, image, shape, or animation. This suggests that “neural weight space dataset” is not a single format but a family of dataset constructions whose common feature is that the primary modality is the learned parameterization itself.

2. Canonical construction patterns

Most neural weight space datasets are built through one of four recurring procedures.

The first is large-scale sampling over training configurations. NWS trains 16,000 CNNs by randomly sampling datasets, architectures, optimizers, learning rates, batch sizes, augmentation, activation functions, and initialization, and exports 20 weight snapshots per model, yielding 320,000 weight vectors (Eilertsen et al., 2020). MLDS instead fixes architecture within each sub-dataset and varies random initialization and training data, ultimately collecting over 180,000 trained neural networks across several RNN-based sub-datasets (Clemens, 2021).

The second is checkpoint trajectory capture. Instead of storing only terminal models, some datasets preserve the temporal path through weight space. NWS always includes initial and final weights among its 20 snapshots per training run (Eilertsen et al., 2020). In a smaller but explicitly trajectory-oriented setting, a model zoo for monitoring training progress stores epoch-wise weight vectors $W_i = \{W_i^1, W_i^2, ..., W_i^k\}$ for 250 MLPs and 250 CNNs, enabling analysis of smoothness, separability, and ordering in weight space (Schürholt et al., 2020).

The third is per-sample network fitting. HyperDiffusion constructs its dataset by “per-sample MLP overfitting”: each sample obtains its own MLP with fixed architecture—3 hidden layers, 128 neurons, ReLU, and positional encoding—and the optimized parameters are flattened into vectors of dimension $h \sim 36{,}000$ (Erkoç et al., 2023). Work on deep weight spaces later identifies a major limitation of this recipe: typical INR datasets use only a single neural view per object, even though one object admits many valid INRs, thereby reducing diversity and encouraging overfitting (Shamsian et al., 2024).

The fourth is paired or annotated dataset construction. Text2Weight introduces an open-source dataset of 36,000 pairs, with 12,000 pairs each from CIFAR-100, Caltech-256, and TinyImageNet; each entry contains a natural language task description, a flattened 16,384-dimensional weight vector for a two-layer adapter, and task metadata (Tian et al., 19 Aug 2025). A different annotation strategy appears in the phase-transition model zoos, which introduce 12 large-scale zoos, around 2,500 unique trained models, and 60,000+ checkpoints, each tagged with performance metrics, loss landscape metrics, and phase assignments (Schürholt et al., 25 Apr 2025).

3. Representative datasets and resources

The literature now includes several recurring reference datasets and dataset families.

Resource	Content	Distinctive property
NWS	320,000 snapshots from 16,000 CNN trainings	Randomized hyperparameters and trajectory snapshots
MLDS	180,000+ trained RNNs	Controlled architecture and training-data variation
Phase transition model zoos	12 zoos, ~2,500 models, 60,000+ checkpoints	Explicit phase coverage and landscape annotation
HyperDiffusion weight set	Flattened per-sample INR MLP weights	Dataset built from object-specific MLPs
Text2Weight dataset	36,000 text-weight pairs	Natural-language conditioning of weights
weights2weights collection	Over 60,000 customized diffusion models	Fine-tuned LoRA weights as semantic dataset

The NWS dataset was introduced specifically to promote research on the neural weight space. It contains randomized small CNNs with 6–10 layers and between approximately 20,000 and 390,000 trainable weights per model, trained on MNIST, CIFAR-10, SVHN, STL-10, and Fashion-MNIST (Eilertsen et al., 2020). Because all trainable parameters are vectorized layer by layer, including biases and batch normalization parameters, NWS became a reference benchmark for meta-classification and weight-space analysis.

MLDS emphasizes controlled comparison. In MLDS-DS1 and MLDS-DS2, all networks in a given sub-dataset share an identical architecture—4 GRU layers, 4 linear layers, and a single output layer—with 4,364 total parameters, while DS3 uses a larger LSTM-based architecture with 136,846 parameters (Clemens, 2021). The paper shows that networks trained on identical data cluster in weight space, whereas even small changes to the training data induce meaningful divergence.

The phase transition model zoos generalize the model-zoo idea beyond ordinary hyperparameter sweeps by explicitly targeting known phases of neural network training. Their design spans computer vision, NLP, and scientific machine learning, with architectures including ResNet-18, ResNet-50, Vision Transformers, GPT-2, and PINNs, and provides metrics such as Hessian sharpness, mode connectivity, and CKA similarity for every model (Schürholt et al., 25 Apr 2025).

The Text2Weight dataset is a paired dataset rather than a conventional zoo. Each example contains a list of template-generated class descriptions such as “A photo of otter,” a CLIP-derived task embedding, and the flattened parameters of a trainable projection head with $W_1 \in \mathbb{R}^{512 \times 16}$ and $W_2 \in \mathbb{R}^{16 \times 512}$, yielding $\theta \in \mathbb{R}^{16{,}384}$ (Tian et al., 19 Aug 2025). The train/test split uses 80% seen tasks and 20% unseen tasks, with no class overlap between train and test subtasks per base dataset.

The weights2weights collection occupies another point in the design space. It consists of over 60,000 models, described more specifically as $N \approx 65{,}000$ identity-specific weight vectors in a roughly 100,000-dimensional space, produced by DreamBooth fine-tuning with rank-1 LoRA on a latent diffusion base model (Dravid et al., 2024). Here the dataset is neither a generic zoo nor an INR set; it is a population of customized diffusion adapters whose structure supports sampling, editing, and inversion in a PCA-defined subspace.

4. Representation, annotation, and formal structure

Neural weight space datasets require explicit choices about how weights are serialized, normalized, and labeled. In NWS, vectorization proceeds layer by layer. For convolutional layers,

$$\theta_i = \theta_{i-1} \shortparallel \mathrm{vec}(\mathbf{h}_{i,1}) \shortparallel \cdots \shortparallel \mathrm{vec}(\mathbf{h}_{i,K}) \shortparallel \mathbf{b}_i,$$

and for fully connected layers,

$f(\mathbf{x}, \theta_i)$0

with the final model weight vector $f(\mathbf{x}, \theta_i)$1 (Eilertsen et al., 2020). HyperDiffusion follows a similar flattening strategy for per-sample MLPs, but also reports a dataset-construction detail intended to smooth the induced weight-space distribution: every sample-specific MLP is optimized from the weights of a single reference MLP rather than from random initialization (Erkoç et al., 2023).

Some datasets preserve internal tensor structure instead of relying only on flat vectors. WeightFlow constructs neural network weights into a graph in which each neuron in linear layers becomes a node and node features are concatenations of incoming weights and biases (Li et al., 1 Aug 2025). This is not a conventional benchmark release, but it defines a representation strategy for datasets of time-indexed network weights and anchor models.

Annotations are equally important. NWS labels each weight snapshot with training setup variables such as optimizer, dataset, activation, initialization, and architecture choice, enabling mappings of the form $f(\mathbf{x}, \theta_i)$2 for meta-classification (Eilertsen et al., 2020). The phase-transition zoos attach train/validation/test performance, Hessian-derived sharpness, mode connectivity, and CKA-based representation similarity, then assign categorical phases using those metrics (Schürholt et al., 25 Apr 2025). Text2Weight stores dataset source, subtask class list, split membership, training configuration, and fused CLIP text embeddings alongside the weights (Tian et al., 19 Aug 2025).

At the theoretical level, recent work argues that the semantics encoded by such datasets need not be accidental. In a HyperINR setting, the Implicit Function Theorem is used to justify a local unique and differentiable mapping from a data sample $f(\mathbf{x}, \theta_i)$3 to a latent code $f(\mathbf{x}, \theta_i)$4, and from there to generated INR weights $f(\mathbf{x}, \theta_i)$5, under a full-rank Hessian condition (Qiu et al., 30 Jan 2026). This suggests a principled link between data manifolds and weight-space datasets in settings where each instance is mapped into weights through a shared hypernetwork.

5. What neural weight space datasets reveal

These datasets support several kinds of empirical claims about weight space.

First, they reveal that weights encode training provenance. NWS shows that meta-classifiers can infer many hyperparameters from trained weights alone, and that even small local subsets of consecutive weights can be informative (Eilertsen et al., 2020). MLDS similarly finds clustering of models with identical training data and meaningful divergence with even small changes to the training data, including “backdoor” variants (Clemens, 2021).

Second, they reveal trajectory structure. In the model-zoo study on training monitoring, DNN models evolve on unique, smooth trajectories in weight space, trajectories can be separated by single-linkage clustering, and epoch order can be partially recovered from distances along the path (Schürholt et al., 2020). This suggests that checkpoint-based weight datasets can function as records of optimization dynamics rather than merely archives of endpoints.

Third, they reveal that dataset design strongly affects generalization in weight-space models. The augmentation studies on INRs argue that overfitting in deep weight space models largely stems from the lack of diversity in DWS datasets: a given object admits many neural views, but typical datasets store only one (Shamsian et al., 2024). Adding multiple neural views per object mitigates overfitting and improves both “internal generalization” to new views of training objects and “external generalization” to novel objects; alignment-based MixUp in weight space can approximate the effect of substantially more data (Shamsian et al., 2023).

Fourth, they reveal semantic regularities in specialized collections. In weights2weights, principal component coefficients of LoRA adapters are approximately independent and Gaussian-distributed, and linear directions in the resulting subspace correspond to semantic edits such as adding a beard (Dravid et al., 2024). In Text2Weight, a paired text-weight dataset enables learning $f(\mathbf{x}, \theta_i)$6, linking natural language task descriptions directly to classifier-head parameters (Tian et al., 19 Aug 2025).

6. Applications, limitations, and current directions

The applications of neural weight space datasets now span analysis, representation learning, and generation. ProbeGen uses collections of model weights to predict properties such as dataset origin or generalization error through learned probes passed through the target models, and reports between 30 and 1000 times fewer FLOPs than other top approaches (Kahana et al., 2024). HyperDiffusion trains diffusion directly in the space of flattened INR weights and reports evaluation metrics including MMD, COV, 1-NNA, and FPD on ShapeNet and DeformingThings4D (Erkoç et al., 2023). DeepWeightFlow uses datasets of fully trained, canonicalized models to generate complete neural network weights through Flow Matching, with Git Re-Basin or TransFusion used to address permutation symmetries for MLPs, ResNets, ViTs, and BERT (Gupta et al., 8 Jan 2026).

Several limitations recur across the literature. Dataset creation overhead is central in INR-based collections because each sample requires a full optimization run; HyperDiffusion quantifies this as 6 min/shape × data size (Erkoç et al., 2023). Lack of diversity is another major issue in deep weight space datasets derived from INRs, where one neural view per object is often insufficient (Shamsian et al., 2024). Permutation symmetry and canonicalization remain structural complications for generative modeling and comparison, motivating alignment procedures such as Git Re-Basin, weight matching for MixUp, and TransFusion (Shamsian et al., 2023). Heterogeneity is a distinct challenge once the field moves beyond curated zoos: training on public repositories such as Hugging Face requires tokenization and normalization schemes that can handle missing metadata, architecture variation, and sparse documentation (Falk et al., 2 Oct 2025).

Current directions therefore move along two axes. One is toward more structured curation, exemplified by phase-aware model zoos with explicit coverage criteria and loss-landscape annotations (Schürholt et al., 25 Apr 2025). The other is toward less curated but larger-scale aggregation, exemplified by weight-space backbones trained on arbitrary models downloaded from public model hubs, where curated zoos are no longer treated as indispensable (Falk et al., 2 Oct 2025). Taken together, these developments indicate that a neural weight space dataset is increasingly understood not merely as a static archive of checkpoints, but as a research object whose construction protocol, symmetry handling, annotation scheme, and diversity profile determine what can be learned from neural weights as a modality (Han et al., 10 Mar 2026).