Weight Space Representation (WSR)

• Weight Space Representation (WSR) is a mapping from high-dimensional neural network weight vectors to low-dimensional embeddings that preserve structural and functional characteristics.
• Various encoder architectures—including linear projections, autoencoders, symmetry-aware methods, and sequence models like SANE—offer scalable and interpretable solutions.
• WSR supports practical applications such as performance prediction, model retrieval, editing, and generative synthesis while raising challenges in symmetry handling and metric coherence.

Weight Space Representation (WSR) most commonly denotes a mapping from neural-network parameter space to a compact embedding that preserves structural or functional information about the model. In the taxonomy of weight space learning, WSR is one of three core dimensions—alongside Weight Space Understanding and Weight Space Generation—and is formalized as $\phi:\Theta\to\mathbb R^d$ with $d\ll n$ for $\Theta\subset\mathbb R^n$ the space of flattened weight vectors (Han et al., 10 Mar 2026). Across recent work, WSR includes linear projections, autoencoding, symmetry-aware and graph-based encoders, probe-based behavioral signatures, and direct operations on pretrained or fine-tuned models for retrieval, performance prediction, editing, and synthesis. In a distinct mathematical usage, the same term also refers to a positive representation of a complex weight on $\mathbb R^d$ by a normalized density on complexified space that matches moments and analytic observables (Ruba et al., 2017).

1. Definition and taxonomic position

Formally, a WSR is a function $\phi:\Theta\to\mathbb R^d$ that maps a high-dimensional weight vector $\theta$ to a compact embedding $z=\phi(\theta)$, typically with an accompanying downstream predictor $g:\mathbb R^d\to\mathcal Y$ so that $\hat y=g(z)\approx y(\theta)$ for properties such as test accuracy or robustness (Han et al., 10 Mar 2026). This formulation treats trained models as data objects in their own right rather than merely as endpoints of optimization. The standard experimental setting is therefore a model zoo: thousands of networks varying architecture depth, optimizer, seed, or data split, divided into train and held-out subsets for representation learning and evaluation (Han et al., 10 Mar 2026).

The survey literature places WSR between descriptive analysis and generative modeling. Weight Space Understanding studies geometry and symmetries of weights; Weight Space Generation synthesizes new parameters through hypernetworks or generative models; WSR occupies the intermediate role of learning embeddings over model weights that can support regression, retrieval, clustering, transfer, or downstream control (Han et al., 10 Mar 2026). This suggests that WSR is best understood not as a single architecture class, but as a representational layer spanning multiple methodological families.

2. Geometry, symmetries, and canonical formulations

A central issue in WSR is that raw weights live in a high-dimensional space with substantial symmetry. The neural-field formulation states this explicitly: raw $\theta_i$ occupy a huge, highly symmetric space with permutations of hidden neurons and arbitrary scaling, which can induce chaotic, non-semantic geometry (Yang et al., 1 Dec 2025). The survey accordingly distinguishes symmetry-agnostic encoders, which flatten $d\ll n$0 and process it with an MLP, CNN, or Transformer, from symmetry-aware encoders such as DWSNets, NFN, and UNF, which enforce invariance or equivariance to hidden-unit permutations through DeepSets-style pooling, layerwise splitting, or universal linear equivariant layers (Han et al., 10 Mar 2026).

The same survey also identifies classical linear and manifold baselines, including PCA/SVD, multidimensional scaling, Isomap, and t-SNE. In these methods the embedding may be as simple as $d\ll n$1 for top principal directions $d\ll n$2, or a lower-dimensional arrangement preserving pairwise or geodesic distances (Han et al., 10 Mar 2026). Autoencoder and VAE formulations extend this by learning encoder–decoder pairs that minimize reconstruction error, optionally with a KL regularizer, while graph-based encoders such as NG and GMN represent the network itself as a directed acyclic graph with node and edge features and apply message passing before a readout step (Han et al., 10 Mar 2026).

A common misconception is that WSR is necessarily a raw-weight embedding. The survey explicitly includes model-free encoders such as ProbeLog, ProbeGen, and interactive probes, where the representation is derived from responses of $d\ll n$3 to fixed or learned probe inputs rather than from $d\ll n$4 directly (Han et al., 10 Mar 2026). Conversely, not every raw-weight embedding is assumed to be meaningful without preprocessing. SANE standardizes each layer’s weights and aligns all models to a common reference via permutation matching before tokenization, while the neural-field work constrains the optimization space through a pre-trained base model and low-rank adaptation in order to induce structure in weight space (Schürholt et al., 2024, Yang et al., 1 Dec 2025).

3. Encoder architectures and scalable sequential autoencoding

A major recent development is the move from monolithic flattened vectors to sequence models over weight tokens. SANE, the “Sequential Autoencoder for Neural Embeddings,” tokenizes each layer’s weight tensor into fixed-size vectors, processes overlapping windows of these tokens with a shared transformer encoder $d\ll n$5 and decoder $d\ll n$6, and aggregates per-token embeddings by mean pooling into a global representation (Schürholt et al., 2024). Each token carries a discrete 3D position $d\ll n$7 consisting of a global index, layer index, and intra-layer token index, and training uses randomly sampled contiguous windows of length $d\ll n$8 together with a binary mask for padded positions (Schürholt et al., 2024).

Its objective combines reconstruction and contrastive guidance: $d\ll n$9, where $\Theta\subset\mathbb R^n$0 is a masked $\Theta\subset\mathbb R^n$1 reconstruction loss and $\Theta\subset\mathbb R^n$2 is an NTXent loss computed on augmented views obtained by random row-permutation or weight noise (Schürholt et al., 2024). The architecture uses stacked transformer encoder blocks, with 4 blocks for small CNNs and 8 blocks for ResNets, multi-head self-attention, and hidden sizes $\Theta\subset\mathbb R^n$3 for CNNs or $\Theta\subset\mathbb R^n$4 for ResNets (Schürholt et al., 2024). Because training operates on small windows, the memory and computation cost is decoupled from the total parameter count, allowing the method to scale to larger models such as ResNet-101 and to variable architectures (Schürholt et al., 2024).

Empirically, SANE attains $\Theta\subset\mathbb R^n$5 on small CNNs versus $\Theta\subset\mathbb R^n$6, and $\Theta\subset\mathbb R^n$7 on ResNet-18 versus $\Theta\subset\mathbb R^n$8, while supporting tasks that earlier hyper-representations could not achieve, including sequential generation of unseen neural network models (Schürholt et al., 2024). For generative sampling, SANE_SUB reaches epoch-0 zero-shot accuracy of approximately $\Theta\subset\mathbb R^n$9 on MNIST versus prior SOTA of approximately $\mathbb R^d$0, and on CIFAR-10 with ResNet-18 it reaches approximately $\mathbb R^d$1 at epoch 0 versus approximately $\mathbb R^d$2 for random initialization; after 10 epochs of fine-tuning it reaches approximately $\mathbb R^d$3 versus approximately $\mathbb R^d$4 from scratch (Schürholt et al., 2024). These results locate WSR not only in analysis, but also in scalable model initialization and transfer.

4. Empirical probing of training footprints

An earlier empirical line of work treats each trained network as a point in Euclidean weight space and asks what information can be inferred from $\mathbb R^d$5 alone. “Classifying the classifier” constructs the Neural Weight Space dataset from 16,000 distinct CNN trainings with 20 snapshots per trajectory, for a total of 320,000 weight vectors, and trains shallow and deep meta-classifiers to recover properties of the training setup from weight snapshots (Eilertsen et al., 2020). Weight vectors are built by concatenating layer parameters, and both Euclidean distance and cosine similarity are used as basic measures, with PCA and UMAP for low-dimensional visualization (Eilertsen et al., 2020).

The quantitative findings show that even simple layer-wise statistics can encode strong hyper-parameter signatures. On a fixed-architecture subset, linear SVMs reach approximately $\mathbb R^d$6 accuracy for dataset classification, approximately $\mathbb R^d$7 for optimizer, approximately $\mathbb R^d$8 for activation, and approximately $\mathbb R^d$9 for initialization; deep global meta-classifiers move these results closer to $\phi:\Theta\to\mathbb R^d$0 for most hyper-parameters (Eilertsen et al., 2020). Local deep meta-classifiers operating on only 5,000 consecutive weights still perform far above chance on 11 hyper-parameters, with initialization at approximately $\phi:\Theta\to\mathbb R^d$1, dataset at approximately $\phi:\Theta\to\mathbb R^d$2–$\phi:\Theta\to\mathbb R^d$3 when the slice falls in early convolutional layers, and optimizer and activation at approximately $\phi:\Theta\to\mathbb R^d$4–$\phi:\Theta\to\mathbb R^d$5 (Eilertsen et al., 2020).

These experiments establish WSR as an interpretive tool. Footprint maps over depth and training progress show that dataset classification peaks in the first 30% of the parameter vector, optimizer information is nearly uniform with a local peak around the first convolutional layer, and initialization signals decay rapidly in early convolutional and late fully connected layers (Eilertsen et al., 2020). The paper frames this as a complementary perspective for explainable AI and raises explicit questions about privacy, compression, and symmetry handling, noting that current WSR ignores weight-space symmetries and that better alignment or graph-based formulations may sharpen meta-classification (Eilertsen et al., 2020).

5. Generative synthesis, editing, inversion, and weight-space dynamics

A second major branch of WSR uses learned low-dimensional spaces as manipulable latents for model synthesis and modification. The weights2weights framework constructs a dataset $\phi:\Theta\to\mathbb R^d$6 from approximately 65,000 Dreambooth+LoRA fine-tunings of a latent diffusion model, where each $\phi:\Theta\to\mathbb R^d$7 is the flattened LoRA update for a different visual identity and $\phi:\Theta\to\mathbb R^d$8 (Dravid et al., 2024). PCA yields a subspace $\phi:\Theta\to\mathbb R^d$9, and sampling is performed by drawing approximately Gaussian coefficients in this basis and reconstructing a new weight vector. Linear classifiers in the same basis produce semantic editing directions, and constrained optimization over $\theta$0 with $\theta$1 enables single-image inversion into the weight-space manifold (Dravid et al., 2024).

The reported applications are sampling, editing, and inversion. Models sampled from $\theta$2 produce novel, realistic faces; edits along learned linear directions achieve the best ID-score among Prompting, ConceptSliders, and weights2weights, with lower LPIPS and comparable or higher CLIPScore; and one-image inversion reaches ID-score $\theta$3 versus multi-image DB-LoRA at $\theta$4 and single-image DB-LoRA at $\theta$5, while being faster at approximately 55 s versus approximately 220 s (Dravid et al., 2024). The same pipeline is also applied to 120 dog breeds, indicating that the linear structure is not restricted to human identities (Dravid et al., 2024).

Weight-space methods also extend beyond static embeddings. WARP reparametrizes recurrent hidden state as the flattened parameter vector $\theta$6 of a root MLP and evolves it via the linear recurrence $\theta$7 (Nzoyem et al., 1 Jun 2025). The initial state is produced by a hypernetwork $\theta$8, and decoding can be performed by the root network itself or by analytic formulas encoding physical priors such as sine curves or mass-spring-damper dynamics (Nzoyem et al., 1 Jun 2025). The reported consequences are high-resolution memory, gradient-free test-time adaptation, and bounded, interpretable weight trajectories; across synthetic dynamical systems, ETT power-forecasting, and UEA classification, WARP matched or outperformed RNNs, SSMs, and Transformers (Nzoyem et al., 1 Jun 2025). This suggests that WSR can function not only as an embedding of trained models, but also as a state parameterization for sequential inference.

The survey places these examples within a larger application space that includes model retrieval, zero-shot performance prediction, model editing, federated learning through personalized hypernetworks, neural architecture search through graph hypernetworks, INR synthesis, and meta-learning with diffusion in weight space (Han et al., 10 Mar 2026).

6. Three-dimensional shape generation and neural-field representations

Weight-space methods have become especially prominent in 3D and neural-field settings, where each instance can be stored as the parameters of a compact coordinate-based network. HyperDiffusion first overfits a small implicit-field MLP to each sample and then trains a DDPM directly over the flattened parameter vector $\theta$9 with $z=\phi(\theta)$0k, partitioned into $z=\phi(\theta)$1 tokens for a transformer denoiser (Erkoç et al., 2023). The model uses a standard Gaussian forward process, a transformer with 12 layers, 16 heads, and hidden size 2880, and an $z=\phi(\theta)$2 denoising objective that predicts the original clean weights $z=\phi(\theta)$3 (Erkoç et al., 2023). Its stated benefits are compactness, continuous arbitrary-resolution surfaces, a unified 3D/4D pipeline, and the absence of any separate encoder–decoder latent model, while its limitations include high dimensionality, imperfect metric structure in Euclidean weight space, and nontrivial scaling to scene-scale collections of MLPs (Erkoç et al., 2023).

The neural-field WSR work of Zhuoqian Yang et al. imposes stronger structure by freezing a pre-trained base network $z=\phi(\theta)$4 and fitting only low-rank updates $z=\phi(\theta)$5 for each data instance (Yang et al., 1 Dec 2025). Standard additive LoRA uses $z=\phi(\theta)$6, whereas multiplicative LoRA introduces $z=\phi(\theta)$7 and can be written as $z=\phi(\theta)$8, which rescales input and output channels without mixing them (Yang et al., 1 Dec 2025). An asymmetric mask fixes $z=\phi(\theta)$9 fused positions in each row to zero so as to break permutation symmetry among rank components (Yang et al., 1 Dec 2025). On FFHQ at $g:\mathbb R^d\to\mathcal Y$0, mLoRA-Asym achieves PSNR 36.91 versus 35.69 for additive LoRA and 35.11 for a standalone MLP; on ShapeNet it achieves Chamfer 2.41/3.35 versus 2.44/3.39 for LoRA and 2.57/3.78 for MLP; and in generative evaluation it reaches FD $g:\mathbb R^d\to\mathcal Y$1 on FFHQ versus $g:\mathbb R^d\to\mathcal Y$2 for LoRA-Asym, $g:\mathbb R^d\to\mathcal Y$3 for MLP-Asym, and $g:\mathbb R^d\to\mathcal Y$4 for HyperDiffusion (Yang et al., 1 Dec 2025). The same work reports ARI $g:\mathbb R^d\to\mathcal Y$5, 1-NN $g:\mathbb R^d\to\mathcal Y$6, and logistic classification $g:\mathbb R^d\to\mathcal Y$7 on ShapeNet-10, exceeding both additive LoRA and unconstrained MLP baselines (Yang et al., 1 Dec 2025).

A related but distinct formulation appears in 3D foundation models. “Shape Generation via Weight Space Learning” treats the entire parameter vector $g:\mathbb R^d\to\mathcal Y$8 of a large 3D shape generator, with $g:\mathbb R^d\to\mathcal Y$9, as a data modality and uses a smooth conditioning map $\hat y=g(z)\approx y(\theta)$0 to move along submanifolds of the full weight space (Plattner et al., 26 Mar 2025). A topology experiment encodes a spherical-linear-interpolation path between two point clouds into a conditioning curve $\hat y=g(z)\approx y(\theta)$1 and thus a weight curve $\hat y=g(z)\approx y(\theta)$2, decodes meshes $\hat y=g(z)\approx y(\theta)$3, and measures the number of connected components $\hat y=g(z)\approx y(\theta)$4 (Plattner et al., 26 Mar 2025). Empirically, $\hat y=g(z)\approx y(\theta)$5 remains 1 for $\hat y=g(z)\approx y(\theta)$6 and then jumps to a large integer for $\hat y=g(z)\approx y(\theta)$7, indicating a sharp phase transition in global connectivity over a narrow interval (Plattner et al., 26 Mar 2025). In a separate local-priors experiment, PCA over $\hat y=g(z)\approx y(\theta)$8 unseen bracket shapes from SimJEB retains the top $\hat y=g(z)\approx y(\theta)$9 eigenvectors to define a 100-dimensional conditioning subspace whose lift through $\theta_i$0 preserves global connectivity while varying local features such as hole sizes, fillets, and overhangs (Plattner et al., 26 Mar 2025). No additional losses such as $\theta_i$1 or $\theta_i$2 are introduced; continuity of $\theta_i$3, PCA, mesh-connectivity counts, and visual inspection are used as observational criteria (Plattner et al., 26 Mar 2025). The paper presents this as early-stage evidence that learnable submanifolds in weight space can separately modulate topology and fine-grained geometry.

7. Separate mathematical usage, evaluation, and unresolved issues

Outside neural weight-space learning, WSR also denotes a constructive representation theorem for complex weights on $\theta_i$4. In the formulation of Ruba and Wyrzykowski, one starts with a complex-valued weight $\theta_i$5 or $\theta_i$6 and seeks a nonnegative, normalized density $\theta_i$7 on the complexified space $\theta_i$8 such that expectations of analytic observables agree, which can be enforced by matching all polynomial moments (Ruba et al., 2017). In one dimension the construction expands $\theta_i$9 in angular Fourier modes, notes that the system is highly underdetermined, introduces a Gaussian radial ansatz, and derives a closed-form representation

$d\ll n$00

for sufficiently small $d\ll n$01 under a decay condition on $d\ll n$02 (Ruba et al., 2017). The paper gives explicit one-dimensional constructions for complex Gaussian and quartic phase weights and emphasizes that infinitely many positive representations exist because the moment equations underdetermine the radial functions (Ruba et al., 2017). This is a separate mathematical lineage from neural-network WSR, linked mainly by the shared idea that a difficult weight object can be replaced by a more tractable representation.

Within neural WSR proper, evaluation protocols are now relatively standardized. The survey lists retrieval accuracy, clustering quality, regression performance, transfer learning, and embedding smoothness under path interpolation as the main criteria, typically measured on model zoos varying architecture, optimizer, seed, or data split (Han et al., 10 Mar 2026). Yet the recent literature also records clear unresolved issues. SANE notes that pretraining currently assumes a homogeneous zoo for alignment simplicity, that large-model zero-shot sampling from a Gaussian prior is still noisy, that experiments focus on vision tasks, and that sensitivity to window size, $d\ll n$03, and transformer depth requires systematic exploration (Schürholt et al., 2024). Earlier probing work notes that better treatment of permutation symmetry and invariance remains necessary (Eilertsen et al., 2020). HyperDiffusion identifies the imperfect correspondence between Euclidean weight distance and perceptual similarity as a practical limitation (Erkoç et al., 2023).

Taken together, these results indicate that WSR has developed from simple vectorization and projection into a broad technical program in which neural weights are embedded, aligned, probed, constrained, generated, and dynamically evolved. The unifying premise is that model parameters are not only optimization outputs but also structured objects amenable to statistical, geometric, and generative analysis (Han et al., 10 Mar 2026).