Super Weights in Imaging, NAS, LLMs & Geometry

• Super Weights are technical constructs with domain-specific definitions, appearing as ensemble coefficients in imaging, parameter-sharing in NAS, critical pruning points in LLMs, and Cartan weight matrices in supergeometry.
• In image super-resolution, Super Weights are MAP-estimated ensemble coefficients that integrate component outputs using a reference prior, reducing artifacts through optimal weight allocation.
• In large language models, Super Weights are individual scalar parameters whose pruning severely degrades performance, emphasizing the disconnect between parameter importance and isolated trainability.

Searching arXiv for papers explicitly using the term “Super Weights” and related variants. Super Weights is a field-dependent technical term rather than a single standardized object. In image super-resolution, it denotes ensemble coefficients that linearly combine component super-resolvers under a MAP formulation with a reference-dataset prior (Jiang et al., 2019). In one-shot neural architecture search, the related expression super-network weights denotes the shared or conditionally specialized parameters of an over-complete network used to evaluate many candidate architectures by activating a path through the super-network (Laube et al., 2021). In recent LLM work, Super Weights denotes individual scalar parameters whose pruning can increase perplexity by orders of magnitude and reduce zero-shot accuracy to guessing, while also serving as anchors for super activations and data-free quantization schemes (Yu et al., 2024, Subramanian et al., 9 Jul 2026). In super higher-Teichmüller geometry, the super-weight matrix $W$ records Cartan weights of an abelian odd slice and controls mutation, horizontality, and a flat logarithmic superconnection (Song, 26 Oct 2025). Related but distinct weight terminology also appears in affine Lie superalgebras and in super-Breuil weights in $p$-adic representation theory (Calixto et al., 2018, Chitrao et al., 18 Apr 2026).

1. Scope and disambiguation

The expression appears in several technically unrelated literatures.

Usage	Mathematical object	Domain
Super Weights	$w=[w_1,\dots,w_K]^T$	Ensemble super-resolution
super-network weights	$w_l^{(c_l)}$ or $w_l^{(c_{l-1},c_l)}$	One-shot NAS
Super Weights	Individual scalar parameters $w_{ij}$	LLMs
super-weight matrix	$W=(W_{\alpha i})$	Super higher-Teichmüller geometry
super-Breuil weights	Weight ranges $k$	$p$-adic representation theory

These usages are not interchangeable. In RefESR, the object is an optimization variable over mixture coefficients; in NAS it is a parameter-sharing device inside a super-network; in LLM analysis it is a pruning-critical coordinate in a pretrained model; and in supergeometry it is an integer matrix of Cartan weights. This suggests that the term has local semantics determined by the surrounding formalism rather than a cross-domain invariant definition.

2. Super Weights as MAP-estimated ensemble coefficients in super-resolution

In RefESR, the so-called Super Weights are simply the ensemble coefficients $w=[w_1,\dots,w_K]^T$ that linearly combine the $p$0 component super-resolvers $p$1 (Jiang et al., 2019). With observed low-resolution image $p$2, component outputs $p$3, blur-plus-downsampling operator $p$4, and ensemble output

$p$5

the degradation model is

$p$6

The likelihood is therefore

$p$7

The prior over $p$8 is learned from a separate reference dataset of HR/LR pairs. Each component super-resolver is scored by its average PSNR/SSIM performance on the reference set, and these scores are converted into a reference weight vector $p$9. With spherical covariance $w=[w_1,\dots,w_K]^T$0, the prior is

$w=[w_1,\dots,w_K]^T$1

The MAP estimator becomes

$w=[w_1,\dots,w_K]^T$2

where $w=[w_1,\dots,w_K]^T$3 and $w=[w_1,\dots,w_K]^T$4.

A notable feature of RefESR is that the constrained problem has an analytical solution. Using augmented data

$w=[w_1,\dots,w_K]^T$5

the problem is rewritten as a constrained least-squares system. Defining

$w=[w_1,\dots,w_K]^T$6

the constrained minimizer is

$w=[w_1,\dots,w_K]^T$7

which automatically enforces $w=[w_1,\dots,w_K]^T$8.

The reference weight prior is constructed from

$w=[w_1,\dots,w_K]^T$9

followed by the softmax-like rule

$w_l^{(c_l)}$0

where $w_l^{(c_l)}$1 is a bandwidth parameter. The limiting interpretations are explicit: $w_l^{(c_l)}$2 concentrates on the single best component, while $w_l^{(c_l)}$3 yields $w_l^{(c_l)}$4. Likewise, $w_l^{(c_l)}$5 ignores the reference prior and $w_l^{(c_l)}$6 forces $w_l^{(c_l)}$7; in practice $w_l^{(c_l)}$8 is chosen via grid search, typically $w_l^{(c_l)}$9.

Empirically, the final Super Weights typically place high mass on solvers that both perform well on the reference set and reduce the current LR reconstruction error, while poorly performing solvers receive near zero weight. The ensemble is reported to reduce artifacts such as ringing by averaging complementary strengths, and even very strong networks such as EDSR can be mildly improved by ensembling with diverse methods.

3. Super-network weights and conditional specialization in one-shot NAS

In one-shot NAS, super-network weights refer to the parameters of an over-complete network that contains every candidate operation in the search space (Laube et al., 2021). If the search space consists of $w_l^{(c_{l-1},c_l)}$0 layers and one must choose exactly one operation $w_l^{(c_{l-1},c_l)}$1 at each layer, the standard super-network stores a tensor $w_l^{(c_{l-1},c_l)}$2 for every layer-operation pair. For a sampled path $w_l^{(c_{l-1},c_l)}$3, the forward pass is

$w_l^{(c_{l-1},c_l)}$4

Because one can evaluate an architecture $w_l^{(c_{l-1},c_l)}$5 by activating the corresponding path and performing one forward pass, no retraining is needed, and the resulting estimate is a rapid proxy of the true performance.

The paper extends this design with conditional weights to model dependencies between consecutive operations. Instead of a context-agnostic tensor $w_l^{(c_{l-1},c_l)}$6, it introduces

$w_l^{(c_{l-1},c_l)}$7

one weight tensor for every ordered pair $w_l^{(c_{l-1},c_l)}$8. The forward pass becomes

$w_l^{(c_{l-1},c_l)}$9

The motivation is that the best parameters for an operation at layer $w_{ij}$0 may depend on the candidate selected at layer $w_{ij}$1.

Directly allocating all pair-specific tensors from epoch 1 multiplies the total weight count by $w_{ij}$2, so the authors propose a split schedule. First, the super-network is trained normally for $w_{ij}$3 epochs using shared $w_{ij}$4. Then, at epoch $w_{ij}$5, each shared tensor is cloned into the family $w_{ij}$6 and training continues with specialized weights. Because each copy is initialized from the shared weights, the specialized tensors start in a good basin and only need to fine-tune toward their respective contexts.

The empirical findings are benchmark-specific. On NAS-Bench-201, which contains 15,625 candidates, splitting at the right epoch $w_{ij}$7 raises the top-1 selected network’s true accuracy by $w_{ij}$8 to $w_{ij}$9, moving the proxy-normalized improvement from approximately $W=(W_{\alpha i})$0 up to approximately $W=(W_{\alpha i})$1. In the “No Zero” variant, the top-1 average accuracy goes from approximately $W=(W_{\alpha i})$2 to approximately $W=(W_{\alpha i})$3. On NAS-Bench-Macro, with 6,561 candidates, splitting late at $W=(W_{\alpha i})$4 improves top-1 picks by more than one standard deviation, and top-5 and top-10 also benefit. Despite an increase of up to $W=(W_{\alpha i})$5 in the number of parameter tensors in multi-path cells, GPU-memory usage rises by only approximately $W=(W_{\alpha i})$6, and training-time expansion is similarly negligible on modern accelerators.

4. Super Weights in LLMs: definition, detection, and pruning impact

In recent LLM work, a Super Weight is an individual scalar parameter whose removal catastrophically raises perplexity (Yu et al., 2024, Subramanian et al., 9 Jul 2026). One formal definition uses pruning impact:

$W=(W_{\alpha i})$7

and Super Weights are the coordinates with the largest $W=(W_{\alpha i})$8. A practical proxy scans weight matrices such as q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj for the largest $W=(W_{\alpha i})$9, caches the top $k$0 positions, and validates them through activation spikes. Empirically, more than $k$1 of the top-10 and more than $k$2 of the top-100 by $k$3 lie in down_proj layers. A second validation records which down_proj coordinates produce the largest activations; nine positions are $k$4 consistent across $k$5 samples.

A complementary identification method is data-free and uses a single forward pass through the model. For each layer $k$6, one records the maximum-magnitude input and output of the mlp.down_proj module and searches for a single layer where both maxima are far above all other layers. If $k$7 indexes the input spike and $k$8 the output spike, then $k$9 is declared a super weight. The induced large activation at the output is termed a super activation.

The pruning effect is large in some models and not universal across all models. Using activation-spike coordinates or proprietary coordinates, zeroing 1–6 such Super Weights yields the following examples: OLMo-1B perplexity goes from $p$0 to $p$1 and ARC-Easy accuracy from $p$2 to $p$3; OLMo-7B perplexity goes from $p$4 to $p$5 and ARC-Easy accuracy from $p$6 to $p$7; Phi-3-mini perplexity goes from $p$8 to $p$9 and ARC-Easy accuracy from $w=[w_1,\dots,w_K]^T$0 to $w=[w_1,\dots,w_K]^T$1; Mistral-7B and Meta-Llama-3-8B both show perplexity going to $w=[w_1,\dots,w_K]^T$2 and ARC-Easy accuracy dropping to approximately random-guessing levels. By contrast, magnitude-based “top-2” pruning on Llama-3.1-8B, Llama-3.2-3B, Llama-3.2-1B, Qwen2.5, and Gemma-2 causes no measurable drop, showing that large $w=[w_1,\dots,w_K]^T$3 alone is not sufficient.

The Llama-7B example in the quantization paper illustrates the same asymmetry. Original zero-shot average accuracy is approximately $w=[w_1,\dots,w_K]^T$4, with C4 perplexity approximately $w=[w_1,\dots,w_K]^T$5 and Wiki-2 perplexity approximately $w=[w_1,\dots,w_K]^T$6. Pruning the single super weight drops zero-shot average accuracy to approximately $w=[w_1,\dots,w_K]^T$7, raises C4 perplexity to approximately $w=[w_1,\dots,w_K]^T$8, and raises Wiki-2 perplexity to approximately $w=[w_1,\dots,w_K]^T$9. Pruning the next 7,000 largest weights while keeping the super weight leaves zero-shot average accuracy at approximately $p$00, with C4 perplexity approximately $p$01 and Wiki-2 perplexity approximately $p$02.

The same papers connect Super Weights to quantization. For activation quantization, one replaces the single super activation with a median value, quantizes and dequantizes the rest by round-to-nearest, and restores the original super activation in FP16. For weight quantization, one identifies Super Weights, clips all weights at a $p$03-score threshold, quantizes and dequantizes the clipped tensor, and restores the Super Weight in FP16. Across Llama models, preserving the super activation retains $p$04–$p$05 of the quality gain that SmoothQuant achieves without calibration data. On Llama-7B C4 perplexity, the reported values are FP16 $p$06, naive W8A8 $p$07, SmoothQuant $p$08, and the super-activation-preserving method $p$09.

5. The failure of selective training and the distinction between importance and trainability

A central finding in the LLM literature is that parameter importance does not imply parameter trainability in isolation (Subramanian et al., 9 Jul 2026). The paper tests Super Weight-aware training on OLMo-1B and OLMo-7B by freezing all parameters except the top-$p$10 Super Weights by magnitude, with $p$11. The setup uses AdamW, learning rate $p$12, three epochs, and evaluates on ARC-Easy. In all such cases, accuracy collapses to approximately $p$13–$p$14 on both scales, i.e. random guessing, and increasing $p$15 by a factor of $p$16 yields no improvement. The training curves show falling training loss but exploding validation perplexity, indicating memorization without generalization.

Expanding the trainable set to local neighborhoods does not rescue performance. A radius-1 neighborhood around each Super Weight yields a $p$17 patch, so for $p$18 one obtains approximately $p$19, $p$20, and $p$21 parameters, yet ARC-Easy accuracy remains approximately $p$22–$p$23. The paper attributes this to the fact that computations involving a Super Weight span entire rows and columns and propagate through residual blocks, so local patches lack the required global coordination.

The collapse is specific to Super Weight coordinates rather than to sparsity itself. Freezing all but 4,096 parameters chosen uniformly at random from down_proj, excluding Super Weight coordinates, yields $p$24 accuracy on OLMo-1B, compared with a baseline of $p$25 and LoRA at $p$26. Vanilla LoRA on attention projections only, with $p$27, $p$28, and $p$29, succeeds using 2.1M parameters for OLMo-1B, which is $p$30 of 1.28B, and reaches ARC-Easy accuracies of $p$31 for OLMo-1B and $p$32 for OLMo-7B. Applying the same low-rank update to down_proj also succeeds.

The paper further shows that constraining LoRA updates at positions corresponding to Super Weight coordinates produces statistically indistinguishable results. In LoRA-dproj-SW-freeze, scaling entries matching down_proj Super Weight coordinates by $p$33 yields $p$34–$p$35 on OLMo-1B and $p$36–$p$37 on OLMo-7B. A 10-seed ablation comparing $p$38 and $p$39 reports $p$40 for both variants on OLMo-1B and $p$41 for both on OLMo-7B, with $p$42 in the 1B case and $p$43 in the 7B case. The stated conclusion is that effective fine-tuning relies on structured decompositions over entire layers rather than targeting individually important weights.

6. Super-weight matrices, highest weights, and super-Breuil weights in mathematics

In super higher-Teichmüller geometry, the super-weight matrix

$p$44

encodes the Cartan weights of an abelian odd slice (Song, 26 Oct 2025). If $p$45 are mutually commuting odd root vectors and $p$46 are the cocharacters whose exponentials give the even cluster $p$47-variables, then

$p$48

Equivalently, $p$49 gives the log-canonical Poisson bracket

$p$50

Under mutation at $p$51, the columns transform by the column $p$52-vector rule:

$p$53

This matrix determines a horizontal odd frame

$p$54

and the logarithmic superconnection

$p$55

The curvature vanishes identically, so $p$56 is flat. The canonical super volume form is the Berezinian

$p$57

which is mutation invariant up to an overall sign cocycle that trivializes globally. On the $p$58-loop fibration, the canonical loop superform is

$p$59

and its horizontality depends on the same $p$60-determined frame.

Other mathematical uses of weight language are adjacent but distinct. In affine Lie superalgebras, highest-weight modules are defined relative to a chosen positive nilpotent subalgebra, and Verma-type modules $p$61 are induced from parabolic data; the main simplicity theorem states that if $p$62, then $p$63 is simple if and only if $p$64 is simple (Calixto et al., 2018). Here “highest weight” is the classical representation-theoretic notion, not an ensemble coefficient or a pruning-critical scalar.

In $p$65-adic representation theory, super-Breuil weights are weight ranges for two-dimensional irreducible semi-stable representations $p$66 with Hodge–Tate weights $p$67 (Chitrao et al., 18 Apr 2026). The paper studies

$p$68

equivalently

$p$69

and proves that if $p$70, then the reduction of $p$71 is the unique supersingular mod $p$72 representation of $p$73 of Serre weight $p$74. In the second range, this weakens the Bergdall–Levin–Liu bound from $p$75 to $p$76. Here again, “super-Breuil weights” refers to arithmetic weight ranges rather than to the computational constructs called Super Weights in SR, NAS, or LLMs.