WeightFormer: Dynamic Parameter Generation

• WeightFormer is a dual-concept framework: one variant dynamically predicts CNN layer weights for global visual modeling, while the other maps teacher parameters to a student model.
• It leverages input-conditioned updates and reformulated attention to achieve linear-time complexity and improved scalability in vision tasks.
• Empirical evaluations on ImageNet, COCO, ADE20K, and CIFAR demonstrate enhanced accuracy, throughput, and efficient ensemble compression compared to standard baselines.

WeightFormer is the name used for two different neural architectures in the arXiv literature. In "Linear-Time Global Visual Modeling without Explicit Attention" (He et al., 3 May 2026), WeightFormer denotes a vision backbone that replaces explicit attention with dynamic parameterization inside standard CNN layers, with the stated goal of achieving Transformer-level global modeling at linear complexity. In "Meta-Ensemble Parameter Learning" (Fei et al., 2022), WeightFormer denotes an encoder-only Transformer hypernetwork that predicts the parameters of a single student model from the parameters of multiple teacher models in one forward pass. The shared theme is weight prediction conditioned on structured context, but the two formulations operate at different levels: one predicts layer parameters from the current visual input, whereas the other predicts an entire student network from teacher checkpoints.

1. Two uses of the name

The term "WeightFormer" therefore refers to two separate research programs rather than a single canonical architecture.

Usage	Core mechanism	Primary setting
WeightFormer (He et al., 3 May 2026)	Dynamic parameterization within CNN layers from compressed global descriptors	Linear-time global visual modeling without explicit attention
WeightFormer (Fei et al., 2022)	Encoder-only Transformer that predicts student weights from teacher parameters	Meta-ensemble parameter learning

In the 2026 usage, the central object is an input-conditioned layer of the form $W(X)=W_0+\Delta W(X)$, where the current feature map determines a dynamic update to a static weight tensor. In the 2022 usage, the central object is a generator $g_\theta$ that maps a set of teacher parameters $\left[W^{(1)},\dots,W^{(M)}\right]$ to predicted student parameters $\widehat{W}_s$. This suggests that the name is best understood as describing a design philosophy centered on parameter generation, while the actual modeling targets—global visual sequence modeling versus ensemble compression—remain distinct (He et al., 3 May 2026, Fei et al., 2022).

2. Attention as a dynamically parameterized MLP

The 2026 WeightFormer is built on a mathematical reformulation of attention. For an input $X\in\mathbb{R}^{N\times d}$, canonical attention is written as

$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$

with

$Y=\mathrm{softmax}(QK^T/\sqrt{d})V.$

For the $i$-th token, this becomes

$o_i=\mathrm{softmax}(q_iK^T/\sqrt{d})V.$

The paper re-brackets this computation as a two-layer per-token MLP with softmax nonlinearity. The first layer computes

$z_i=(q_iK^T)/\sqrt{d},$

which is a linear map with weight $g_\theta$0 and no bias, followed by $g_\theta$1 across the hidden width. The second layer computes

$g_\theta$2

again as a linear map with weight $g_\theta$3 and no bias. The crucial point is that $g_\theta$4 and $g_\theta$5 are not static parameters: they are generated from the input by linear projections, so the effective network parameters are

$g_\theta$6

and the overall computation may be written as

$g_\theta$7

Within this view, attention’s global modeling power is attributed not to explicit token-wise aggregation as such, but to input-conditioned parameters that act as a compressed representation of global context. The formulation is exact rather than approximate: no approximation is required to establish the equivalence, and no explicit bias terms are necessary in the mapping. The same analysis also explains the quadratic cost of attention. Because $g_\theta$8 and $g_\theta$9, the corresponding dynamic MLP has width proportional to $\left[W^{(1)},\dots,W^{(M)}\right]$0, which yields the familiar $\left[W^{(1)},\dots,W^{(M)}\right]$1 compute and $\left[W^{(1)},\dots,W^{(M)}\right]$2 memory behavior (He et al., 3 May 2026).

3. WeightFormer for linear-time global visual modeling

On the basis of that reformulation, the 2026 WeightFormer instantiates dynamic parameterization inside CNN blocks by generating layer weights from fixed-shape global descriptors. The stated objective is to retain implicit global sequence modeling while avoiding construction of a quadratic $\left[W^{(1)},\dots,W^{(M)}\right]$3 attention matrix (He et al., 3 May 2026).

Global context is compressed by descriptors $\left[W^{(1)},\dots,W^{(M)}\right]$4 whose shape is decoupled from sequence length. Two forms are given. Pooling-based compression uses

$\left[W^{(1)},\dots,W^{(M)}\right]$5

for example by Global Average Pooling or Adaptive Average Pooling. Correlation-based compression uses

$\left[W^{(1)},\dots,W^{(M)}\right]$6

which captures second-order statistics independent of $\left[W^{(1)},\dots,W^{(M)}\right]$7.

For dynamic linear layers acting on channels, the model predicts an update $\left[W^{(1)},\dots,W^{(M)}\right]$8 and combines it with a static weight:

$\left[W^{(1)},\dots,W^{(M)}\right]$9

The paper studies several prediction strategies. Pooling-based prediction uses

$\widehat{W}_s$0

Correlation-based variants include

$\widehat{W}_s$1

$\widehat{W}_s$2

$\widehat{W}_s$3

and the Bilateral Activation form

$\widehat{W}_s$4

Here $\widehat{W}_s$5 denotes a pointwise nonlinearity such as SiLU, and the $\widehat{W}_s$6 are learned, resolution-independent matrices.

For dynamic depthwise convolution layers, the model predicts a per-channel $\widehat{W}_s$7 kernel update with $\widehat{W}_s$8:

$\widehat{W}_s$9

The strategies include global pooling-based prediction, spatially adaptive prediction using

$X\in\mathbb{R}^{N\times d}$0

Amp-Dir stabilization with

$X\in\mathbb{R}^{N\times d}$1

and a convolutional predictor in which $X\in\mathbb{R}^{N\times d}$2 for a small two-layer $X\in\mathbb{R}^{N\times d}$3 conv net with channel bottleneck and GELU.

The default WeightFormer block inserts dynamic parameterization sparsely: one dynamic block every third block, which corresponds to $X\in\mathbb{R}^{N\times d}$4 dynamic blocks in a small model. Each dynamic block contains a dynamic DWC using Spatially Adaptive Prediction and an MLP whose first linear layer is dynamically parameterized through Bilateral Activation, while the second linear layer remains static. Standard CNN components, including normalization, residual connections, and pointwise convolutions, are retained; LayerNorm is used. Because parameter generation depends on fixed-shape summaries such as GAP, AAP, or $X\in\mathbb{R}^{N\times d}$5, the resulting complexity is $X\in\mathbb{R}^{N\times d}$6 in compute and memory, with parameter generation cost $X\in\mathbb{R}^{N\times d}$7 or $X\in\mathbb{R}^{N\times d}$8 independent of $X\in\mathbb{R}^{N\times d}$9 (He et al., 3 May 2026).

4. Empirical profile of the 2026 model

The 2026 WeightFormer is evaluated on ImageNet-1K classification, COCO 2017 detection and instance segmentation, ADE20K semantic segmentation, and class-conditional image generation on ImageNet-1K. The reported classification setup trains from scratch for 300 epochs with AdamW, initial learning rate $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$0, cosine decay, 20-epoch linear warm-up, weight decay $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$1, total batch size $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$2, and RandAugment, Mixup, CutMix, and Random Erasing on $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$3 RTX 3090 (He et al., 3 May 2026).

For ImageNet-1K classification, the reported results are: WeightFormer-T with $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$4M parameters and $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$5G FLOPs achieves $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$6 top-1, compared with DeiT-T at $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$7 with $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$8M and $Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$9G; WeightFormer-S with $Y=\mathrm{softmax}(QK^T/\sqrt{d})V.$0M and $Y=\mathrm{softmax}(QK^T/\sqrt{d})V.$1G achieves $Y=\mathrm{softmax}(QK^T/\sqrt{d})V.$2, compared with DeiT-S at $Y=\mathrm{softmax}(QK^T/\sqrt{d})V.$3 and ConvNeXt-S iso. at $Y=\mathrm{softmax}(QK^T/\sqrt{d})V.$4; WeightFormer-B with $Y=\mathrm{softmax}(QK^T/\sqrt{d})V.$5M and $Y=\mathrm{softmax}(QK^T/\sqrt{d})V.$6G FLOPs at $Y=\mathrm{softmax}(QK^T/\sqrt{d})V.$7 achieves $Y=\mathrm{softmax}(QK^T/\sqrt{d})V.$8, compared with ConvNeXt-B at $Y=\mathrm{softmax}(QK^T/\sqrt{d})V.$9 and DeiT-B at $i$0. Dynamic-strategy ablations show a static CNN baseline at $i$1; Dynamic Linear 1 with Bilateral Activation at $i$2; Dynamic DWC with Spatially Adaptive prediction at $i$3; and the combined Dynamic Linear 1 plus Dynamic DWC configuration at $i$4, which is identified as the best accuracy-efficiency trade-off.

The measured efficiency claims are central. At resolution $i$5, WeightFormer-T reaches throughput $i$6 img/s versus DeiT-T at $i$7, and WeightFormer-S reaches $i$8 img/s versus ConvNeXt-S iso. at $i$9. At high resolution, however, the linear-scaling advantage becomes explicit: at $o_i=\mathrm{softmax}(q_iK^T/\sqrt{d})V.$0 (approximately $o_i=\mathrm{softmax}(q_iK^T/\sqrt{d})V.$1 tokens), WeightFormer achieves $o_i=\mathrm{softmax}(q_iK^T/\sqrt{d})V.$2 higher throughput and $o_i=\mathrm{softmax}(q_iK^T/\sqrt{d})V.$3 memory reduction versus DeiT on RTX 3090.

Downstream dense prediction results follow the same pattern. On COCO 2017 with Cascade Mask R-CNN and ViTDet-style backbone integration, WeightFormer-T reports $o_i=\mathrm{softmax}(q_iK^T/\sqrt{d})V.$4 $o_i=\mathrm{softmax}(q_iK^T/\sqrt{d})V.$5 versus DeiT-T $o_i=\mathrm{softmax}(q_iK^T/\sqrt{d})V.$6 and $o_i=\mathrm{softmax}(q_iK^T/\sqrt{d})V.$7 $o_i=\mathrm{softmax}(q_iK^T/\sqrt{d})V.$8 versus $o_i=\mathrm{softmax}(q_iK^T/\sqrt{d})V.$9, with total/backbone FLOPs $z_i=(q_iK^T)/\sqrt{d},$0G/$z_i=(q_iK^T)/\sqrt{d},$1G versus $z_i=(q_iK^T)/\sqrt{d},$2G/$z_i=(q_iK^T)/\sqrt{d},$3G. On ADE20K with UperNet, WeightFormer-T reports $z_i=(q_iK^T)/\sqrt{d},$4 mIoU against DeiT-T at $z_i=(q_iK^T)/\sqrt{d},$5, and WeightFormer-S reports $z_i=(q_iK^T)/\sqrt{d},$6 mIoU against DeiT-S at $z_i=(q_iK^T)/\sqrt{d},$7, again with reduced backbone FLOPs.

The generation experiments extend the claim beyond discriminative vision. On class-conditional ImageNet-1K generation, WeightFormer-S/2 reports FID $z_i=(q_iK^T)/\sqrt{d},$8 versus DiT-S/2 at $z_i=(q_iK^T)/\sqrt{d},$9 and DiG-S/2 at $g_\theta$00, while WeightFormer-B/2 reports FID $g_\theta$01 versus DiT-B/2 at $g_\theta$02 and DiG-B/2 at $g_\theta$03. Effective Receptive Field analysis further reports that static CNNs remain localized, whereas dynamic variants exhibit global ERFs after training. The ablation on dynamic block frequency also establishes a stability boundary: too many dynamic blocks, specifically $g_\theta$04, destabilize training, while the default $g_\theta$05 gives the best throughput-accuracy balance. The paper’s stated limitations are that the study focuses on vision, that deeper analysis of expressivity and inductive biases is needed, and that training stability requires careful normalization, gating, and sparse placement of dynamic blocks (He et al., 3 May 2026).

5. WeightFormer for meta-ensemble parameter learning

The 2022 WeightFormer addresses a different problem: replacing an ensemble of teacher models with a single student whose parameters are predicted directly from teacher checkpoints. The motivation is that ensembles improve performance and robustness but incur inference latency and memory that grow with the number of models $g_\theta$06, while conventional knowledge distillation requires retraining whenever the teacher set changes (Fei et al., 2022).

The standard KD objective is described as

$g_\theta$07

whereas meta-ensemble parameter learning replaces retraining with direct parameter prediction:

$g_\theta$08

The student $g_\theta$09 is then deployed directly.

Architecturally, the model is an encoder-only Transformer with 24 Transformer blocks, hidden size $g_\theta$10, 20 attention heads, and feed-forward size $g_\theta$11. Prediction proceeds layer by layer. For a student layer $g_\theta$12, WeightFormer forms tokens from all teachers’ parameters for that layer and prepends a special "[cross]" token. The Transformer encoder processes that sequence, decodes the relevant hidden states into $g_\theta$13, and propagates the hidden state at the "[cross]" position as the context token for layer $g_\theta$14. This mechanism is explicitly designed to capture both cross-teacher and cross-layer correlations.

Teacher parameters are tokenized by layer type. Convolutional kernels are sliced along the output-channel dimension and flattened into $g_\theta$15 tokens; fully connected layers use columns of the weight matrix as tokens; and self-attention layers in ViT-B/32 flatten Q/K/V/O into a weight sequence encoding all heads jointly. Biases and normalization parameters are treated like other parametric tensors through the same tokenization view. A layer-specific weight embedding dictionary $g_\theta$16 maps raw weight-token dimensions to $g_\theta$17 and back, while relative position embeddings index output channels or units within a layer and model-id embeddings distinguish teachers.

The training procedure combines weight-level pretraining and task-driven optimization. To stabilize learning, the model is first pretrained against a KD-trained student using

$g_\theta$18

optimized with SGD at learning rate $g_\theta$19. Main training uses a task loss

$g_\theta$20

plus shift consistency regularization. The consistency term is computed by running WeightFormer twice, once on the original teacher order and once on a cyclically shifted order, with different "weight cutoff" masks as feature-space dropout:

$g_\theta$21

The combined loss is

$g_\theta$22

with $g_\theta$23. Main training uses initial learning rate $g_\theta$24, decay by $g_\theta$25 every five epochs, checkpoint resampling every $g_\theta$26k steps, and early stopping at validation plateau. Once trained, the generator can produce a student for a new teacher set in one forward pass, after which serving uses only the student rather than the full ensemble (Fei et al., 2022).

6. Empirical profile of the 2022 model and comparative interpretation

The 2022 WeightFormer is evaluated on CIFAR-10, CIFAR-100, and ImageNet with VGG-11, ResNet-50, and ViT-B/32. The reported ensemble size is $g_\theta$27, and for each architecture a pool of 72 checkpoints is constructed, with 60 used to train WeightFormer and 12 for evaluation. The principal metrics are top-1 accuracy, top-5 accuracy, and Expected Calibration Error (Fei et al., 2022).

Across all reported settings, WeightFormer outperforms the single network and standard KD baseline, and the optional fine-tuned variant "WF*" often slightly exceeds the average ensemble. For VGG-11 on ImageNet, ACC-1 is reported as Single $g_\theta$28, Ensemble $g_\theta$29, KD $g_\theta$30, WF $g_\theta$31, and WF* $g_\theta$32; for ResNet-50 on ImageNet, Single $g_\theta$33, Ensemble $g_\theta$34, KD $g_\theta$35, WF $g_\theta$36, and WF* $g_\theta$37; for ViT-B/32 on ImageNet, Single $g_\theta$38, Ensemble $g_\theta$39, KD $g_\theta$40, WF $g_\theta$41, and WF* $g_\theta$42. Calibration follows the same pattern: for ResNet-50 on ImageNet, ECE is Single $g_\theta$43, Ensemble $g_\theta$44, KD $g_\theta$45, WF $g_\theta$46, and WF* $g_\theta$47.

The CIFAR results show the same trend. For example, with VGG-11 on CIFAR-100, ACC-1 is Single $g_\theta$48, Ensemble $g_\theta$49, KD $g_\theta$50, WF $g_\theta$51, and WF* $g_\theta$52; with ResNet-50 on CIFAR-10, ACC-1 is Single $g_\theta$53, Ensemble $g_\theta$54, KD $g_\theta$55, WF $g_\theta$56, and WF* $g_\theta$57. The paper also reports that a simple MLP regressor baseline underperforms WeightFormer, which is used as evidence that cross-layer and cross-teacher attentional aggregation matters for parameter prediction.

Ablations attribute performance gains to the specific architectural devices rather than to parameter regression in general. Removing cross-layer fusion or shift consistency reduces ACC-1 by up to approximately $g_\theta$58, and weight cutoff is also reported as helpful. The ViT-B/32 ImageNet example gives ACC-1 values of WF $g_\theta$59, $g_\theta$60Cross-layer $g_\theta$61, $g_\theta$62Shift $g_\theta$63, and $g_\theta$64Cutoff $g_\theta$65. Increasing the number of teachers improves predicted-student performance, and the paper notes that concatenating more teacher parameters yields gains with only marginal increases in learnable parameters because the encoder is shared.

The limitations are also explicit. The method assumes homogeneous teacher architectures: student and teachers share the same architecture, whether VGG-11, ResNet-50, or ViT-B/32. Cross-architecture teacher-student mapping is left to future work. The approach is not data-free, because training optimizes $g_\theta$66 on labeled data through the predicted student. Sensitivity to normalization and bias handling is not separately studied, and extreme shape differences may require extending the layer-specific embedding dictionary.

Taken together, the two WeightFormer lines clarify a recurring misconception. WeightFormer is not a single fixed architecture but a name attached to two parameter-generation frameworks with different operating regimes. One compresses global visual context into input-conditioned layer weights to obtain implicit global modeling without explicit attention; the other compresses an ensemble of teacher checkpoints into a single student model through Transformer-based weight prediction. The common technical motif is dynamic parameterization, but the empirical claims, computational trade-offs, and intended deployment scenarios are different (He et al., 3 May 2026, Fei et al., 2022).