μTransfer Paradigm in Deep Learning

• μTransfer paradigm is a theoretically grounded framework that enables robust zero-shot transfer of hyperparameters, gradients, and representations across diverse models and tasks.
• It employs μ-parameterization and scaling laws to transfer optimal base learning rates from small proxy models to large-scale models, avoiding costly hyperparameter sweeps.
• Extensions of μTransfer include adaptive gating, mixture formulations, and applications in reinforcement learning and MoE architectures, with empirical evidence supporting improved performance.

The μTransfer paradigm refers to a set of principled frameworks and methodologies for transferring knowledge—typically hyperparameters, optimizers, gradients, or representations—across scales, architectures, or task domains in deep learning and reinforcement learning. In contrast to heuristic or ad-hoc transfer approaches, μTransfer relies on theoretically grounded scaling rules, mixture formulations, or adaptive gating, yielding predictable and robust transferability of key model properties from proxy configurations to broader settings.

1. Theoretical Foundations and General Definition

The μTransfer paradigm crystallizes in multiple lines of research, most notably in the context of hyperparameter transfer for model scaling—primarily via μ-Parameterization (μP)—and in transfer reinforcement learning as a formal framework for weighting and combining information from multiple Markov Decision Processes (MDPs). The unifying principle is the controlled and theoretically analyzable transfer of information based on scaling laws, mixture formulations, or differentiable gates, yielding performance guarantees or robust empirical transfer.

In μ-Parameterization, μTransfer enables zero-shot transfer of initialization and optimization hyperparameters (e.g., base learning rates) from a small proxy model of width $P$ to a much larger target model of width $M$ by enforcing parameter scalings that keep forward signal and update magnitudes $O(1)$ as $M\rightarrow\infty$ (Lingle, 2024). In reinforcement learning, μTransfer encompasses the sharing of sample transitions from source MDPs to a target MDP, where mixture weights $\lambda$ control the tradeoff between estimation error and transfer bias (Lazaric et al., 2011). In parameter transfer units, μTransfer refers to per-activation learnable blending between source and target domain activations, subsuming classical discrete transfer modes (Zhang et al., 2018).

2. μ-Parameterization and Scaling Laws for Zero-Shot Hyperparameter Transfer

The μ-Parameterization formalism, developed from the Tensor Programs framework, addresses the instability of standard parameter scaling as model width $M$ increases. Under μP, every non-embedding weight matrix $W$ is initialized with variance $\sim 1/M$ and trained with a per-parameter learning rate $\sim P/M$, where $P$ is the proxy width for which $M$0 (the optimal base learning rate) was found. Specifically, for a decoder-only transformer:

• Embedding matrix: $M$1, learning rate $M$2.
• Attention/MLP/unembedding matrices: $M$3 or $M$4 (as appropriate), learning rate $M$5.
• Attention scale: $M$6 (contrasting the standard $M$7).

This parameterization ensures that forward signal variance and backward update statistics remain order-one across scaling, leading to smooth $M$8 corrections in the dynamics. As a consequence, the learning rate optimum found at small scale $M$9 transfers exactly to large scale: $O(1)$0 (Lingle, 2024). Thus, costly hyperparameter sweeps at large scale are avoided.

3. Extensions: Complete-μE for MoE and General Transformer Hyperparameter Transfer

While μP provides zero-shot transfer in dense models, it does not directly address Mixture-of-Experts (MoE) architectures or batch/duration scaling. Complete-μE generalizes μTransfer to arbitrary MoE setups by factorizing the transfer problem:

• Bridge I: Maps dense FFNs to dense MoEs via active-width μP and a normalized router scale ($O(1)$1 for $O(1)$2 experts).
• Bridge II: Maps dense MoE to sparse MoE using the SDE (stochastic differential equation) transfer rule, which shows that first-order learning rate and weight decay corrections cancel when changing the number of activated experts $O(1)$3 if total batch and duration are fixed. The only residual is a bounded signal-to-noise shift, so the original hyperparameters remain near-optimal across MoE axes (Peng et al., 22 May 2026).

Empirically, tuning on a single dense FFN transfers near-optimally to all MoE configurations and batch/duration settings, leading to the "tune dense once, transfer to all" practical recipe.

4. μTransfer in Reinforcement Learning

In reinforcement learning, μTransfer is instantiated both as universal successor features (USFs) for efficient transfer across goal parametrizations and as sample-based transfer from multiple MDPs.

• Universal Successor Features: For goal-conditioned MDPs, USFs $O(1)$4 generalize the action-value decomposition $O(1)$5 so that a single learned network transfers policy representations across unseen goals $O(1)$6 with zero-shot capability, yielding strong empirical gains in both discrete (gridworld) and continuous (MuJoCo) domains (Ma et al., 2020).
• Sample-based Transfer Across MDPs: Transfer from multiple source MDPs is formulated via a mixture of Bellman operators $O(1)$7, corresponding to an "average" MDP. Adaptive algorithms (Best Average Transfer, Best Trade-off Transfer) select mixture weights $O(1)$8 or sample fractions $O(1)$9 to minimize expected transfer error or avoid negative transfer, with provable finite-sample error bounds and robust empirical performance (Lazaric et al., 2011).

5. Parameter Transfer Units: Microscopic μTransfer via Gating

The μTransfer paradigm also encompasses learnable, per-activation transfer mechanisms in neural architectures. The Parameter Transfer Unit (PTU) learns per-layer, per-coordinate mixing via two neural gates—fine-tune and update—that control, for each unit, the degree of retention, adaptation, or replacement of source activations within a target network. The PTU's gating equations subsume classical transfer options (frozen share, fine-tune, random-init) and enable a continuous, data-driven transfer spectrum. Empirical results demonstrate improved accuracy and convergence across vision and sequence tasks versus heuristic transfer (Zhang et al., 2018).

6. Empirical Evidence and Best Practices

Empirical studies consistently report:

• Perfect or near-perfect transfer of the small-model hyperparameter optimum to large-scale models under baseline μTransfer setups, even across parameter increases up to $M\rightarrow\infty$0 (Lingle, 2024).
• Robustness of μTransfer to most architectural and optimizer ablations, with exceptions including RMSNorm gains, nonstandard attention scaling, or non-matching optimizers.
• In MoE and hybrid architectures, Complete-μE maintains stable optima across a wide range of activated experts, total experts, batch sizes, and duration, with only minor and theoretically understood residual drift (Peng et al., 22 May 2026).
• In RL, USFs accelerate transfer and yield superior generalization to held-out goals, while adaptive mixture transfer in multi-MDP settings provably trades off sample efficiency and negative transfer (Lazaric et al., 2011, Ma et al., 2020).

Practical μTransfer deployment requires adherence to theoretically motivated initialization, scaling, and optimizer rules per architecture class, and careful avoidance of known transfer-breaking configurations.

7. Significance, Limitations, and Scope

μTransfer formalizes and systematizes knowledge transfer, reducing the need for repeated hyperparameter sweeps and mitigating transfer bias or negative transfer in multi-task settings. Its effectiveness depends on adherence to scaling laws and model assumptions; known limitations arise for architectures or optimizer choices that violate μP invariants. The paradigm is extensible, as evidenced by its adaptation to MoE (Peng et al., 22 May 2026), continuous RL domains (Ma et al., 2020), and per-activation transfer (Zhang et al., 2018).

A plausible implication is that future transfer learning research may focus increasingly on theoretically grounded, scale-consistent transfer mechanisms, with μTransfer as a core conceptual and technical foundation.