Vector Cloning & Linear Layer Expansion

• Vector cloning is the mechanism by which principal singular modes are replicated and synchronized across layers in deep linear networks.
• Linear layer expansion deconstructs a single transformation into sequential factors, altering the optimization landscape without adding representational capacity.
• These techniques enhance training dynamics and parameter efficiency, underpinning practical architectures like ExpandNets and Dynamic Clone Transformers.

Vector cloning and linear layer expansion are central concepts in the structural and dynamical analysis of deep (linear and linearized) neural architectures. They describe how the effective information-carrying modes of a network's end-to-end map are embedded, replicated, and manipulated within and across layers, especially in over-parameterized or expanded networks. Recent research provides rigorous mathematical, geometric, and algorithmic frameworks for understanding these phenomena, with widespread implications for optimization dynamics, architecture design, parameter sharing, and the efficient training and deployment of deep networks.

1. Definitions and Foundations

Vector cloning denotes the mechanism whereby the singular directions (principal input-output modes) of a deep network's overall linear mapping are embedded within each individual layer. In over-parameterized multi-layer linear networks (LNNs), any singular vector of the end-to-end transformation is effectively “cloned” into every layer's corresponding set of weights, growing in norm synchronously during training by gradient flow (Basu et al., 2019).

Formally, for a depth-$L$ linear network with output $\hat{y} = W_L\,W_{L-1}\cdots W_1\,x$, where $W_\ell$ are learnable matrices and $\hat{y}$ is ultimately a function of their product, the vector cloning property implies that each singular direction of $W_L\cdots W_1$ appears in every $W_\ell$:

• The singular value spectrum evolves such that for every mode $k$ and layer $\ell$, the $k$-th singular value $\sigma_{k,\ell}$ grows in lockstep with all other layers (modulo initialization), under the ODE:

$$\tau \frac{d\sigma_{k,\ell}}{dt} = \left(\prod_{i\neq \ell} \sigma_{k,i}\right) \left( s_k - \prod_{i=1}^L \sigma_{k,i} \right)$$

where $s_k$ is the $k$-th singular value of the target covariance (Basu et al., 2019).

Linear layer expansion refers to the process of replacing a single linear transformation $W$ by its factorization into a sequence of multiple linear layers whose composition equals $W$. In contrast with non-linear expansion, no new representational capacity is added, but the optimization landscape and the population of equivalent parameterizations (the "fiber") are dramatically altered (Guo et al., 2018, Shewchuk et al., 23 Apr 2024).

2. Geometric and Algebraic Structure of Vector Cloning

The geometry of vector cloning and its manifestation in deep linear nets has been precisely characterized by the theory of "fibers"—the set of all tuples of layer weights $(W_L, \ldots, W_1)$ producing an identical effective map $W = W_L\cdots W_1$ (Shewchuk et al., 23 Apr 2024). These fibers stratify into manifolds corresponding to the cloning configuration within each layer, as described by the interval multiplicities $\omega_{ji}$ measuring, for each hidden layer $j$, the number of "clones" assigned to each embedded direction.

This structure leads to the following principles:

• Any SVD basis of $W$ can be embedded via almost-identity blocks in the hidden layers, distributing the modes as clones into the layer dimensions while preserving the overall transformation.
• Each stratum in the space of parameters (a subset of the fiber with a fixed cloning pattern) is associated with a well-defined manifold structure, whose dimension is determined explicitly by the cloning arrangement:

$$D(\underline{r}) = \sum_{j=1}^{L-1}(d_j - r)\,(d_{j+1} + d_{j-1} - 2r) + r^2$$

where $d_j$ is the width of layer $j$, $r$ is the rank of $W$, and the $r^2$ term corresponds to the degrees of freedom of the non-cloned part (Shewchuk et al., 23 Apr 2024).

3. Dynamics and Layer-wise Growth in Linear and Linearized Nets

The learning dynamics in LNNs are governed by coupled ODEs enforcing synchrony in the evolution of the singular spectrum across layers—a direct consequence of vector cloning:

• For all adjacent layers $l$,

$$\frac{d}{dt}(W_{l+1}^T W_{l+1}) = \frac{d}{dt}(W_l W_l^T)$$

enforcing equality in the growth rates of squared singular spectra ("Layer-Growth Symmetry") (Basu et al., 2019).

• This symmetry underpins distinct training phases: initial super-exponential growth of all singular modes, a plateau ("feature consolidation") at intermediate strengths, and slow fine-tuning near equilibrium (Basu et al., 2019).

Nonlinear networks (e.g., ReLU MLPs) locally inherit these properties under linearization, but sample-dependent masking disrupts perfect symmetry except in deep layers where similar masks dominate within class clusters.

4. Expansion Mechanisms: Parametric, Structural, and Efficient Implementations

Linear layer expansion is utilized both for expressively increasing depth during training and for parameter sharing. Several practical mechanisms exploit vector cloning, including:

• ExpandNets: Replace any compact linear or convolutional layer $W$ with a sequence $W_k...W_1$, training the over-parameterized expanded net to improve optimization and generalization, and then algebraically contracting to $W_\text{eff}$ for inference (Guo et al., 2018). Two notable expansion types are channel-expansion and kernel-expansion.
• Dynamic Clone Transformer (DCT)/Multi-Path FC: Expand a vector $x\in\mathbb R^C$ into $p$ “clones” concatenated with learnable offsets, capturing the mapping

$$Y = [x + \Delta_1(x);\ x + \Delta_2(x);\ ...;\ x + \Delta_p(x)]$$

for efficient channel increase with lower parameter/memory cost than a full FC layer (Ye, 2021).

• TLEG (Linear Expansion of "Learngene"): Synthesize per-layer parameters for transformers as a linear interpolation of two core tensors $\theta_a, \theta_b$, effectively "cloning" a learnable genesis vector across different layers with a linearly varying offset, yielding initialization flexibility and parameter efficiency (Xia et al., 2023).

Expansion Mechanism	Principle	Practical Advantages
ExpandNets	Deep linear factorization	Optimization, better generalization, contractible
DCT/MPFC	Cloning + learnable offset	Efficient channel expansion, lower cost
TLEG	Linear weight interpolation	Depth flexibility, strong parameter sharing

5. Cloning in Convolution and Fully Connected Equivalence

Vector cloning is also a structural requirement for the algebraic translation of convolutional layers into fully connected forms. The classic "im2col" transformation implements vector cloning by replicating each input spatial location across all overlapping receptive fields, stacking them into a large matrix $X_\text{cloned}$ so that convolution becomes a matrix product:

$$Y_\text{flat} = X_\text{cloned}\,W_\text{expanded} + \mathbf{1}b^T$$

This recasting reveals the intrinsic role of cloning in ensuring that each input feature is correctly distributed across all necessary filter applications in the "fully connected" emulation of a convolution (Ma et al., 2017).

6. Practical Layer Expansion and Initialization Strategies

Effective vector cloning and expansion have direct implications for neural network design and training:

• When expanding the dimension of a linear layer (e.g., $n_\ell \to n_\ell'$), fast convergence is maintained only if existing singular directions are copied ("cloned") into the new directions, avoiding random initializations that would disconnect them from the current subspace structure (Basu et al., 2019).
• Proper initialization (e.g., orthogonal or Glorot) aligned across layers prevents bottlenecks in the propagation of cloned modes, ensuring stable training and effective learning dynamics.
• Cloning-inspired parameter sharing (as in TLEG or ALBERT-style transformers) underlays architectures that interpolate between full-sharing and full-independence, facilitating trade-offs between efficiency and expressivity (Xia et al., 2023).

7. Broader Implications and Theoretical Insights

The theory of vector cloning and linear expansion unifies perspectives on over-parameterization, symmetry-induced regularization, and the manifold structure of networks’ parameter spaces:

• Fibers of equivalent weight-parameterizations stratify into manifolds labeled by cloning profiles, determining the geometric and optimization properties of the network (Shewchuk et al., 23 Apr 2024).
• Over-parameterization via expansion modifies the loss landscape, often flattening minima and improving gradient flow; vector cloning provides the algebraic backbone permitting such transformations without altering the network’s functional output (Guo et al., 2018).
• The principle generalizes to nonlinear networks in the context of batch normalization (which aligns singular spaces) and advanced initialization or mask-sharing techniques aimed at maintaining high-gradient fidelity during training (Basu et al., 2019).

Altogether, vector cloning and linear layer expansion are not only structural features but also foundational tools for neural architecture manipulation, efficient training, algorithmic flexibility, and interpretability across modern deep learning practice.