Function-Preserving Network Expansion

• Function-preserving network expansion is a set of techniques that enable neural networks to expand their architecture while maintaining exact input-output equivalence.
• These methods utilize operations like neuron widening, layer insertion, and module morphism with analytic reparameterizations and activation constraints to preserve functionality.
• Empirical results show that such expansions facilitate dynamic architecture search, improve convergence speed, and allow scalable, efficient deep learning model development.

Function-preserving network expansion comprises a set of architectural and algorithmic techniques allowing neural networks to increase depth, width, or module complexity while exactly preserving the input–output function of the original (“teacher”) network. These approaches enable dynamic architecture search, facilitate rapid exploration of larger or structurally modified networks, and provide the theoretical basis for multi-phase training strategies in which the network can be grown adaptive to data or computing resource constraints without incurring a performance drop or loss of learned knowledge.

1. Theoretical Foundations and Formalism

The core objective in function-preserving network expansion is to construct a mapping from a parameterized function $f(x; \theta)$ (“teacher”) to an expanded function $g(x; \theta')$ (“student”) such that

$\forall x: f(x; \theta) = g(x; \theta')$

where $g$ may have additional neurons, layers, heads, or structural modules not present in $f$. Preservation of function is guaranteed through analytic reparameterizations or construction rules that define the weights and biases of newly introduced components based on the parameters of the original network and, where applicable, a constrained family of activation functions or initialization schemes (Wei et al., 2017, López-Ureña, 2024, Lu et al., 2018, Gesmundo et al., 2023, Painter, 2024).

A central technical principle is that certain classes of activation functions and architectural motifs admit algebraic identities ensuring decomposability, invertibility, or partition-of-unity properties, thus supporting exact expansions. This includes spline-based activations satisfying refinability and sum-to-identity (subdivision-theoretic) properties (López-Ureña, 2024), explicit graph-morphism constructs for convolutional modules (Wei et al., 2017), and coefficient rescalings for transformer blocks and convolutional layers (Gesmundo et al., 2023).

2. Atomic Expansion Operations and Algorithms

Function-preserving expansion is implemented via a taxonomy of atomic operations, each defined by precise parameter manipulation and, where relevant, update rules for biases and nonlinearity arguments. Principal expansion types include:

a. Neuron and Channel Widening:

Addition of new units to a layer, with outgoing weights set to zero or distributed such that the aggregate effect matches the teacher's output. For example, in MLPs and transformers, if the new neurons' output weights are zero-initialized, the expanded layer's post-activation representation reduces to the original under affine-linear transformations (Gesmundo et al., 2023). For convolutional nets, parallel splits or duplicated feature maps with appropriate summing ensure functional equivalence (Wei et al., 2017).

b. Layer Insertion (“Deepening”):

New layers can be inserted as identity mappings (e.g., $W=I$, $b=0$ for linear activations, or via residual blocks $z_\theta(x)=0$ to preserve the identity in ResNets) (Painter, 2024). Subdivision-theoretic activations allow analytic decomposition of an affine+nonlinear mapping into two or more layers via the sum-to-identity property, ensuring functional invariance even in the presence of nonlinearities (López-Ureña, 2024).

c. Module Morphism (Graph Expansion):

A single connection or layer can be morphed into arbitrarily complex, single-source, single-sink DAG modules by recursive application of two operations: sequential splits (serial connection, “TYPE-I”) and parallel splits (multiple branches, “TYPE-II”). These operations induce convolutional decompositions or additive branches, with function preservation proven via explicit convolution equations and, in the case of irreducible modules, by solving linear deconvolution systems (Wei et al., 2017).

d. Sparse or Compact Expansion:

After expansion, redundant units may be pruned using sparsity-promoting optimization (e.g., independently interpretable Lasso (iiLasso)), resulting in a more compact architecture without altering the computed function (Lu et al., 2018). This approach ensures that only those neurons that contribute uniquely to functional representation are retained.

e. Architectural Move Composability:

All basic expansion moves (widening, deepening, attention-head addition, etc.) are mutually commutative and can be chained in arbitrary order as long as appropriate parameter constraints are enforced (Gesmundo et al., 2023). This supports flexible scheduling during training or architecture search.

3. Role of Activation Functions and Algebraic Constraints

Most expansion techniques are limited by the algebraic properties of the activation function. Standard componentwise activations such as ReLU and GELU are compatible with several expansion moves (notably MLP and attention width expansion) because zeroing outgoing weights or using identity weight matrices with ReLU preserves function (Gesmundo et al., 2023, Painter, 2024).

However, deeper theoretical guarantees—such as the ability to freely insert layers or subdivide neurons—require activations satisfying refinable and sum-to-identity properties as defined in subdivision theory (López-Ureña, 2024). Let $\sigma$ be such an activation; then:

• Refinability:

$$\sigma(t) = \sum_{l=0}^{A-1} a_l\,\sigma(2t + \tau - l)$$

permits neuron subdivision.

• Sum-to-Identity:

$$t = \sum_{l=0}^{B-1} \sigma(t + \mu - l)\quad \text{for } t \in I$$

permits insertion of “identity-splitting” layers.

Spline activations derived from B-splines provide a concrete family with closed-form expressions, refinability masks, and explicit domains of validity. Only these specialized activations admit fully general, closed-form function-preserving moves via analytic initialization.

4. Representative Expansion Protocols

The following table provides a summary of the atomic expanders across major architectures (notation per-referenced papers):

Expansion Type	Mechanism/papers	Exactness Domain
MLP/Conv neuron widening	Zero outgoing weights	All ReLU/GELU
Transformer attention growth	Zero/scale constraints	All ReLU/GELU
Layer insertion	Residual/identity layers	ReLU, sum-to-identity
Graph-based module morphism	TYPE-I/II + linear solve	ConvNets, all DAGs
Spline-based subdivision	Subdivision-theory rules	Domain-limited (Ω)
Sparse expansion + pruning	iiLasso, coordinate step	Any, preserves on-batch

Specific pseudocode, proof sketches, and parameter update rules for all expansion types—encompassing both fully-connected and transformer architectures—are given in (Wei et al., 2017, Lu et al., 2018, Gesmundo et al., 2023, López-Ureña, 2024).

5. Empirical Properties and Theoretical Guarantees

Empirical studies confirm that function-preserving transformations permit architecture scaling without drop in model performance, preserve convergence speed, and can reduce the total compute cost and data requirements for larger models when expansion is staged adaptively (Lu et al., 2018, Painter, 2024). In comparative analyses:

• R2R transforms support exact preservation for residual networks, initialization of all new filters with independent free parameters, and match or exceed both Net2Net and standard Network Morphism in test accuracy and filter diversity (Painter, 2024).
• CompNet demonstrates accelerated convergence after compact expansion and pruning, with up to 55% reduction in redundant filters and no loss in predictive performance—e.g., on VGG architectures for CIFAR-10 and MNIST (Lu et al., 2018).
• Spline-based expansion maintains exactness on all inputs in domain Ω (determined by the support of the sum-to-identity interval) (López-Ureña, 2024).
• For transformer architectures, six composable expanders (MLP width, head count, value-dim, key/query-dim, hidden dim, depth) provide analytic guarantees of output invariance under prescribed initializations (Gesmundo et al., 2023).

Theoretical results establish that for any single-source, single-sink convolutional DAG, there is always a solution in the expanded parameter space guaranteeing function preservation (Wei et al., 2017). In simple perceptron models, expansion plus pruning admits a mean-field description in which the learning dynamics correspond to the addition of quadratic slack variables in the SVM loss, resulting in better generalization performance (Steinberg et al., 2020).

6. Limitations, Conditions, and Open Directions

Several constraints limit universality of these techniques:

• Analytic expansions requiring refinable/sum-to-identity activations are only applicable on restricted input domains; practically, the input must be scaled or the activation interval chosen to encompass the entire data range (López-Ureña, 2024).
• Function-preserving insertion of nonlinearities is activation-dependent: certain moves (e.g., Net2DeeperNet) require strictly idempotent activations (e.g., ReLU), precluding generality for sigmoid or tanh nonlinearities (Painter, 2024).
• For complex module morphisms, the linear system to solve in parameter space may in theory be overdetermined; expanding the dimensionality of new filters ensures the solution space is nonempty, but this increases parameter count (Wei et al., 2017).
• In practice, deep or wide expansions may degrade numerical conditioning or increase compute cost—often mitigated by interleaving training/pruning cycles or scheduling growth at convergence plateaus (Lu et al., 2018, Gesmundo et al., 2023).

A plausible implication is that further exploration of advanced activation functions, domain-adaptive scaling, and algorithmic sparsification techniques could enlarge the practical scope of function-preserving expansion moves. The design of dynamic, data-driven architecture search procedures leveraging such operations remains an active research area.

7. Applications and Implementation Best Practices

Function-preserving expansions underpin several methodologies in neural architecture search, continual learning, dynamic scaling, and transfer/pre-training pipelines. Key practices include:

• Scheduling expansion moves after initial convergence or at training plateaus to minimize optimization shock (Gesmundo et al., 2023).
• Inserting new capacity (neurons, layers, attention heads) via parameterizations that guarantee output invariance, followed by resumed or fine-tuned training.
• Selecting activation functions and normalization schemes compatible with the expansion types required by the target model class.
• Pruning of redundant units after expansion using sparsity-inducing regularizers to control model size and computation (Lu et al., 2018).
• In transformer-based architectures, updating all relevant parameter matrices simultaneously, ensuring consistent dimensionality (especially for skip-connected or residual modules) (Gesmundo et al., 2023).

Function-preserving network expansion continues to play a central role in the scalable, efficient, and interpretable design of modern deep learning architectures.