Homotopy Training Algorithm

• HTA is a strategy that continuously deforms a simple neural network into a complex one, leveraging smooth parameter transitions to track optimal solutions.
• The method systematically advances from low-complexity models to full-scale architectures, reducing the risk of poor local minima and ensuring stable convergence.
• HTA enables adaptive architecture selection and has shown empirical performance improvements, with significant reductions in test error rates on deep networks.

A Homotopy Training Algorithm (HTA) is a general strategy for solving difficult optimization or learning problems by starting from an easier surrogate and morphing it continuously, via a parameterized path (the “homotopy”), into the target problem. In the context of fully connected neural networks, an HTA constructs a continuous deformation from a simplified model (smaller or shallower network) to the original, complex network, tracking optima at each stage. This approach leverages the intuition that easier models are less likely to yield poor local minima and that following a continuous solution path increases the probability of reaching an optimal or high-quality solution for the original, highly nonconvex optimization landscape (Chen et al., 2019).

1. Methodological Framework

The HTA is anchored in the homotopy (continuation) principle, which defines a continuous family of models parameterized by $t\in[0,1]$. The simplest instantiation is a convex combination of a simplified network $y_1(x;\theta)$ and a more complex network $y_2(x;\theta)$: $y(x;t) = (1-t) y_1(x;\theta) + t\, y_2(x;\theta),$ with $t=0$ corresponding to the simple model and $t=1$ to the full model. For fully connected neural networks, $y_1$ might represent a single-hidden-layer network and $y_2$ a network with an additional layer. The homotopy can be constructed for layer-wise, node-wise, or other architectural differences. The functional $H_i(x;\theta, t)$ may be formed for successive model pairs: $$H_i(x;\theta, t) = (1-t) y_i(x; \theta) + t \, y_{i+1}(x;\theta).$$ Training then proceeds by incrementally stepping $t$ from 0 to 1, optimizing $\theta$ at each $t$ using the previous solution as the initialization.

2. Optimization Path and Theoretical Properties

HTA establishes a solution path in network parameter space,

$$\theta^*(t) = \arg \min_\theta f(\theta; t), \quad f(\theta; t) = \sum_{j=1}^N \left\| H_i(x^j; \theta, t) - y^j \right\|^2,$$

which varies smoothly under reasonable regularity conditions. Analogous to predictor-corrector or path-following algorithms in numerical algebraic geometry, this continuous path helps avoid abrupt transitions to unfavorable regions of the loss surface—thereby, with high probability, steering the optimization towards improved minima for the target model. The algorithm is particularly suited for highly nonconvex landscapes where direct training of a large network may fail to reach a satisfactory solution.

3. Algorithmic Procedure

The standard procedure for HTA in neural networks comprises the following steps:

• Initialization: Train the simplest (smallest) model to convergence using standard optimization (e.g., SGD).
• Homotopy Progression: For each incremental $t$:
  • Build the homotopy model $H(x;\theta, t)$.
  • Optimize network parameters $\theta$, initialized from the previous $t$, to minimize training loss for $H(\cdot;\theta, t)$.
  • Continue incrementally increasing $t$ until $t=1$, at which point the original network is recovered.

Pseudocode for a two-hidden-layer network homotopy might be as follows (compactly, omitting indexing over minibatches for SGD):

for t in np.linspace(0, 1, num_steps): # Construct the current homotopy model H(x; θ, t) loss = lambda θ: np.sum((H(x_data, θ, t) - y_data)**2) θ = optimize(loss, θ_init=θ_prev) θ_prev = θ

At each step, the current optimizer state $\theta_{t-\delta t}$ is reused to initialize the next step, enabling rapid convergence.

4. Adaptive Structure Learning

A notable benefit is the adaptive search for optimal network architecture. By augmenting layers or nodes one at a time, and monitoring whether the new parameters converge to values near zero, the algorithm can determine whether additional capacity is needed. If after the addition of nodes their associated weights remain negligible, this indicates the model has reached sufficient expressivity. This node-wise or layer-wise continuation not only facilitates model selection but also yields structured simplification over brute force grid search.

5. Empirical Performance

HTA demonstrates empirically significant performance improvements on complex models. For example, on the VGG13 architecture with batch normalization and trained on CIFAR-10, HTA reduced the test error rate by approximately 11.86% compared to conventional direct training. Across other VGG models (VGG11, VGG16, VGG19), error rate improvements ranged from roughly 7% to over 11%. In both classical function fitting and deep vision tasks, validation loss and test performance benefited consistently from homotopic progression.

6. Comparative Analysis

Relative to direct (non-homotopic) training:

• Efficiency: Early stages involve low-complexity models, so initial optimization is computationally inexpensive and well-conditioned. This avoids early trapping in poor optima.
• Optimization Landscape: Continuous deformation prevents solution “jumps” between disconnected basins that often beset deep nonconvex objectives.
• Generalization: The stepwise approach is empirically observed to reduce test error, suggesting improved generalization potentially due to the path-following process restricting the parameter search to well-behaved regions.
• Structure Discovery: The method enables efficient and automated discovery of minimum sufficient architecture, as larger networks are only adopted if justified by performance improvements during continuation.

7. Application Scope and Considerations

HTA is applicable to a wide array of nonconvex optimization tasks in deep learning and beyond where a smooth parameter or structural deformation from an easy-to-learn model to a complex target exists. It is especially advantageous for complicated architectures, large parameter spaces, or applications requiring robust model selection. Implementation is straightforward, as it requires only sequential problem definition and careful homotopy parameter scheduling.

Resource requirements scale with the total number of homotopy steps; however, since each optimization at small $t$ is faster and much better conditioned than direct large-scale model training, the amortized cost is often favorable. The step size in $t$ should be chosen to balance computational efficiency and tracking fidelity, with smaller increments required if the optimization path is highly curved or when transitions induce sharp landscape changes.

---

In summary, Homotopy Training Algorithms combine the path-following theory of numerical continuation with neural network optimization by constructing a continuous, smooth pathway in model architecture or loss landscape, enabling consistently improved convergence, architectural adaptability, and test performance in highly nonconvex settings (Chen et al., 2019).