Adaptive Ansatz Construction Algorithm

• Adaptive Ansatz Construction Algorithm is a method that dynamically refines neural ODE architectures via optimal control principles and dual-weighted residual error estimation.
• It employs an explicit Euler forward solve, backward adjoint propagation, and an H1 gradient reconstruction to identify and insert new layers where loss sensitivity is highest.
• The method achieves computational efficiency by starting with a coarse architecture and strategically refining layers, leading to enhanced performance and training depth economy.

Searching arXiv for the specified paper and closely related work on adaptive ansatz construction for neural ODEs. The Adaptive Ansatz Construction Algorithm is a layerwise adaptive construction method for neural network architectures derived from the optimal control of neural differential equations. In the formulation introduced in “Layerwise goal-oriented adaptivity for neural ODEs: an optimal control perspective” (Hintermüller et al., 12 Jan 2026), the network architecture is identified with the discrete ansatz of a neural ODE, namely the depth $K$, the time grid $\{t_k\}$, and the piecewise control parameterization $\theta_\tau \in CG(1)$. The method combines a goal-oriented dual-weighted residual technique, a $DG(0)$ Galerkin discretization leading to an explicit Euler time marching scheme, and gradient-based optimization to insert layers where the loss is most sensitive (Hintermüller et al., 12 Jan 2026).

1. Continuous model and optimal-control interpretation

The algorithm starts from a continuous-time neural ODE in which the trainable coefficients are treated as controls. Let $W(t)\in\mathbb R^{d\times d}$ and $b(t)\in\mathbb R^d$ be time-dependent weights and biases collected in

$$\theta(t) = (\operatorname{vec} W(t), b(t))^\top \in \mathbb R^n,\qquad n=d^2+d.$$

The neural vector field is

$f(x,\theta)=\sigma(Wx+b),$

with componentwise activation $\sigma$. For a batch of $m$ samples, the state dynamics on $\{t_k\}$0 are

$\{t_k\}$1

where

$\{t_k\}$2

The controls $\{t_k\}$3 are the coefficients $\{t_k\}$4 themselves. The regularizer enforces $\{t_k\}$5, and the learning objective depends on the terminal state:

$\{t_k\}$6

The paper uses the regularization

$\{t_k\}$7

leading to the ODE-constrained optimization problem

$\{t_k\}$8

subject to

$\{t_k\}$9

The associated control-to-state map is denoted by $\theta_\tau \in CG(1)$0 (Hintermüller et al., 12 Jan 2026).

This formulation makes the architecture itself part of the control problem. The discrete ansatz is therefore not fixed in advance; it is refined according to an error representation tied directly to the quantity of interest, namely the loss $\theta_\tau \in CG(1)$1.

2. Discrete ansatz, time stepping, and gradient equation

The temporal discretization defines the network architecture. On a grid

$\theta_\tau \in CG(1)$2

the state trial space uses $\theta_\tau \in CG(1)$3, so $\theta_\tau \in CG(1)$4 is piecewise constant on $\theta_\tau \in CG(1)$5. With midpoint evaluation

$\theta_\tau \in CG(1)$6

the forward solve becomes the explicit Euler update

$\theta_\tau \in CG(1)$7

The adjoint is discretized by a backward Euler-like recurrence:

$\theta_\tau \in CG(1)$8

At the continuous level, the adjoint satisfies

$\theta_\tau \in CG(1)$9

A distinctive feature of the method is that the $DG(0)$0-regularizer induces a boundary-value problem for the reduced gradient $DG(0)$1:

$DG(0)$2

Its $DG(0)$3 discretization yields the linear system

$DG(0)$4

with mass matrix $DG(0)$5, stiffness matrix $DG(0)$6, and $DG(0)$7 (Hintermüller et al., 12 Jan 2026).

The algorithm therefore alternates among three coupled solves on the current grid: forward state evolution, backward adjoint propagation, and a gradient reconstruction in the $DG(0)$8 geometry. In the paper’s terminology, the architecture is the discrete ansatz for the continuous neural ODE, and the refinement of this ansatz is driven by the same optimality system that governs training.

3. Dual-weighted residual estimator and layerwise indicators

The adaptive criterion is derived from a goal-oriented dual-weighted residual representation. For solutions $DG(0)$9 of the continuous optimality system and $W(t)\in\mathbb R^{d\times d}$0 of the discrete one,

$W(t)\in\mathbb R^{d\times d}$1

where $W(t)\in\mathbb R^{d\times d}$2 is a higher-order cubic remainder under $W(t)\in\mathbb R^{d\times d}$3 (Hintermüller et al., 12 Jan 2026).

The resulting layerwise bound is

$W(t)\in\mathbb R^{d\times d}$4

with decompositions

$W(t)\in\mathbb R^{d\times d}$5

The dual weights are the local state and adjoint errors,

$W(t)\in\mathbb R^{d\times d}$6

The paper emphasizes that these dual weights amplify residuals where the quantity of interest, the loss, is most sensitive (Hintermüller et al., 12 Jan 2026).

The $W(t)\in\mathbb R^{d\times d}$7 formulation yields layerwise residuals containing both volume and jump contributions. For the state,

$W(t)\in\mathbb R^{d\times d}$8

and for the adjoint,

$W(t)\in\mathbb R^{d\times d}$9

The computable layer indicator is

$b(t)\in\mathbb R^d$0

where $b(t)\in\mathbb R^d$1 is a piecewise-quadratic reconstruction of $b(t)\in\mathbb R^d$2 (Hintermüller et al., 12 Jan 2026). This is the central criterion used to decide where new layers are inserted.

4. Layer insertion, refinement strategy, and optimization loop

The adaptive procedure is explicitly layerwise. On each refinement event, the algorithm marks the interval with largest indicator,

$b(t)\in\mathbb R^d$3

and splits that interval by inserting

$b(t)\in\mathbb R^d$4

This increases the depth from $b(t)\in\mathbb R^d$5 to $b(t)\in\mathbb R^d$6. The newly inserted layer parameters are initialized by averaging neighboring parameters,

$b(t)\in\mathbb R^d$7

which, according to the paper, leverages the $b(t)\in\mathbb R^d$8 regularization to maintain smoothness (Hintermüller et al., 12 Jan 2026). The local step size is halved on the refined interval.

The parameter update itself is a steepest descent step in the $b(t)\in\mathbb R^d$9 metric,

$$\theta(t) = (\operatorname{vec} W(t), b(t))^\top \in \mathbb R^n,\qquad n=d^2+d.$$0

where $$\theta(t) = (\operatorname{vec} W(t), b(t))^\top \in \mathbb R^n,\qquad n=d^2+d.$$1 solves the linear system above. The dual-weighted control term is assembled from the reconstructed adjoint:

$$\theta(t) = (\operatorname{vec} W(t), b(t))^\top \in \mathbb R^n,\qquad n=d^2+d.$$2

The paper states that backtracking or fixed schedules may be used for $$\theta(t) = (\operatorname{vec} W(t), b(t))^\top \in \mathbb R^n,\qquad n=d^2+d.$$3, damping may be used when the loss increases, and convergence can be monitored through loss decrease, gradient norm, and validation accuracy. Although the formal optimization description uses steepest descent, the implementation also uses Adam in practice, with optimizer moments reset or adjusted when layers are inserted (Hintermüller et al., 12 Jan 2026).

A concise rendering of the algorithmic loop is as follows.

1. Solve the forward problem by explicit Euler on the current grid.
2. Compute the terminal loss derivative and propagate the discrete adjoint backward.
3. Assemble and solve the $$\theta(t) = (\operatorname{vec} W(t), b(t))^\top \in \mathbb R^n,\qquad n=d^2+d.$$4 gradient system.
4. Update the control parameters.
5. Every $$\theta(t) = (\operatorname{vec} W(t), b(t))^\top \in \mathbb R^n,\qquad n=d^2+d.$$5 iterations, compute $$\theta(t) = (\operatorname{vec} W(t), b(t))^\top \in \mathbb R^n,\qquad n=d^2+d.$$6, refine the interval with largest indicator, insert the new node, initialize the new parameters, and continue on the refined grid.

Stopping criteria include $$\theta(t) = (\operatorname{vec} W(t), b(t))^\top \in \mathbb R^n,\qquad n=d^2+d.$$7 or reaching $$\theta(t) = (\operatorname{vec} W(t), b(t))^\top \in \mathbb R^n,\qquad n=d^2+d.$$8, with safeguards such as minimal step size $$\theta(t) = (\operatorname{vec} W(t), b(t))^\top \in \mathbb R^n,\qquad n=d^2+d.$$9 and maximal depth $f(x,\theta)=\sigma(Wx+b),$0 (Hintermüller et al., 12 Jan 2026).

This interleaving of optimization and architectural modification is the defining algorithmic characteristic. The network is not trained on a static depth and then pruned or expanded afterward; instead, ansatz construction is endogenous to the optimization process.

5. Computational profile and empirical behavior

The per-iteration complexity is linear in the depth $f(x,\theta)=\sigma(Wx+b),$1 up to the cost of evaluating the neural vector field and its Jacobians. For a single forward/backward pass on one batch, the forward explicit Euler solve costs

$f(x,\theta)=\sigma(Wx+b),$2

with $f(x,\theta)=\sigma(Wx+b),$3 for dense $f(x,\theta)=\sigma(Wx+b),$4, and the adjoint solve has similar scaling. Gradient assembly is $f(x,\theta)=\sigma(Wx+b),$5, and solving the $f(x,\theta)=\sigma(Wx+b),$6 block-diagonal linear system induced by $f(x,\theta)=\sigma(Wx+b),$7 yields an overall $f(x,\theta)=\sigma(Wx+b),$8 contribution for that stage. The DWR estimator is $f(x,\theta)=\sigma(Wx+b),$9 plus sampling points on each interval for supremum-norm approximations. Memory is $\sigma$0 for states and adjoints and $\sigma$1 for the control nodal values (Hintermüller et al., 12 Jan 2026).

The paper reports results on binary classification and multiclass classification.

For the Swiss roll example, the settings are $\sigma$2, $\sigma$3, $\sigma$4, $\sigma$5, $\sigma$6, $\sigma$7, and refinement every $\sigma$8. The adaptive method achieved approximately $\sigma$9 accuracy with depths $m$0–$m$1 and iterations approximately $m$2–$m$3 across five seeds. Random insertion did not reach tolerance within $m$4, with accuracy $m$5–$m$6. A non-adaptive uniform-$m$7 network reached comparable accuracy, but iteration counts varied widely, for example between $m$8 and $m$9, indicating higher sensitivity to initialization. The adaptive grids concentrated more layers near the beginning of $\{t_k\}$00, with fewer near the end (Hintermüller et al., 12 Jan 2026).

For the Peaks multiclass problem, the settings are $\{t_k\}$01, $\{t_k\}$02, $\{t_k\}$03, $\{t_k\}$04, $\{t_k\}$05, $\{t_k\}$06, and $\{t_k\}$07. The adaptive method achieved approximately $\{t_k\}$08–$\{t_k\}$09 accuracy with depths $\{t_k\}$10–$\{t_k\}$11 and iterations approximately $\{t_k\}$12–$\{t_k\}$13. Random insertion reached similar accuracy but converged more slowly and often hit $\{t_k\}$14. Fixed uniform-$\{t_k\}$15 networks converged faster, approximately $\{t_k\}$16–$\{t_k\}$17 iterations, but began with large networks from the outset. The adaptive grids again placed more layers early in time (Hintermüller et al., 12 Jan 2026).

A plausible implication is that the method trades some per-iteration sophistication for depth economy during training: early stages operate on coarse architectures, while refinement is concentrated where the loss is most sensitive. The paper states this explicitly as a scalability advantage, since depth adaptivity starts from coarse $\{t_k\}$18 and grows as needed, reducing early-stage costs (Hintermüller et al., 12 Jan 2026).

6. Theoretical properties, limitations, and methodological position

The theoretical analysis assumes smooth activations, with $\{t_k\}$19 given as a typical example. Under $\{t_k\}$20 Lipschitz, existence and uniqueness hold for $\{t_k\}$21 when $\{t_k\}$22, and stronger $\{t_k\}$23 yields $\{t_k\}$24. The paper gives the stability bound

$\{t_k\}$25

The control-to-state map $\{t_k\}$26 is continuous; Fréchet derivatives exist; the derivative of the constraint map is surjective; and the adjoint exists and is unique. With terminal loss, the adjoint satisfies $\{t_k\}$27 and $\{t_k\}$28 (Hintermüller et al., 12 Jan 2026).

For the DWR analysis, the representation formula with residuals $\{t_k\}$29, $\{t_k\}$30, and $\{t_k\}$31 includes a cubic remainder under $\{t_k\}$32. The computable indicators are obtained by replacing unknown continuous errors with reconstructions and local jumps. The paper also notes that non-smooth activations such as ReLU can be approximated or smoothed for the theory (Hintermüller et al., 12 Jan 2026).

The method’s limitations are stated directly. Regularity assumptions are needed for the theoretical DWR bounds. Practical reconstructions approximate unknown continuous solutions. The gradient equation introduces a boundary-value solve, or its discrete linear system, which adds overhead. Architecture changes require re-running forward, adjoint, and gradient computations on the refined grid and reinitializing optimizer state for new variables (Hintermüller et al., 12 Jan 2026).

Within the paper’s own comparative framing, the algorithm differs from sensitivity-based layer insertion and adaptive successive approximation because refinement is derived from a dual-weighted residual estimator tied to the optimality system and directly targets the quantity of interest, the loss. The paper characterizes this as a goal-oriented architectural design guided by optimal control principles (Hintermüller et al., 12 Jan 2026).

This establishes the central meaning of “adaptive ansatz construction” in the neural ODE setting: the ansatz is the discretized architecture, and its refinement is governed not by heuristic depth schedules but by a layerwise error estimator derived from the continuous control problem.