---
title: Adaptive Ansatz Construction Algorithm
url: https://www.emergentmind.com/topics/adaptive-ansatz-construction-algorithm
type: topic
---

# Adaptive Ansatz Construction Algorithm

Searching arXiv for the specified paper and closely related work on adaptive ansatz construction for neural ODEs.
The Adaptive Ansatz Construction Algorithm is a layerwise adaptive construction method for neural network architectures derived from the optimal control of neural differential equations. In the formulation introduced in “Layerwise goal-oriented adaptivity for neural ODEs: an optimal control perspective” [2601.07397], the network architecture is identified with the discrete ansatz of a neural ODE, namely the depth $K$, the time grid $\{t_k\}$, and the piecewise control parameterization $\theta_\tau \in CG(1)$. The method combines a goal-oriented dual-weighted residual technique, a $DG(0)$ Galerkin discretization leading to an explicit Euler time marching scheme, and gradient-based optimization to insert layers where the loss is most sensitive [2601.07397].

## 1. Continuous model and optimal-control interpretation

The algorithm starts from a continuous-time neural ODE in which the trainable coefficients are treated as controls. Let $W(t)\in\mathbb R^{d\times d}$ and $b(t)\in\mathbb R^d$ be time-dependent weights and biases collected in
$$
\theta(t) = (\operatorname{vec} W(t), b(t))^\top \in \mathbb R^n,\qquad n=d^2+d.
$$
The neural vector field is
$$
f(x,\theta)=\sigma(Wx+b),
$$
with componentwise activation $\sigma$. For a batch of $m$ samples, the state dynamics on $I=[0,T]$ are
$$
\dot x(t)=F(x(t),\theta(t)),\qquad x(0)=x_{\mathrm{in}},
$$
where
$$
F(x,\theta)=\big(f(x^1,\theta),\ldots,f(x^m,\theta)\big)^\top.
$$

The controls $u(t)$ are the coefficients $\theta(t)$ themselves. The regularizer enforces $\theta\in H^1(I;\mathbb R^n)$, and the learning objective depends on the terminal state:
$$
J(x)=l(x(T)).
$$
The paper uses the regularization
$$
R(\theta)=\frac{\lambda}{2}\int_0^T \big(\|\theta(t)\|^2+\|\dot\theta(t)\|^2\big)\,dt,\qquad \lambda>0,
$$
leading to the ODE-constrained optimization problem
$$
\min_{\theta\in H^1(I)} \mathcal J(x,\theta):=J(x)+R(\theta)
$$
subject to
$$
\dot x(t)=F(x(t),\theta(t)),\qquad x(0)=x_{\mathrm{in}}.
$$
The associated control-to-state map is denoted by $S:\theta\mapsto x$ [2601.07397].

This formulation makes the architecture itself part of the control problem. The discrete ansatz is therefore not fixed in advance; it is refined according to an error representation tied directly to the quantity of interest, namely the loss $J$.

## 2. Discrete ansatz, time stepping, and gradient equation

The temporal discretization defines the network architecture. On a grid
$$
0=t_0<t_1<\cdots<t_K=T,\qquad h_k=t_k-t_{k-1},
$$
the state trial space uses $DG(0)$, so $x_\tau$ is piecewise constant on $[t_{k-1},t_k)$. With midpoint evaluation
$$
\theta^{k-1/2}_\tau=\theta_\tau(m_k),\qquad m_k=\frac{t_{k-1}+t_k}{2},
$$
the forward solve becomes the explicit Euler update
$$
x_\tau^0=x_{\mathrm{in}},\qquad
x_\tau^k=x_\tau^{k-1}+h_k F(x_\tau^{k-1},\theta_\tau^{k-1/2}),\quad k=1,\ldots,K.
$$

The adjoint is discretized by a backward Euler-like recurrence:
$$
p_\tau^K=l'(x_\tau^K),\qquad
p_\tau^{k-1}=p_\tau^k+h_k D_1F(x_\tau^{k-1},\theta_\tau^{k-1/2})^\top p_\tau^k,\quad k=1,\ldots,K.
$$
At the continuous level, the adjoint satisfies
$$
-\dot p=D_1F(x,\theta)^\top p,\qquad p(T)=l'(x(T)).
$$

A distinctive feature of the method is that the $H^1$-regularizer induces a boundary-value problem for the reduced gradient $g(t)\in H^1(I;\mathbb R^n)$:
$$
g-\ddot g=\lambda(\theta-\ddot\theta)+D_2F(x,\theta)^\top p,\qquad \dot g(0)=\dot g(T)=0.
$$
Its $CG(1)$ discretization yields the linear system
$$
(B^\tau\otimes I_n)g_\tau=\lambda(B^\tau\otimes I_n)\Theta_\tau+(M^\tau\otimes I_n)z^{p_\tau},
$$
with mass matrix $M^\tau$, stiffness matrix $A^\tau$, and $B^\tau=A^\tau+M^\tau$ [2601.07397].

The algorithm therefore alternates among three coupled solves on the current grid: forward state evolution, backward adjoint propagation, and a gradient reconstruction in the $H^1$ geometry. In the paper’s terminology, the architecture is the discrete ansatz for the continuous neural ODE, and the refinement of this ansatz is driven by the same optimality system that governs training.

## 3. Dual-weighted residual estimator and layerwise indicators

The adaptive criterion is derived from a goal-oriented dual-weighted residual representation. For solutions $(x^*,\theta^*,p^*)$ of the continuous optimality system and $(x_\tau^*,\theta_\tau^*,p_\tau^*)$ of the discrete one,
$$
\mathcal J(x^*,\theta^*)-\mathcal J(x_\tau^*,\theta_\tau^*)
=\frac12\rho_x+\frac12\rho_\theta+\frac12\rho_p+R,
$$
where $R$ is a higher-order cubic remainder under $\sigma\in W^{3,\infty}$ [2601.07397].

The resulting layerwise bound is
$$
|\Delta\mathcal J|
\le \frac12|\rho_x|+\frac12|\rho_\theta|+\frac12|\rho_p|+|R|,
$$
with decompositions
$$
|\rho_x|\le \sum_k R^k_{p_\tau^*}\,\omega^k_{x^*},\qquad
|\rho_p|\le \sum_k R^k_{x_\tau^*}\,\omega^k_{p^*},\qquad
|\rho_\theta|\le \sum_k |\rho^k_{\theta^*,\theta_\tau^*}|.
$$
The dual weights are the local state and adjoint errors,
$$
\omega^k_{x^*}=\|x^*-\tilde x_\tau\|_{I_k}\approx \|x_\tau^k-x_\tau^{k-1}\|,
\qquad
\omega^k_{p^*}=\|p^*-\tilde p_\tau\|_{I_k}\approx \|p_\tau^k-p_\tau^{k-1}\|.
$$
The paper emphasizes that these dual weights amplify residuals where the quantity of interest, the loss, is most sensitive [2601.07397].

The $DG(0)$ formulation yields layerwise residuals containing both volume and jump contributions. For the state,
$$
R^k_{x_\tau}\approx h_k\|F(x_\tau,\theta_\tau)\|_{I_k}
+\frac{h_k}{h_k+h_{k+1}}\|x_\tau^k-x_\tau^{k-1}\|
+\mathbf 1_{2\le k\le K-1}\frac{h_k}{h_k+h_{k-1}}\|x_\tau^{k-1}-x_\tau^{k-2}\|,
$$
and for the adjoint,
$$
R^k_{p_\tau}\approx h_k\|D_1F(x_\tau,\theta_\tau)^\top p_\tau\|_{I_k}
+\frac{h_k}{h_{k-1}+h_k}\|p_\tau^k-p_\tau^{k-1}\|
+\mathbf 1_{1\le k\le K-1}\frac{h_k}{h_k+h_{k+1}}\|p_\tau^{k+1}-p_\tau^k\|.
$$

The computable layer indicator is
$$
\eta_k:=R^k_{p_\tau}\,\omega^k_{x_\tau}+\rho^k_{\vartheta_\tau,\theta_\tau}+R^k_{x_\tau}\,\omega^k_{p_\tau},
$$
where $\vartheta_\tau$ is a piecewise-quadratic reconstruction of $\theta$ [2601.07397]. This is the central criterion used to decide where new layers are inserted.

## 4. Layer insertion, refinement strategy, and optimization loop

The adaptive procedure is explicitly layerwise. On each refinement event, the algorithm marks the interval with largest indicator,
$$
k^*=\arg\max_k \eta_k,
$$
and splits that interval by inserting
$$
t_{\mathrm{new}}=\frac{t_{k^*-1}+t_{k^*}}{2}.
$$
This increases the depth from $K$ to $K+1$. The newly inserted layer parameters are initialized by averaging neighboring parameters,
$$
\theta_{\mathrm{new}}=\frac{\theta_\tau^{k^*-1}+\theta_\tau^{k^*}}{2},
$$
which, according to the paper, leverages the $H^1$ regularization to maintain smoothness [2601.07397]. The local step size is halved on the refined interval.

The parameter update itself is a steepest descent step in the $H^1$ metric,
$$
\theta_\tau\leftarrow \theta_\tau-\alpha g_\tau,
$$
where $g_\tau$ solves the linear system above. The dual-weighted control term is assembled from the reconstructed adjoint:
$$
z^{p_\tau}(t)=D_2F(x_\tau(t),\theta_\tau(t))^\top \hat p_\tau(t).
$$
The paper states that backtracking or fixed schedules may be used for $\alpha$, damping may be used when the loss increases, and convergence can be monitored through loss decrease, gradient norm, and validation accuracy. Although the formal optimization description uses steepest descent, the implementation also uses Adam in practice, with optimizer moments reset or adjusted when layers are inserted [2601.07397].

A concise rendering of the algorithmic loop is as follows.

1. Solve the forward problem by explicit Euler on the current grid.
2. Compute the terminal loss derivative and propagate the discrete adjoint backward.
3. Assemble and solve the $CG(1)$ gradient system.
4. Update the control parameters.
5. Every $it_{\mathrm{up}}$ iterations, compute $\eta_k$, refine the interval with largest indicator, insert the new node, initialize the new parameters, and continue on the refined grid.

Stopping criteria include $J\le \mathrm{tol}$ or reaching $\mathrm{it}_{\max}$, with safeguards such as minimal step size $h_{\min}$ and maximal depth $K_{\max}$ [2601.07397].

This interleaving of optimization and architectural modification is the defining algorithmic characteristic. The network is not trained on a static depth and then pruned or expanded afterward; instead, ansatz construction is endogenous to the optimization process.

## 5. Computational profile and empirical behavior

The per-iteration complexity is linear in the depth $K$ up to the cost of evaluating the neural vector field and its Jacobians. For a single forward/backward pass on one batch, the forward explicit Euler solve costs
$$
O(K\cdot \mathrm{cost}(F)),
$$
with $\mathrm{cost}(F)\approx O(md\,d^2)$ for dense $W$, and the adjoint solve has similar scaling. Gradient assembly is $O(K\cdot n)$, and solving the $(K+1)\times (K+1)$ block-diagonal linear system induced by $(B^\tau\otimes I_n)$ yields an overall $O(K\cdot n)$ contribution for that stage. The DWR estimator is $O(K)$ plus sampling points on each interval for supremum-norm approximations. Memory is $O(K\cdot md)$ for states and adjoints and $O((K+1)\cdot n)$ for the control nodal values [2601.07397].

The paper reports results on binary classification and multiclass classification.

For the Swiss roll example, the settings are $d=4$, $T=20$, $\lambda=10^{-3}$, $\sigma=\tanh$, $\mathrm{tol}=0.025$, $\mathrm{lr}=5\cdot 10^{-3}$, and refinement every $it_{\mathrm{up}}=50$. The adaptive method achieved approximately $0.99$ accuracy with depths $K\approx 39$–$52$ and iterations approximately $1794$–$2447$ across five seeds. Random insertion did not reach tolerance within $it_{\max}=3000$, with accuracy $0.74$–$0.94$. A non-adaptive uniform-$K$ network reached comparable accuracy, but iteration counts varied widely, for example between $317$ and $1978$, indicating higher sensitivity to initialization. The adaptive grids concentrated more layers near the beginning of $[0,T]$, with fewer near the end [2601.07397].

For the Peaks multiclass problem, the settings are $d=20$, $T=10$, $\lambda=10^{-3}$, $\sigma=\tanh$, $\mathrm{tol}=0.05$, $\mathrm{lr}=10^{-3}$, and $it_{\mathrm{up}}=75$. The adaptive method achieved approximately $0.96$–$0.98$ accuracy with depths $K\approx 24$–$27$ and iterations approximately $1622$–$1865$. Random insertion reached similar accuracy but converged more slowly and often hit $it_{\max}$. Fixed uniform-$K$ networks converged faster, approximately $775$–$1021$ iterations, but began with large networks from the outset. The adaptive grids again placed more layers early in time [2601.07397].

A plausible implication is that the method trades some per-iteration sophistication for depth economy during training: early stages operate on coarse architectures, while refinement is concentrated where the loss is most sensitive. The paper states this explicitly as a scalability advantage, since depth adaptivity starts from coarse $K$ and grows as needed, reducing early-stage costs [2601.07397].

## 6. Theoretical properties, limitations, and methodological position

The theoretical analysis assumes smooth activations, with $\tanh$ given as a typical example. Under $\sigma$ Lipschitz, existence and uniqueness hold for $x\in H^1(I)$ when $\theta\in L^2(I)$, and stronger $\theta\in H^1(I)$ yields $x\in C^1([0,T])$. The paper gives the stability bound
$$
\|x\|_{H^1}\le C_1 \exp(C_2\|\theta\|_{L^2}).
$$
The control-to-state map $S:L^2(I)\to H^1(I)$ is continuous; Fréchet derivatives exist; the derivative of the constraint map is surjective; and the adjoint exists and is unique. With terminal loss, the adjoint satisfies $p\in H^1(I)$ and $p(T)=l'(x(T))$ [2601.07397].

For the DWR analysis, the representation formula with residuals $\rho_x$, $\rho_\theta$, and $\rho_p$ includes a cubic remainder under $\sigma\in W^{3,\infty}$. The computable indicators are obtained by replacing unknown continuous errors with reconstructions and local jumps. The paper also notes that non-smooth activations such as ReLU can be approximated or smoothed for the theory [2601.07397].

The method’s limitations are stated directly. Regularity assumptions are needed for the theoretical DWR bounds. Practical reconstructions approximate unknown continuous solutions. The gradient equation introduces a boundary-value solve, or its discrete linear system, which adds overhead. Architecture changes require re-running forward, adjoint, and gradient computations on the refined grid and reinitializing optimizer state for new variables [2601.07397].

Within the paper’s own comparative framing, the algorithm differs from sensitivity-based layer insertion and adaptive successive approximation because refinement is derived from a dual-weighted residual estimator tied to the optimality system and directly targets the quantity of interest, the loss. The paper characterizes this as a goal-oriented architectural design guided by optimal control principles [2601.07397].

This establishes the central meaning of “adaptive ansatz construction” in the neural ODE setting: the ansatz is the discretized architecture, and its refinement is governed not by heuristic depth schedules but by a layerwise error estimator derived from the continuous control problem.

Source: https://www.emergentmind.com/topics/adaptive-ansatz-construction-algorithm