---
title: Volterra Neural ODE (VNODE)
url: https://www.emergentmind.com/topics/volterra-neural-ordinary-differential-equations-vnode
type: topic
---

# Volterra Neural ODE (VNODE)

Searching arXiv for the cited VNODE paper and related foundational work to ground the article.
Volterra Neural Ordinary Differential Equations (VNODE) are a hybrid, continuous-time deep architecture that combines nonlinear Volterra filtering and Neural ODE–style continuous dynamics in a piecewise continuous manner for image classification. The architecture alternates discrete Volterra “event” layers with continuous ODE integration blocks, drawing inspiration from the visual cortex, where discrete event processing is interleaved with continuous integration [2509.24659]. In this formulation, Volterra filtering supplies multiplicative, higher-order feature interactions, while continuous-time evolution refines these features through ODE dynamics whose vector fields are themselves implemented using Volterra filters. The resulting model is described as a piecewise continuous Volterra Neural Network, or Volterra Neural ODE, and is reported to require substantially fewer parameters than conventional deep architectures while consistently outperforming state-of-the-art models on benchmark datasets such as CIFAR10 and ImageNet1K [2509.24659].

## 1. Formal definition and mathematical structure

VNODE is formulated by combining a truncated Volterra filter with continuous-depth dynamics. The Volterra series is introduced as a classical tool to represent nonlinear systems with memory. For a one-dimensional signal \(x(t)\), the output \(y(t)\) is approximated as

\[
y(t) = b + \sum_{k=1}^K \Bigg[ \sum_{\tau_1=0}^{L-1}\dots\sum_{\tau_k}^{L-1} \mathbf{W}^k(\tau_1,\dots,\tau_k) \prod_{j=1}^k x(t-\tau_j) \Bigg],
\]

where \(b\) is a bias term, \(K\) is the order of the Volterra series, \(L\) is the memory length, \(\mathbf{W}^k(\tau_1,\dots,\tau_k)\) is the \(k\)-th order Volterra kernel, and \(\prod_{j=1}^k x(t-\tau_j)\) is the multiplicative interaction of \(k\) delayed copies of the input [2509.24659]. For images, the paper gives a two-dimensional Volterra filter truncated to second order:

\[
\begin{aligned}
y_{\left[\substack{m \  n }\right]} = b &+ \sum_{k_1,k_2} \mathbf{W}^1_{\left[\substack{k_1 \ k_2}\right]} \, x_{\left[\substack{m-k_1 \ n-k_2}\right]} \\
&+ \sum_{\substack{k_1,k_2 \ l_1,l_2} \mathbf{W}^2 _{\left[\substack{k_1\ k_2}\right] \left[\substack{l_1 \ l_2}\right]} \bigg[ x_{\left[\substack{m-k_1 \ n-k_2}\right]} \cdot x_{\left[\substack{m-l_1 \ n-l_2}\right]} \bigg],
\end{aligned}
\]

with \(\mathbf{W}^1\) as a first-order linear kernel and \(\mathbf{W}^2\) modeling pairwise multiplicative interactions between pixels in its receptive field [2509.24659].

The continuous-time component follows the Neural ODE framework, where the hidden state satisfies

\[
\frac{d\mathbf{h}(t)}{dt} = \mathbf{f}(\mathbf{h}(t), t, \theta),
\]

and the output \(\mathbf{h}(T)\) is obtained by numerically solving the ODE from \(t=0\) to \(t=T\) [2509.24659; 1806.07366]. The relation to residual networks is explicit: the discrete update

\[
\mathbf{h}(t+1) = \mathbf{h}(t) + \mathbf{f}(\mathbf{h}(t), \theta_t)
\]

is an Euler discretization of the continuous flow [2509.24659].

In VNODE, time is normalized to \([0,1]\) and divided into event points

\[
t_0 = 0 < t_1 < t_2 < \dots < t_M = 1.
\]

At each stage \(m\), the model first applies a discrete Volterra feature extraction

\[
\mathbf{S}_m = \mathbf{V}_m(\mathbf{X}_{m-1}) = \sum_{k=1}^{K_m} \mathbf{V}_m^k (\mathbf{X}_{m-1}),
\]

and then evolves these features continuously according to

\[
\frac{d\mathbf{h}(t)}{dt} = \mathbf{g}_m(\mathbf{h}(t), t, \boldsymbol{\theta}_m), \quad t \in [t_{m-1}, t_m],
\]

with initial condition \(\mathbf{h}(t_{m-1})=\mathbf{S}_m\), and terminal feature

\[
\mathbf{X}_m = \mathbf{h}(t_m).
\]

Equivalently, the interval solution is written as

\[
\mathbf{X}_m = \mathbf{h}(t_m) = \mathbf{h}(t_{m-1}) + \int_{t_{m-1}}^{t_m} \mathbf{g}_m(\mathbf{h}(t), t; \boldsymbol{\theta}_m)\, dt,
\]

or abstractly as

\[
\mathbf{h}(t_m) = \text{ODESOLVE}\big( \mathbf{h}(t_{m-1}), \mathbf{g}_m, t_{m-1}, t_m; \boldsymbol{\theta}_m \big).
\]

The piecewise continuous character arises because \(\mathbf{h}(t)\) evolves continuously within each interval, but is reinitialized by a discrete Volterra step at each event time [2509.24659].

## 2. Architectural organization and computational mechanism

The overall architecture consists of an input image \(\mathbf{X}\) and a sequence of \(M\) stages, each containing a discrete Volterra block \(\mathbf{V}_m\), a continuous ODE block with Volterra right-hand side \(\mathbf{g}_m\), and a classifier head \(\mathbf{c}_m\) used during training [2509.24659]. The computation alternates as

\[
\mathbf{X}_0 = \mathbf{X} \to \mathbf{S}_1 = \mathbf{V}_1(\mathbf{X}_0) \to \mathbf{X}_1 = \text{ODESOLVE}(\mathbf{S}_1) \to \hat{\mathbf{y}}_1 \to \dots \to \hat{\mathbf{y}}_M.
\]

When \(M=1\), the model is described as a fully continuous VNN: a single discrete Volterra front-end followed by one long ODE interval. When \(M>1\), discrete and continuous processing are repeatedly interleaved [2509.24659].

The paper specifies two concrete instantiations. For CIFAR-10, inputs are \(32 \times 32\) RGB images, the model uses \(3 \times 3\) Volterra convolutional kernels, and the initial discrete Volterra filter expands channels from 3 to 64,

\[
\mathbf{X}_0 \in \mathbb{R}^{3 \times 32 \times 32}
\to
\mathbf{S}_1 \in \mathbb{R}^{64 \times 32 \times 32},
\]

with the channel dimension then kept constant at 64 across the network [2509.24659]. For ImageNet-1K, inputs are \(224 \times 224\) images, the initial discrete Volterra filter is a \(7 \times 7\) Volterra convolution kernel extracting 64 feature channels, and subsequent stages use multi-scale Volterra filters inspired by Inception, with parallel branches of sizes \(1 \times 1\), \(3 \times 3\), and \(5 \times 5\), whose outputs are concatenated. The ImageNet-1K configuration has six stages, \(M=6\), and progressively increases channels up to 1024 [2509.24659].

In implementation terms, the ODE solver is treated as a black box, consistent with Neural ODE methodology [1806.07366]. An off-the-shelf ODE solver, such as adaptive Runge–Kutta, numerically integrates the ODE over each interval \([t_{m-1}, t_m]\), and the number of function evaluations determines computational cost [2509.24659]. The vector field \(\mathbf{g}_m\) is implemented using a second-order Volterra filter. A specific modification distinguishes VNODE from standard time-conditioned Neural ODEs: rather than concatenating time \(t\) to \(\mathbf{h}(t)\), the model uses element-wise addition, so the input to \(\mathbf{g}_m\) is \(\mathbf{h}(t)+t\). The stated purpose is to allow grouped Volterra convolutions because the tensor remains the same shape, thereby reducing parameter count and FLOPs [2509.24659].

Direct second-order Volterra filters are noted to be expensive, so the architecture uses a lossy approximation of the Volterra filter, following MR-VNet, through structured kernels and sharing to keep costs manageable [2509.24659]. The discrete Volterra blocks and classifier heads are standard differentiable layers, and the ODE solver is wrapped as a differentiable operator with custom adjoint gradients [2509.24659].

## 3. Relation to Neural ODEs and other deep architectures

Relative to standard Neural ODEs, VNODE differs in three stated ways. First, the vector field is a Volterra filter rather than a small CNN or MLP, so the dynamics directly model polynomial or multiplicative interactions in the hidden state. Second, the architecture is piecewise discrete-continuous rather than a single continuous flow from \(t_0\) to \(T\). Third, time is injected through \(\mathbf{h}(t)+t\) rather than by concatenating \(t\) as an extra channel or dimension [2509.24659]. The foundational Neural ODE formulation treats the solver as a black-box differential equation solver, parameterizes hidden-state derivatives using a neural network, and supports scalable backpropagation through any ODE solver without access to its internal operations [1806.07366]. VNODE adopts that continuous-depth infrastructure while altering the form of the vector field and the stagewise organization.

The paper contrasts VNODE with conventional deep architectures such as ResNets, CNNs, and Transformers by emphasizing higher-order convolutions via Volterra filters and continuous depth via ODEs. Instead of stacking many linear convolutions and pointwise nonlinearities, VNODE uses fewer highly expressive Volterra filters capturing polynomial interactions in each layer. The ODE blocks are described as “infinitely thin” residual layers integrated over time, reusing the same Volterra vector field throughout the interval [2509.24659]. This suggests a view of VNODE as combining explicit higher-order local structure with parameter sharing across continuous depth.

For ImageNet-1K, the paper reports the following comparisons: VNODE \((M=6)\) has 9.1M parameters, 2.4 GFLOPs, and 83.5% top-1 accuracy; ResNet-152 has 60.2M parameters, 11.51 GFLOPs, and 78.31%; ConvNeXt-T has 29M parameters, 4.5 GFLOPs, and 82.1% [2509.24659]. The stated interpretation is that VNODE reaches or surpasses the accuracy of heavy CNNs and modern ConvNeXt variants with significantly fewer parameters and FLOPs.

The design is also explicitly connected to a proposed pattern of brain processing: discrete events, modeled by discrete Volterra layers, and continuous integration, modeled by continuous-time ODE evolution. On that basis, VNODE is described as a biologically inspired hybrid system rather than a purely discrete feed-forward network [2509.24659]. A plausible implication is that the piecewise architecture is intended not only as a computational device but also as a structural analogy for mixed event-driven and continuous integration regimes.

## 4. Optimization, supervision, and computational complexity

At each stage \(m\), the model applies a classifier to the stage output \(\mathbf{X}_m\),

\[
\hat{\mathbf{y}}_m = \mathbf{c}_m(\mathbf{X}_m) = \sigma \Big( \mathbf{W}_m \, \text{flatten}(\mathbf{X}_m) \Big) + b_m,
\]

where \(\sigma\) is softmax, and the final prediction is \(\hat{\mathbf{y}}_M\), although intermediate classifiers also contribute to training [2509.24659]. The stagewise loss is a cross-entropy between \(\hat{\mathbf{y}}_m\) and the ground truth \(\mathbf{y}_{gt}\),

\[
\mathcal{L}_m(\hat{\mathbf{y}}_m, \mathbf{y}_{gt}) = \sum_i -y_{gt,i} \log(\hat{y}_{m,i}),
\]

and the overall cost function is written as

\[
\min_{\boldsymbol{\Theta}, \mathbf{W}, \mathbf{V}} \sum_{m=1}^M \mathcal{L}_m(\hat{\mathbf{y}}_m, \mathbf{y}_{gt}; \boldsymbol{\theta}_m, \mathbf{W}_m, \mathbf{V}_m).
\]

The paper characterizes the intermediate losses as deep supervision, guiding each stage to learn useful features and improving the optimization landscape [2509.24659].

Backpropagation through the ODE solver uses the adjoint sensitivity method, as in the original Neural ODE work [1806.07366]. The ODE solver is treated as a black box; adjoint variables satisfy another ODE backward in time; and gradients with respect to parameters and initial states are computed by solving this adjoint ODE [2509.24659]. The reported benefits are memory usage \(\mathcal{O}(1)\) with respect to the number of solver steps, complexity \(\mathcal{O}(L)\) where \(L\) is the number of function evaluations, and explicit control over numerical error through solver tolerances [2509.24659]. These statements align with the Neural ODE framework, which presents constant memory cost, adaptive evaluation strategies, and an explicit precision-speed trade-off [1806.07366].

The paper does not describe special regularizers beyond cross-entropy, but identifies several design choices that contribute to stability: second-order, truncated Volterra filters; deep supervision via stage-wise classifiers; grouped Volterra filters in \(\mathbf{g}_m\); and element-wise addition of time instead of concatenation [2509.24659]. These are presented not as separate optimization algorithms but as architectural constraints that control complexity and stabilize training.

The ImageNet-1K parameter and FLOP comparisons stated in the paper are summarized below.

| Model | Params / GFLOPs | Top-1 |
|---|---:|---:|
| VGG-16 | 138M / 15.4 | 75.2% |
| ResNet-50 | 25.6M / 4.09 | 76.1% |
| ResNet-152 | 60.2M / 11.51 | 78.31% |
| ConvNeXt-T | 29M / 4.5 | 82.1% |
| E-ConvNeXt-Tiny | 13.2M / 2.04 | 80.6% |
| E-ConvNeXt-Small | 19.4M / 3.12 | 81.9% |
| TinyViT | 21M / 4.4 | 83.1% |
| MobileViT-XS | 2.3M / 0.7 | 74.8% |
| NODE | 0.7M / 0.3 | 68.3% |
| MALI | 11.2M / not reported | 70.17% |
| Vanilla VNN | 12M / 3.6 | 83.3% |
| VNODE (M=1) | 1.5M / 0.4 | 78.6% |
| VNODE (M=3) | 4.5M / 1.2 | 81.1% |
| VNODE (M=6) | 9.1M / 2.4 | 83.5% |

The accompanying observations in the source are that fully continuous VNODE \((M=1)\) already outperforms standalone NODEs by approximately 10 percentage points with about twice the FLOPs and parameters, and that increasing \(M\) improves accuracy while retaining lower parameter and FLOP budgets than a number of heavier baselines [2509.24659].

## 5. Empirical behavior on CIFAR-10 and ImageNet-1K

The paper evaluates VNODE on CIFAR-10 and ImageNet-1K using top-1 classification accuracy as the evaluation metric [2509.24659]. For CIFAR-10, the architecture uses \(3 \times 3\) Volterra kernels, a single channel expansion to 64, and then fixed channels. The description states that experiments compare against state-of-the-art CNN and Transformer baselines and that VNODE is superior and more efficient, although the exact table is not reproduced in the supplied details [2509.24659]. For ImageNet-1K, the model uses Inception-inspired multi-scale Volterra filters with \(1 \times 1\), \(3 \times 3\), and \(5 \times 5\) kernels, six stages, and channels increased to 1024 [2509.24659].

The central quantitative ImageNet-1K result is that VNODE \((M=6)\) achieves 83.5% top-1 accuracy with 9.1M parameters and 2.4 GFLOPs [2509.24659]. The source further states that this exceeds ResNet-152 at approximately one-sixth of the parameters and one-fifth of the FLOPs, exceeds ConvNeXt-T with about one-third of the parameters and one-half of the FLOPs, and performs slightly better than Vanilla VNN while being more parameter-efficient [2509.24659].

Ablation through the settings \(M=1\), \(M=3\), and \(M=6\) is used to compare fully continuous and piecewise formulations. The reported pattern is that increasing the number of stages materially improves performance, indicating benefits from the piecewise structure relative to a single long continuous block [2509.24659]. The comparison between Vanilla VNN and VNODE \((M=6)\) is used to argue that adding continuous dynamics via ODE blocks improves both accuracy and efficiency [2509.24659].

The qualitative analysis includes Grad-CAM visualization. In comparisons with ResNet-152, VNODE is reported to focus more on the object of interest and reduce attention on irrelevant background regions [2509.24659]. This is interpreted in the source as suggesting that higher-order and continuous dynamics help the model learn more object-centric features. The robustness analysis evaluates CIFAR-10C, comprising 15 corruption types and 5 severity levels, for a total of 75 scenarios. The source says the corresponding table shows “impressive robustness” without extra augmentations or fine-tuning, and characterizes the overall takeaway as better generalization under distribution shifts or corruptions than many baselines [2509.24659]. Because the specific robustness numbers are not supplied, no stronger quantitative statement is warranted.

## 6. Expressive rationale, constraints, and open questions

The paper’s mechanistic intuition is that Volterra filters provide rich local interactions by explicitly computing products of input components, thereby yielding a polynomial mapping that can approximate a broad class of nonlinear functions with fewer layers [2509.24659]. ODE blocks then act as continuous refinement flows: once higher-order interactions are extracted, the same Volterra-based vector field is applied at many infinitesimal steps, producing a deep but parameter-sharing transformation [2509.24659]. Repeated alternation between discrete Volterra steps and continuous Volterra ODE evolution is presented as enabling approximation of complex hierarchical transformations with comparatively few parameters [2509.24659].

The dependencies captured by Volterra terms are described in terms of multiplicative interactions among pixels or features, such as second-order products \(x_i x_j\) within a neighborhood, polynomial structure and higher-order statistics, and complex local patterns including texture patterns, edge intersections, or complex shape cues [2509.24659]. The comparison point is that typical CNN interactions are mostly additive before a pointwise nonlinearity, whereas Volterra convolutions explicitly represent products of components [2509.24659]. This suggests that VNODE is intended to shift some representational burden from depth to higher-order local operators.

The paper also identifies several limitations. Even with approximations, second-order Volterra filters are more complex to design and optimize than plain convolutional layers, and efficient GPU implementations are non-trivial [2509.24659]. Runtime remains tied to the number of function evaluations and can be sensitive to solver choice and step-size control, as in Neural ODEs more generally [2509.24659; 1806.07366]. The piecewise architecture introduces additional hyperparameters, including the number of stages \(M\) and the time partition, and scaling to larger datasets, higher resolutions, or other modalities is stated to require further exploration [2509.24659]. Although the model is described as biologically inspired, the learned continuous dynamics are still difficult to interpret [2509.24659].

Within the broader continuous-time literature, Neural ODEs supply the conceptual and algorithmic groundwork: continuous-depth hidden-state evolution, black-box integration, adaptive evaluation, and adjoint-based gradient computation [1806.07366]. VNODE remains within that paradigm but replaces standard vector fields with structured Volterra operators and inserts discrete event-style feature extraction between ODE intervals [2509.24659]. In the broader Volterra network context, previous VNNs are noted as stacked Volterra convolutional networks with residual connections applied to action recognition, restoration, and noise cancellation, while VNODE extends that line into the continuous-time regime [2509.24659]. The future directions identified in the source include sequence modeling and time series, video and spatiotemporal modeling, control systems and physical modeling, audio and speech, sensor data, and architectural refinements such as adaptive event points, learnable event times, and combinations with attention mechanisms or transformers [2509.24659].

Source: https://www.emergentmind.com/topics/volterra-neural-ordinary-differential-equations-vnode