Volterra Neural ODE (VNODE)

• VNODE is a hybrid continuous-time architecture that interleaves discrete Volterra filtering with Neural ODE evolution to capture higher-order feature interactions.
• It employs truncated Volterra filters for multiplicative, higher-order interactions and adaptive ODE solvers to refine features across piecewise intervals.
• Empirical results on CIFAR-10 and ImageNet-1K show VNODE achieves state-of-the-art accuracy with significantly fewer parameters and FLOPs compared to conventional models.

Searching arXiv for the cited VNODE paper and related foundational work to ground the article. Volterra Neural Ordinary Differential Equations (VNODE) are a hybrid, continuous-time deep architecture that combines nonlinear Volterra filtering and Neural ODE–style continuous dynamics in a piecewise continuous manner for image classification. The architecture alternates discrete Volterra “event” layers with continuous ODE integration blocks, drawing inspiration from the visual cortex, where discrete event processing is interleaved with continuous integration (Roheda et al., 29 Sep 2025). In this formulation, Volterra filtering supplies multiplicative, higher-order feature interactions, while continuous-time evolution refines these features through ODE dynamics whose vector fields are themselves implemented using Volterra filters. The resulting model is described as a piecewise continuous Volterra Neural Network, or Volterra Neural ODE, and is reported to require substantially fewer parameters than conventional deep architectures while consistently outperforming state-of-the-art models on benchmark datasets such as CIFAR10 and ImageNet1K (Roheda et al., 29 Sep 2025).

1. Formal definition and mathematical structure

VNODE is formulated by combining a truncated Volterra filter with continuous-depth dynamics. The Volterra series is introduced as a classical tool to represent nonlinear systems with memory. For a one-dimensional signal $x(t)$, the output $y(t)$ is approximated as

$$y(t) = b + \sum_{k=1}^K \Bigg[ \sum_{\tau_1=0}^{L-1}\dots\sum_{\tau_k}^{L-1} \mathbf{W}^k(\tau_1,\dots,\tau_k) \prod_{j=1}^k x(t-\tau_j) \Bigg],$$

where $b$ is a bias term, $K$ is the order of the Volterra series, $L$ is the memory length, $\mathbf{W}^k(\tau_1,\dots,\tau_k)$ is the $k$-th order Volterra kernel, and $\prod_{j=1}^k x(t-\tau_j)$ is the multiplicative interaction of $k$ delayed copies of the input (Roheda et al., 29 Sep 2025). For images, the paper gives a two-dimensional Volterra filter truncated to second order:

$y(t)$0

with $y(t)$1 as a first-order linear kernel and $y(t)$2 modeling pairwise multiplicative interactions between pixels in its receptive field (Roheda et al., 29 Sep 2025).

The continuous-time component follows the Neural ODE framework, where the hidden state satisfies

$y(t)$3

and the output $y(t)$4 is obtained by numerically solving the ODE from $y(t)$5 to $y(t)$6 (Roheda et al., 29 Sep 2025, Chen et al., 2018). The relation to residual networks is explicit: the discrete update

$y(t)$7

is an Euler discretization of the continuous flow (Roheda et al., 29 Sep 2025).

In VNODE, time is normalized to $y(t)$8 and divided into event points

$y(t)$9

At each stage $$y(t) = b + \sum_{k=1}^K \Bigg[ \sum_{\tau_1=0}^{L-1}\dots\sum_{\tau_k}^{L-1} \mathbf{W}^k(\tau_1,\dots,\tau_k) \prod_{j=1}^k x(t-\tau_j) \Bigg],$$0, the model first applies a discrete Volterra feature extraction

$$y(t) = b + \sum_{k=1}^K \Bigg[ \sum_{\tau_1=0}^{L-1}\dots\sum_{\tau_k}^{L-1} \mathbf{W}^k(\tau_1,\dots,\tau_k) \prod_{j=1}^k x(t-\tau_j) \Bigg],$$1

and then evolves these features continuously according to

$$y(t) = b + \sum_{k=1}^K \Bigg[ \sum_{\tau_1=0}^{L-1}\dots\sum_{\tau_k}^{L-1} \mathbf{W}^k(\tau_1,\dots,\tau_k) \prod_{j=1}^k x(t-\tau_j) \Bigg],$$2

with initial condition $$y(t) = b + \sum_{k=1}^K \Bigg[ \sum_{\tau_1=0}^{L-1}\dots\sum_{\tau_k}^{L-1} \mathbf{W}^k(\tau_1,\dots,\tau_k) \prod_{j=1}^k x(t-\tau_j) \Bigg],$$3, and terminal feature

$$y(t) = b + \sum_{k=1}^K \Bigg[ \sum_{\tau_1=0}^{L-1}\dots\sum_{\tau_k}^{L-1} \mathbf{W}^k(\tau_1,\dots,\tau_k) \prod_{j=1}^k x(t-\tau_j) \Bigg],$$4

Equivalently, the interval solution is written as

$$y(t) = b + \sum_{k=1}^K \Bigg[ \sum_{\tau_1=0}^{L-1}\dots\sum_{\tau_k}^{L-1} \mathbf{W}^k(\tau_1,\dots,\tau_k) \prod_{j=1}^k x(t-\tau_j) \Bigg],$$5

or abstractly as

$$y(t) = b + \sum_{k=1}^K \Bigg[ \sum_{\tau_1=0}^{L-1}\dots\sum_{\tau_k}^{L-1} \mathbf{W}^k(\tau_1,\dots,\tau_k) \prod_{j=1}^k x(t-\tau_j) \Bigg],$$6

The piecewise continuous character arises because $$y(t) = b + \sum_{k=1}^K \Bigg[ \sum_{\tau_1=0}^{L-1}\dots\sum_{\tau_k}^{L-1} \mathbf{W}^k(\tau_1,\dots,\tau_k) \prod_{j=1}^k x(t-\tau_j) \Bigg],$$7 evolves continuously within each interval, but is reinitialized by a discrete Volterra step at each event time (Roheda et al., 29 Sep 2025).

2. Architectural organization and computational mechanism

The overall architecture consists of an input image $$y(t) = b + \sum_{k=1}^K \Bigg[ \sum_{\tau_1=0}^{L-1}\dots\sum_{\tau_k}^{L-1} \mathbf{W}^k(\tau_1,\dots,\tau_k) \prod_{j=1}^k x(t-\tau_j) \Bigg],$$8 and a sequence of $$y(t) = b + \sum_{k=1}^K \Bigg[ \sum_{\tau_1=0}^{L-1}\dots\sum_{\tau_k}^{L-1} \mathbf{W}^k(\tau_1,\dots,\tau_k) \prod_{j=1}^k x(t-\tau_j) \Bigg],$$9 stages, each containing a discrete Volterra block $b$0, a continuous ODE block with Volterra right-hand side $b$1, and a classifier head $b$2 used during training (Roheda et al., 29 Sep 2025). The computation alternates as

$b$3

When $b$4, the model is described as a fully continuous VNN: a single discrete Volterra front-end followed by one long ODE interval. When $b$5, discrete and continuous processing are repeatedly interleaved (Roheda et al., 29 Sep 2025).

The paper specifies two concrete instantiations. For CIFAR-10, inputs are $b$6 RGB images, the model uses $b$7 Volterra convolutional kernels, and the initial discrete Volterra filter expands channels from 3 to 64,

$b$8

with the channel dimension then kept constant at 64 across the network (Roheda et al., 29 Sep 2025). For ImageNet-1K, inputs are $b$9 images, the initial discrete Volterra filter is a $K$0 Volterra convolution kernel extracting 64 feature channels, and subsequent stages use multi-scale Volterra filters inspired by Inception, with parallel branches of sizes $K$1, $K$2, and $K$3, whose outputs are concatenated. The ImageNet-1K configuration has six stages, $K$4, and progressively increases channels up to 1024 (Roheda et al., 29 Sep 2025).

In implementation terms, the ODE solver is treated as a black box, consistent with Neural ODE methodology (Chen et al., 2018). An off-the-shelf ODE solver, such as adaptive Runge–Kutta, numerically integrates the ODE over each interval $K$5, and the number of function evaluations determines computational cost (Roheda et al., 29 Sep 2025). The vector field $K$6 is implemented using a second-order Volterra filter. A specific modification distinguishes VNODE from standard time-conditioned Neural ODEs: rather than concatenating time $K$7 to $K$8, the model uses element-wise addition, so the input to $K$9 is $L$0. The stated purpose is to allow grouped Volterra convolutions because the tensor remains the same shape, thereby reducing parameter count and FLOPs (Roheda et al., 29 Sep 2025).

Direct second-order Volterra filters are noted to be expensive, so the architecture uses a lossy approximation of the Volterra filter, following MR-VNet, through structured kernels and sharing to keep costs manageable (Roheda et al., 29 Sep 2025). The discrete Volterra blocks and classifier heads are standard differentiable layers, and the ODE solver is wrapped as a differentiable operator with custom adjoint gradients (Roheda et al., 29 Sep 2025).

3. Relation to Neural ODEs and other deep architectures

Relative to standard Neural ODEs, VNODE differs in three stated ways. First, the vector field is a Volterra filter rather than a small CNN or MLP, so the dynamics directly model polynomial or multiplicative interactions in the hidden state. Second, the architecture is piecewise discrete-continuous rather than a single continuous flow from $L$1 to $L$2. Third, time is injected through $L$3 rather than by concatenating $L$4 as an extra channel or dimension (Roheda et al., 29 Sep 2025). The foundational Neural ODE formulation treats the solver as a black-box differential equation solver, parameterizes hidden-state derivatives using a neural network, and supports scalable backpropagation through any ODE solver without access to its internal operations (Chen et al., 2018). VNODE adopts that continuous-depth infrastructure while altering the form of the vector field and the stagewise organization.

The paper contrasts VNODE with conventional deep architectures such as ResNets, CNNs, and Transformers by emphasizing higher-order convolutions via Volterra filters and continuous depth via ODEs. Instead of stacking many linear convolutions and pointwise nonlinearities, VNODE uses fewer highly expressive Volterra filters capturing polynomial interactions in each layer. The ODE blocks are described as “infinitely thin” residual layers integrated over time, reusing the same Volterra vector field throughout the interval (Roheda et al., 29 Sep 2025). This suggests a view of VNODE as combining explicit higher-order local structure with parameter sharing across continuous depth.

For ImageNet-1K, the paper reports the following comparisons: VNODE $L$5 has 9.1M parameters, 2.4 GFLOPs, and 83.5% top-1 accuracy; ResNet-152 has 60.2M parameters, 11.51 GFLOPs, and 78.31%; ConvNeXt-T has 29M parameters, 4.5 GFLOPs, and 82.1% (Roheda et al., 29 Sep 2025). The stated interpretation is that VNODE reaches or surpasses the accuracy of heavy CNNs and modern ConvNeXt variants with significantly fewer parameters and FLOPs.

The design is also explicitly connected to a proposed pattern of brain processing: discrete events, modeled by discrete Volterra layers, and continuous integration, modeled by continuous-time ODE evolution. On that basis, VNODE is described as a biologically inspired hybrid system rather than a purely discrete feed-forward network (Roheda et al., 29 Sep 2025). A plausible implication is that the piecewise architecture is intended not only as a computational device but also as a structural analogy for mixed event-driven and continuous integration regimes.

4. Optimization, supervision, and computational complexity

At each stage $L$6, the model applies a classifier to the stage output $L$7,

$L$8

where $L$9 is softmax, and the final prediction is $\mathbf{W}^k(\tau_1,\dots,\tau_k)$0, although intermediate classifiers also contribute to training (Roheda et al., 29 Sep 2025). The stagewise loss is a cross-entropy between $\mathbf{W}^k(\tau_1,\dots,\tau_k)$1 and the ground truth $\mathbf{W}^k(\tau_1,\dots,\tau_k)$2,

$\mathbf{W}^k(\tau_1,\dots,\tau_k)$3

and the overall cost function is written as

$\mathbf{W}^k(\tau_1,\dots,\tau_k)$4

The paper characterizes the intermediate losses as deep supervision, guiding each stage to learn useful features and improving the optimization landscape (Roheda et al., 29 Sep 2025).

Backpropagation through the ODE solver uses the adjoint sensitivity method, as in the original Neural ODE work (Chen et al., 2018). The ODE solver is treated as a black box; adjoint variables satisfy another ODE backward in time; and gradients with respect to parameters and initial states are computed by solving this adjoint ODE (Roheda et al., 29 Sep 2025). The reported benefits are memory usage $\mathbf{W}^k(\tau_1,\dots,\tau_k)$5 with respect to the number of solver steps, complexity $\mathbf{W}^k(\tau_1,\dots,\tau_k)$6 where $\mathbf{W}^k(\tau_1,\dots,\tau_k)$7 is the number of function evaluations, and explicit control over numerical error through solver tolerances (Roheda et al., 29 Sep 2025). These statements align with the Neural ODE framework, which presents constant memory cost, adaptive evaluation strategies, and an explicit precision-speed trade-off (Chen et al., 2018).

The paper does not describe special regularizers beyond cross-entropy, but identifies several design choices that contribute to stability: second-order, truncated Volterra filters; deep supervision via stage-wise classifiers; grouped Volterra filters in $\mathbf{W}^k(\tau_1,\dots,\tau_k)$8; and element-wise addition of time instead of concatenation (Roheda et al., 29 Sep 2025). These are presented not as separate optimization algorithms but as architectural constraints that control complexity and stabilize training.

The ImageNet-1K parameter and FLOP comparisons stated in the paper are summarized below.

Model	Params / GFLOPs	Top-1
VGG-16	138M / 15.4	75.2%
ResNet-50	25.6M / 4.09	76.1%
ResNet-152	60.2M / 11.51	78.31%
ConvNeXt-T	29M / 4.5	82.1%
E-ConvNeXt-Tiny	13.2M / 2.04	80.6%
E-ConvNeXt-Small	19.4M / 3.12	81.9%
TinyViT	21M / 4.4	83.1%
MobileViT-XS	2.3M / 0.7	74.8%
NODE	0.7M / 0.3	68.3%
MALI	11.2M / not reported	70.17%
Vanilla VNN	12M / 3.6	83.3%
VNODE (M=1)	1.5M / 0.4	78.6%
VNODE (M=3)	4.5M / 1.2	81.1%
VNODE (M=6)	9.1M / 2.4	83.5%

The accompanying observations in the source are that fully continuous VNODE $\mathbf{W}^k(\tau_1,\dots,\tau_k)$9 already outperforms standalone NODEs by approximately 10 percentage points with about twice the FLOPs and parameters, and that increasing $k$0 improves accuracy while retaining lower parameter and FLOP budgets than a number of heavier baselines (Roheda et al., 29 Sep 2025).

5. Empirical behavior on CIFAR-10 and ImageNet-1K

The paper evaluates VNODE on CIFAR-10 and ImageNet-1K using top-1 classification accuracy as the evaluation metric (Roheda et al., 29 Sep 2025). For CIFAR-10, the architecture uses $k$1 Volterra kernels, a single channel expansion to 64, and then fixed channels. The description states that experiments compare against state-of-the-art CNN and Transformer baselines and that VNODE is superior and more efficient, although the exact table is not reproduced in the supplied details (Roheda et al., 29 Sep 2025). For ImageNet-1K, the model uses Inception-inspired multi-scale Volterra filters with $k$2, $k$3, and $k$4 kernels, six stages, and channels increased to 1024 (Roheda et al., 29 Sep 2025).

The central quantitative ImageNet-1K result is that VNODE $k$5 achieves 83.5% top-1 accuracy with 9.1M parameters and 2.4 GFLOPs (Roheda et al., 29 Sep 2025). The source further states that this exceeds ResNet-152 at approximately one-sixth of the parameters and one-fifth of the FLOPs, exceeds ConvNeXt-T with about one-third of the parameters and one-half of the FLOPs, and performs slightly better than Vanilla VNN while being more parameter-efficient (Roheda et al., 29 Sep 2025).

Ablation through the settings $k$6, $k$7, and $k$8 is used to compare fully continuous and piecewise formulations. The reported pattern is that increasing the number of stages materially improves performance, indicating benefits from the piecewise structure relative to a single long continuous block (Roheda et al., 29 Sep 2025). The comparison between Vanilla VNN and VNODE $k$9 is used to argue that adding continuous dynamics via ODE blocks improves both accuracy and efficiency (Roheda et al., 29 Sep 2025).

The qualitative analysis includes Grad-CAM visualization. In comparisons with ResNet-152, VNODE is reported to focus more on the object of interest and reduce attention on irrelevant background regions (Roheda et al., 29 Sep 2025). This is interpreted in the source as suggesting that higher-order and continuous dynamics help the model learn more object-centric features. The robustness analysis evaluates CIFAR-10C, comprising 15 corruption types and 5 severity levels, for a total of 75 scenarios. The source says the corresponding table shows “impressive robustness” without extra augmentations or fine-tuning, and characterizes the overall takeaway as better generalization under distribution shifts or corruptions than many baselines (Roheda et al., 29 Sep 2025). Because the specific robustness numbers are not supplied, no stronger quantitative statement is warranted.

6. Expressive rationale, constraints, and open questions

The paper’s mechanistic intuition is that Volterra filters provide rich local interactions by explicitly computing products of input components, thereby yielding a polynomial mapping that can approximate a broad class of nonlinear functions with fewer layers (Roheda et al., 29 Sep 2025). ODE blocks then act as continuous refinement flows: once higher-order interactions are extracted, the same Volterra-based vector field is applied at many infinitesimal steps, producing a deep but parameter-sharing transformation (Roheda et al., 29 Sep 2025). Repeated alternation between discrete Volterra steps and continuous Volterra ODE evolution is presented as enabling approximation of complex hierarchical transformations with comparatively few parameters (Roheda et al., 29 Sep 2025).

The dependencies captured by Volterra terms are described in terms of multiplicative interactions among pixels or features, such as second-order products $\prod_{j=1}^k x(t-\tau_j)$0 within a neighborhood, polynomial structure and higher-order statistics, and complex local patterns including texture patterns, edge intersections, or complex shape cues (Roheda et al., 29 Sep 2025). The comparison point is that typical CNN interactions are mostly additive before a pointwise nonlinearity, whereas Volterra convolutions explicitly represent products of components (Roheda et al., 29 Sep 2025). This suggests that VNODE is intended to shift some representational burden from depth to higher-order local operators.

The paper also identifies several limitations. Even with approximations, second-order Volterra filters are more complex to design and optimize than plain convolutional layers, and efficient GPU implementations are non-trivial (Roheda et al., 29 Sep 2025). Runtime remains tied to the number of function evaluations and can be sensitive to solver choice and step-size control, as in Neural ODEs more generally (Roheda et al., 29 Sep 2025, Chen et al., 2018). The piecewise architecture introduces additional hyperparameters, including the number of stages $\prod_{j=1}^k x(t-\tau_j)$1 and the time partition, and scaling to larger datasets, higher resolutions, or other modalities is stated to require further exploration (Roheda et al., 29 Sep 2025). Although the model is described as biologically inspired, the learned continuous dynamics are still difficult to interpret (Roheda et al., 29 Sep 2025).

Within the broader continuous-time literature, Neural ODEs supply the conceptual and algorithmic groundwork: continuous-depth hidden-state evolution, black-box integration, adaptive evaluation, and adjoint-based gradient computation (Chen et al., 2018). VNODE remains within that paradigm but replaces standard vector fields with structured Volterra operators and inserts discrete event-style feature extraction between ODE intervals (Roheda et al., 29 Sep 2025). In the broader Volterra network context, previous VNNs are noted as stacked Volterra convolutional networks with residual connections applied to action recognition, restoration, and noise cancellation, while VNODE extends that line into the continuous-time regime (Roheda et al., 29 Sep 2025). The future directions identified in the source include sequence modeling and time series, video and spatiotemporal modeling, control systems and physical modeling, audio and speech, sensor data, and architectural refinements such as adaptive event points, learnable event times, and combinations with attention mechanisms or transformers (Roheda et al., 29 Sep 2025).