Analysis of Error Propagation in Autoencoder-Based Reduced-Order Neural Ordinary Differential Equations

Abstract: Neural ODE reduced-order models often achieve comparable local prediction accuracy, yet their long-horizon extrapolation behavior can differ substantially. To analyze this discrepancy, we develop a path-integral identity that separates local discrepancy injection from amplification in the learned latent dynamics. The associated multi-step Jacobian norms quantify transport sensitivity and distinguish different propagation regimes. Experiments on the Burgers and Gray--Scott systems exhibit two distinct patterns of error evolution. In Burgers systems, prediction errors remain bounded and are primarily associated with persistent local discrepancies. In contrast, Gray--Scott systems exhibit pronounced amplification during extrapolation, where Jacobian norms serve as sensitivity diagnostics rather than direct indicators of physical prediction accuracy.

• The paper introduces an exact path-integral decomposition and Jacobian-product diagnostic that separate local discrepancy injection from amplification of existing rollout errors.
• Experiments show Burgers systems remain injection-dominated with errors near 1.5% and transport sensitivity near one, making rollout exposure strategies largely irrelevant.
• Gray–Scott results show that reducing local discrepancy can lower mean relative error from 0.340 to 0.159 even when transport amplification remains similar, highlighting the need for mechanism-specific training.

This paper develops a theoretical and empirical framework for analyzing how errors propagate during autonomous rollout of autoencoder-based reduced-order Neural ODE models (2608.13132). The central observation motivating the work is that models with comparable local prediction accuracy can exhibit markedly different long-horizon extrapolation behavior, and that standard rollout error metrics cannot explain why. The authors' response is a path-integral decomposition of rollout error into two mechanisms—local discrepancy injection at each prediction step and transport amplification of existing deviations by the learned dynamics—paired with a Jacobian-based transport diagnostic that quantifies the amplification capacity of the learned flow map along the reference trajectory.

Framework and error decomposition

The model class is standard: an autoencoder $(E_w, D_w)$ maps physical states $u(t)$ to a latent space, and a Neural ODE $dz/dt = g_\theta(z)$, integrated with fourth-order Runge–Kutta, defines a one-step latent flow operator $\Psi_\theta$ whose repeated composition generates autonomous rollout from the last training state. Training proceeds in two stages—reconstruction loss for the autoencoder, followed by multi-step rollout loss over the latent dynamics—with three rollout exposure strategies (Base: maximum horizon only; Mixed: uniform weighting; Mix2: horizon-weighted with $p(k) \propto k$) plus a gradient-clipping baseline that modifies only optimization dynamics.

The core analytical object is the rollout deviation $\epsilon_n = \Psi_\theta^{(n-m)}(z_m) - z_n$ relative to the encoded reference trajectory. The key identity is exact: under a compactness and differentiability assumption, the deviation satisfies

$$\epsilon_{n+1} = \left(\int_0^1 D\Psi_\theta(z_n + s\epsilon_n)\,ds\right)\epsilon_n + \delta_n,$$

where $\delta_n = \Psi_\theta(z_n) - z_{n+1}$ is the one-step local discrepancy. This cleanly separates transport of the current deviation from fresh injection at each step. A recursive Taylor expansion yields a multi-step decomposition in which the accumulated deviation equals the transported initial deviation, plus the cumulative transported discrepancy $\Xi_{n,k}$, plus a quadratic remainder $R_{n,k}$—a linearization around the reference trajectory whose validity degrades when deviations grow large or Jacobian products amplify strongly. A conservative domain-wide bound $u(t)$0 follows, with the familiar geometric-growth form for $u(t)$1 and linear accumulation for $u(t)$2.

The transport diagnostic is the spectral norm $u(t)$3 of the ordered Jacobian product evaluated along the encoded reference trajectory, computed via automatic differentiation and SVD (with matrix-free power iteration suggested for high-dimensional latent spaces). Values of order one indicate a weak amplification regime; values far above one indicate capacity for strong amplification, with the caveat that realized amplification depends on alignment of the deviation with sensitive directions.

Weak amplification: Burgers systems

The 1D and 2D Burgers equations instantiate the injection-dominated regime. For 1D Burgers, relative $u(t)$4 test errors remain near 1.5% (Base: 1.549 ± 0.487; Mixed: 1.419 ± 0.377), local discrepancies are approximately 0.01, log-slopes of $u(t)$5 cluster near zero (0.054–0.055), and the transport diagnostic confirms $u(t)$6. For 2D Burgers, all exposure variants stay within roughly 1% of the AE-only baseline (test relative errors ≈ 0.114), with log-slopes of 0.0316 across variants. A direct implication is that in this regime, rollout exposure strategy is essentially irrelevant: paired tests across ten seeds yield $u(t)$7 for all strategy comparisons, and the paper notes that seed variation dominates over optimization choices. The paper also reports that linear POD/DMD baselines fail badly at extrapolation (relative errors above 2.4, degrading further with latent dimension), while the nonlinear autoencoder at dimension 16 achieves 0.110—supporting the claim that nonlinear latent representation, not the dynamics model, is the limiting factor here.

Strong amplification: Gray–Scott

The Gray–Scott reaction–diffusion system exhibits the opposite behavior, and here the diagnostics become genuinely informative. Mean $u(t)$8 is approximately 0.006 for 2D Burgers but ranges from 1.16 to 1.71 for Gray–Scott models—more than an order of magnitude in linearized amplification. The most consequential quantitative finding is that Base and Mixed operate under nearly identical transport sensitivity (log$u(t)$9 of 1.713 vs. 1.628) yet achieve very different physical accuracy: Mixed attains a mean relative error of 0.159 with standard deviation 0.016, versus 0.340 ± 0.189 for Base. The decomposition explains this: Mixed reduces local discrepancy injection ($dz/dt = g_\theta(z)$0 from 6.708 to 4.254) and consequently the latent deviation (9.716 to 3.788) while the amplification regime is unchanged. This is a strong, well-supported claim that neither mechanism alone governs long-horizon error—their joint contribution does.

The gradient-clipping result carries a notable caveat. Clipping achieves a comparable mean error (0.165) but with extreme seed variability (0.069 to 0.474), and it reduces latent deviation without proportional improvement in physical accuracy—consistent with the decomposition, since latent deviation reflects both transport and injection. Statistical evidence is test-dependent: Base vs. Clipped is significant under both tests ($dz/dt = g_\theta(z)$1 t-test, $dz/dt = g_\theta(z)$2 Wilcoxon), but Mixed vs. Clipped is significant only under Wilcoxon ($dz/dt = g_\theta(z)$3 vs. $dz/dt = g_\theta(z)$4 for the t-test). The paper appropriately flags this as inconclusive for that pairwise comparison. Cross-seed $dz/dt = g_\theta(z)$5 between transport sensitivity and latent deviation remains moderately high (0.736–0.816 for Gray–Scott), but the authors correctly note this association does not imply transport sensitivity alone dictates prediction accuracy.

Limitations and open questions

Several limitations are acknowledged or evident. The multi-step decomposition is a local linearization around the encoded reference trajectory; the nonlinear remainder is bounded only quadratically and may become non-negligible precisely in the strong-amplification regime where the diagnostic is most needed. The diagnostic $dz/dt = g_\theta(z)$6 measures amplification capacity along the reference trajectory, not realized nonlinear error evolution, and its predictive power for physical-space accuracy is explicitly limited—Jacobian norms serve as sensitivity diagnostics rather than direct accuracy indicators. The empirical validation covers three canonical benchmark systems with a fixed architecture (latent dimension 16, width 128), so the regime classification's generality to other PDEs, stiffer dynamics, or higher latent dimensions is untested. The statistical evidence for some Gray–Scott comparisons is sample-limited. The paper leaves open the design of manifold-aware regularization to constrain off-manifold latent solver trajectories, which the conclusion identifies as the natural next step for strongly amplifying systems.

Conclusion

The paper contributes an exact path-integral identity and a trajectory-dependent Jacobian diagnostic that together convert the vague observation that "long-horizon behavior differs" into a mechanistic classification of reduced-order Neural ODE rollout. The empirical results substantiate the classification: Burgers systems are injection-dominated with strategy-invariant, sub-2% errors, while Gray–Scott exhibits amplification-dominated drift in which reducing local discrepancy injection—rather than transport sensitivity—drives the improvement from 0.340 to 0.159 mean relative error. The framework's main practical value is diagnostic: it identifies when training effort should target reconstruction fidelity versus when it must target trajectory regularization, and it makes explicit that Jacobian amplification is a necessary but not sufficient condition for large physical prediction error.