Learning Video Dynamics with Predictive Differentiable Rendering

Abstract: How to accurately predict a high-fidelity future world? While the visual world is inherently continuous, existing deterministic video prediction models operate in discrete pixel space and are mainly optimized with pixel-wise mean squared error (MSE), which often leads to over-smoothed predictions and a lack of fine-grained visual details. To address these limitations, we propose Predictive Differentiable Rendering (PDR), a novel end-to-end video prediction paradigm that bridges the gap between discrete and continuous representations. Inspired by recent progress in 3D reconstruction with 3D Gaussian Splatting, we introduce PredGS, a lightweight and plug-and-play adapter based on 2D Gaussian representation, which could be seamlessly integrated with existing pixel space predictors, significantly improving spatial detail preservation with negligible computational overhead. Furthermore, we develop predgsplat, a CUDA-accelerated differentiable 2D Gaussian renderer supporting arbitrary channels. Each Gaussian is defined by 5 + C learnable parameters (position, scale, rotation, and C channel amplitudes) and achieves up to 10x faster rendering than the baseline. Optimized by a combined L1 and SSIM loss, PDR overcomes the inherent blurring tendencies of MSE Loss, significantly enhancing the prediction performance. Extensive experiments on diverse real-world benchmarks, including TaxiBJ, WeatherBench, KTH, and Human3.6M, demonstrate that PDR consistently surpasses existing methods, delivering superior detail preservation, visual fidelity, and predictive accuracy.

• The paper introduces PDR, a novel hybrid approach that models video dynamics using both discrete pixels and continuous 2D Gaussian parameterization to prevent over-smoothing.
• It integrates a lightweight plug-and-play adapter and a CUDA-optimized differentiable renderer to ensure real-time inference (up to 1000 FPS) with superior spatial fidelity.
• Empirical evaluations across multiple datasets demonstrate significant improvements, including up to 29% reduction in LPIPS and notable gains in SSIM/PSNR over traditional pixel-based models.

Predictive Differentiable Rendering for Video Dynamics

Motivation and Paradigm Shift

Deterministic video prediction conventionally operates in discrete pixel space, inducing systemic over-smoothing under pixel-wise regression losses such as MSE. This severely constrains the preservation of spatial detail, especially in long-range forecasting tasks with high-dimensional spatiotemporal signals. The paper introduces Predictive Differentiable Rendering (PDR), an end-to-end paradigm that jointly models video dynamics in discrete (pixel) and continuous (2D Gaussian) domains. The core concept leverages continuous 2D Gaussian parameterization for each frame, inspired by advances in 3D Gaussian Splatting, and directly addresses the tendency of deterministic baselines to collapse multi-modal motion into blurry averages. The framework integrates a lightweight, plug-and-play adapter (PredGS) and a CUDA-optimized differentiable renderer (predgsplat), enabling both superior spatial fidelity and efficient real-time inference.

Figure 1: Qualitative and quantitative comparison of pixel-based baseline vs. PDR, illustrating sharper, more accurate predictions and substantial metric improvements with marginal compute overhead.

Technical Formulation

PDR operates via a hybrid architecture: a pixel-space predictor $\mathcal{F}_p$ produces coarse future predictions from observed frames; a parameter-space adapter $\mathcal{F}_c$ predicts per-frame Gaussian parameters (position, scale, rotation, channel amplitudes). These Gaussians are rendered into pixel-aligned maps through the predgsplat module, and fused with the original pixel-space output via a learnable $1 \times 1$ convolution for final prediction. Each Gaussian is defined by $5 + C$ learnable parameters, supporting arbitrary channel dimensionality for generalization across modalities such as RGB, weather, and traffic fields.

Figure 2: PDR pipeline overview: pixel-space backbone, PredGS parameter adapter, CUDA-accelerated renderer, and fusion.

The continuous Gaussian representation offers spatial smoothness and differentiability, circumventing limitations of discrete grid sampling and improving the modeling of fine-grained motion, boundaries, and textures. All Gaussian parameters are rigorously constrained via nonlinear activations to ensure physical validity (e.g., positions normalized, angles mapped to $[−\pi/2, \pi/2]$, amplitudes to $[0, 1]$).

Optimization Approach

Training incorporates joint optimization of both branches using a hybrid $\mathcal{L}_1$ + SSIM loss, favoring perceptual quality and structural coherence over pixel averaging. Empirical ablation shows that the loss design is critical: while the continuous representation alone is insufficient to outperform pixel baselines under MSE, the combination with perceptually-aligned supervision enables PDR to converge faster and reliably yield higher fidelity.

Figure 3: Training and validation loss curves on KTH: PDR converges faster and yields lower validation error than pixel-only models, evidencing improved stability and generalization.

The framework employs robust Gaussian coordinate initialization, with uniform grid sampling providing optimal stability and consistency. The predgsplat renderer is custom designed for real-time requirements, with vectorized CUDA kernels supporting arbitrary channels and eliminating expensive affine transformations, achieving up to $10\times$ speedup and exceeding $1000$ FPS at $256\times256$ resolution.

Empirical Evaluation

PDR is benchmarked across TaxiBJ, WeatherBench, KTH, and Human3.6M datasets, each representing distinct spatiotemporal and channel challenges. PDR consistently yields superior predictive accuracy, visual fidelity, and metric performance compared to exhaustive deterministic baselines (TAU, SimVP, PredRNN, MAU, etc.):

• TaxiBJ: 8.9% reduction in MSE, 4.1% in MAE, improvements in SSIM/PSNR.
• WeatherBench: 2.5% reduction in MSE, consistent accuracy gain.
• KTH and Human3.6M: up to 29.0% reduction in LPIPS, gains in SSIM and PSNR.

PDR also surpasses diffusion-based models (MCVD, ARFree) in real-time settings. While diffusion models offer maximal sharpness, PDR attains comparable PSNR/SSIM at $\mathcal{F}_c$0 higher FPS, making it uniquely suited for closed-loop and large-scale forecasting scenarios.

Figure 4: Visualization on KTH: PDR preserves limb boundaries and sharp contours, unlike pixel-based baselines.

Figure 5: Left: Diagnostic analysis on KTH evidencing mode collapse in pixel baselines and restoration of distributional structure in PDR. Right: Qualitative comparison across TaxiBJ, WeatherBench, Human3.6M: PDR retains fine spatial detail.

Distributional analysis shows PDR recovers much of the kurtosis and intensity diversity lost by pixel-based models (e.g., 56% kurtosis gap, 57% intensity diversity gap), confirming that continuous Gaussian modeling preserves salient multi-modal motion cues. Temporal coherence is maintained, with warping error closely tracking ground-truth (RAFT floor), remaining well above the over-smoothed regime.

Practical and Theoretical Implications

PDR demonstrates backbone-agnostic enhancement, improving diverse architectures (SimVP, ConvNeXt, ViT, HorNet, MLP-Mixer) with consistent performance gains. The plug-and-play PredGS adapter generalizes across modalities and backbone types without architectural modifications, suggesting broad applicability for video prediction, spatiotemporal forecasting, nowcasting, and motion modeling.

The framework opens new directions for unified discrete-continuous dynamics modeling, establishing that fine-grained spatial fidelity can be obtained with deterministic models at minimal computational cost. It further shows that real-time, perceptually-faithful prediction can be achieved without iterative diffusion or excessive parameter counts.

Future Directions

Potential research avenues include extending Gaussian parameterization to additional modalities (e.g., radar, multichannel weather fields), integrating temporal constraints for coherence over longer horizons, joint optimization with reinforcement agents for closed-loop systems, and hierarchical blending of multiple continuous representations for further fidelity and interpretability.

Conclusion

PDR establishes a novel paradigm for deterministic video prediction by bridging discrete pixel and continuous Gaussian spaces. Through lightweight parametric adapters and efficient differentiable rendering, it achieves substantial gains in spatial fidelity and predictive accuracy with real-time efficiency. The universal, backbone-agnostic nature of PredGS positions PDR as an effective booster for spatiotemporal forecasting tasks and advances the theoretical understanding of hybrid representation learning.