Frame Flow: Multi-Domain Motion & Inference

• Frame flow is a family of formulations that represent, propagate, or condition motion between discrete frames across applications like video interpolation, separation logic, and protein backbone generation.
• Methods such as optical flow, deep voxel flow, multi-interflow, and scene flow address complex motion dynamics, non-rigid deformations, and temporal continuity in diverse systems.
• Innovative approaches in frame flow drive research forward by enabling self-supervised learning, Transformer-based attention, event fusion, and efficient multi-frame estimation.

“Frame flow” (Editor’s term) denotes a family of formulations in which a model represents, propagates, or conditions on motion between discrete frames. In computer vision, these formulations appear in video frame interpolation, video frame synthesis, multi-frame optical flow estimation, scene flow estimation, and frame–event fusion. The same label also appears in a flow framework for separation logic reasoning about programs that manipulate general graphs and in an SE(3) flow-matching method for fast protein backbone generation (Gao et al., 2023, Meyer et al., 2023, Yim et al., 2023).

1. Core formulations across frame-based modeling

In video frame interpolation, the task is to synthesize one or more intermediate frames $I_t$, $0$I_0$ and $I_1$. A common strategy is to estimate an optical-flow field that describes the per-pixel motion between the inputs and then warp and blend the inputs to form the intermediate image. In traditional optical flow, a one-to-one flow from frame $A$ to frame $B$, $$F_{A\to B}=(f_x,f_y)\in\mathbb R^{H\times W\times 2}$$, assigns to each pixel $(x,y)$ a single 2D offset $(f_x(x,y),f_y(x,y))$. By contrast, ST-MFNet defines a multi-interflow, or many-to-one mapping, in which each output pixel draws from a weighted combination of $N$ source locations, which can model spatially complex, non-rigid motion such as water, fire, and foliage (Danier et al., 2021).

In video frame synthesis, Deep Voxel Flow represents motion by a 3D “voxel-flow” vector $0Liu et al., 2017).

In dynamic stereo, scene flow is the per-pixel 3D motion, equivalently $0Taniai et al., 2017).

Formulation	Core state	Representative use
Optical flow	$I_0$0	Frame interpolation, multi-frame flow
Deep voxel flow	$I_0$1	Interpolation and extrapolation
Multi-interflow	$I_0$2	Complex non-rigid motion
Scene flow	$I_0$3	Dynamic stereo and motion segmentation

A consistent theme is that “frame flow” is not a single representation. It ranges from pairwise 2D correspondence, to 3D voxel warping, to many-to-one weighted sampling, to coupled disparity–motion estimation.

2. Flow-guided frame interpolation and frame synthesis

Deep Voxel Flow combines the advantages of flow-based copying and end-to-end learning. The network is trained without human supervision: on UCF-101, the middle frame is dropped, the two outer frames are fed into the network, and the dropped frame is used as supervision. The method uses only self-supervised reconstruction losses plus simple smoothness regularizers; no adversarial or perceptual losses are needed to obtain sharp results. On UCF-101 frame interpolation, Deep Voxel Flow reports PSNR $I_0$4 and SSIM $I_0$5, compared with $I_0$6 and $I_0$7 for EpicFlow and $I_0$8 and $I_0$9 for Beyond MSE (Liu et al., 2017).

“Video Frame Interpolation with Flow Transformer” proposes Video Frame Interpolation Flow Transformer, which incorporates motion dynamics from optical flows into the self-attention mechanism. Its Flow Transformer Block computes temporal self-attention in a matched local area with the guidance of flow. The $I_1$0 search window $I_1$1 is not centered at the query location $I_1$2 but rather at $I_1$3. VFIFT adopts a three-stage pyramid of FTFB modules at $I_1$4, $I_1$5, and full resolution, and its local flow-guided self-attention has per-block complexity $I_1$6, far below global attention $I_1$7. On Vimeo90K, VFIFT with $I_1$8 M parameters reaches PSNR $I_1$9, SSIM $A$0; on UCF101 it reaches PSNR $A$1, SSIM $A$2; on DAVIS480p it reaches PSNR $A$3, SSIM $A$4. Runtime on an RTX 3090 is $A$5 s per frame, versus $A$6 s for VFIformer-S and $A$7 s for VFIT-S. Omitting the optical-flow shift drops PSNR by $A$8–$A$9 dB (Gao et al., 2023).

“Frame Interpolation for Dynamic Scenes with Implicit Flow Encoding” addresses brightness and illumination changes by taking advantage of existing optical flow methods that are highly robust to the variations in the illumination. It uses bidirectional flows from a pre-trained flow network, predicts flows from an intermediate frame to the two input images through a coordinate-based network powered by a hypernetwork, and then uses the estimated flows within an existing blending network. On Xiph 2K, Xiph 4K, and MPI-Sintel, the interpolated-flow EPE is $B$0, versus FILM’s learned intermediate flows at $B$1 (Figueirêdo et al., 2022).

EA-Net integrates the edge information into the frame interpolation task through edge-guided flow estimation and edge-protected frame synthesis. In the flow-estimation stage, it uses three edge-aware mechanisms—Edge-Augmentation, Edge-Concatenation, and Two-Stream—and in the synthesis stage it predicts refined flows and an attention map $B$2, with $B$3, to blend the two warped candidates. The final intermediate frame is

$B$4

On Vimeo90K, EA-Net reports PSNR $B$5, SSIM $B$6; on UCF101, PSNR $B$7, SSIM $B$8; on Adobe240-fps, PSNR $B$9, SSIM $$F_{A\to B}=(f_x,f_y)\in\mathbb R^{H\times W\times 2}$$0 (Zhao et al., 2021).

FG-DFPN addresses video frame prediction rather than interpolation, but it shares the same design principle: explicit motion cues capture the dominant, coarse motion, and deformable convolutions learn to correct the residual, fine-grain motion. It receives four consecutive grayscale frames $$F_{A\to B}=(f_x,f_y)\in\mathbb R^{H\times W\times 2}$$1, predicts four dense flow fields $$F_{A\to B}=(f_x,f_y)\in\mathbb R^{H\times W\times 2}$$2, warps multi-scale features in feature space, and refines deformable offsets by

$$F_{A\to B}=(f_x,f_y)\in\mathbb R^{H\times W\times 2}$$3

Across eight MPEG test sequences, FG-DFPN reports average PSNR $$F_{A\to B}=(f_x,f_y)\in\mathbb R^{H\times W\times 2}$$4, average SSIM $$F_{A\to B}=(f_x,f_y)\in\mathbb R^{H\times W\times 2}$$5, and runtime per frame $$F_{A\to B}=(f_x,f_y)\in\mathbb R^{H\times W\times 2}$$6 ms, compared with DFPN at $$F_{A\to B}=(f_x,f_y)\in\mathbb R^{H\times W\times 2}$$7, $$F_{A\to B}=(f_x,f_y)\in\mathbb R^{H\times W\times 2}$$8, and $$F_{A\to B}=(f_x,f_y)\in\mathbb R^{H\times W\times 2}$$9 ms (Yılmaz et al., 14 Mar 2025).

3. Multi-frame optical flow and scene flow

VideoFlow redefines the basic optical-flow unit as a tri-frame estimator with full bidirectional fusion. Its TRi-frame Optical Flow module estimates flows from a center frame $(x,y)$0 to both neighbors $(x,y)$1 and $(x,y)$2 in a single module, while the MOtion Propagation module bridges multiple TROFs and propagates motion features between adjacent TROFs. The iterative update is

$(x,y)$3

On Sintel, VideoFlow achieves $(x,y)$4 average end-point-error on the clean pass and $(x,y)$5 on the final pass; on KITTI-2015 it achieves an F1-all error of $(x,y)$6. It ranks 1st on all public benchmarks and reduces AEPE on large-motion $(x,y)$7 px pixels from $(x,y)$8 to $(x,y)$9 (Shi et al., 2023).

StreamFlow targets the inefficiency of recursive multi-frame estimation. Its Streamlined In-batch Multi-frame pipeline computes features and adjacent-pair correlations exactly once per group, the Integrative Spatio-temporal Coherence module introduces spatio-temporal coherence in the encoder with zero new trainable weights, and the Global Temporal Regressor explores temporal relations during decoding. On Sintel with all modules enabled, end-to-end latency is $(f_x(x,y),f_y(x,y))$0 ms, compared with $(f_x(x,y),f_y(x,y))$1 ms for a three-frame recursive baseline, and the method reports a $(f_x(x,y),f_y(x,y))$2 enhancement in speed compared with previous multi-frame methods such as VideoFlow and TransFlow. On Sintel test, it reports clean $(f_x(x,y),f_y(x,y))$3 and final $(f_x(x,y),f_y(x,y))$4, with unmatched EPE $(f_x(x,y),f_y(x,y))$5 on clean and $(f_x(x,y),f_y(x,y))$6 on final (Sun et al., 2023).

DeltaFlow extends frame flow into multi-frame scene flow estimation for LiDAR. Instead of concatenating temporal features, it forms a single $(f_x(x,y),f_y(x,y))$7-feature

$(f_x(x,y),f_y(x,y))$8

Because the result lives in $(f_x(x,y),f_y(x,y))$9 regardless of $N$0, the backbone always sees the same feature shape. The method adds a Category-Balanced Loss and an Instance Consistency Loss to address imbalanced object class distributions and motion inconsistency. On Argoverse 2, $N$1Flow at 5 frames reports dynamic bucket-normalized EPE mean $N$2, compared with Flow4D at $N$3, and three-way EPE mean $N$4 cm, compared with $N$5 cm. On Waymo, $N$6Flow at 2 frames reports $N$7 cm, versus $N$8 cm for Flow4D, with runtime $N$9 s per sequence versus $0Zhang et al., 23 Aug 2025).

Fast multi-frame stereo scene flow with motion segmentation follows a different decomposition. It estimates disparity using SGM, refines 6-DOF camera motion by direct visual odometry, identifies regions inconsistent with the estimated camera motion, computes per-pixel optical flow only at these regions, and fuses the rigid and non-rigid flow proposals by graph-cut. On KITTI 2015 scene-flow, it reports disparity error $0Taniai et al., 2017).

4. Auxiliary signals, multimodal fusion, and frame-wise representations

“Fusing Frame and Event Vision for High-speed Optical Flow for Edge Application” combines a frame-based Farneback pipeline with an event-stream “leaky CNN” pipeline. The event pipeline maintains a leaky temporal accumulator

$0

applies differential kernels to estimate horizontal and vertical motion channels, and smooths them with a $0AEE $0FPS to $0Lele et al., 2022).

FFNeRV uses flow information inside frame-wise neural representations for videos. Instead of mapping $0

$0

and the final output is

$0

With about $0quantization-aware training and entropy coding, it outperforms H.264 and HEVC and performs on par with state-of-the-art video compression algorithms (Lee et al., 2022).

These systems broaden the scope of frame flow. In one case, the frame signal is complemented by asynchronous events; in the other, frame flow is built directly into a neural video representation and rate–distortion pipeline.

5. Formal-methods and generative uses of “FrameFlow”

Outside vision, “Make flows small again: revisiting the flow framework” develops a flow framework for separation logic reasoning about programs that manipulate general graphs. A flow graph $0

$0

and by Kleene’s theorem its least fixed point $0Contextual equivalence is defined by comparing transfer from inflows to outflows: $0Meyer et al., 2023).

“Fast protein backbone generation with SE(3) flow matching” uses the title FrameFlow for a generative model on protein residue frames. On the manifold $0ODE

$0

matches the data distribution at $0SDE steps and $0GENIE at $0Yim et al., 2023).

A plausible implication is that the word “flow” retains a common structural role across very different domains: it specifies a state evolution law, a transfer function, or a motion field, while “frame” refers either to temporal video frames, logical framing in separation logic, or residue frames in protein geometry.

6. Limitations, failure modes, and recurring research questions

Several recurring limitations appear across frame-flow formulations. In Deep Voxel Flow, scenes with highly repetitive textures or ambiguous motion may cause voxel copying to latch onto wrong regions, and the method cannot hallucinate entirely new content outside the support of the input frames, such as large dis-occlusions and new objects (Liu et al., 2017). In FG-DFPN, extremely fast motions that exceed the maximum deformable receptive field can still produce ghosting, and severe occlusions/unocclusions are only implicitly handled because explicit occlusion masks are not modeled (Yılmaz et al., 14 Mar 2025).

Dynamic lighting introduces a different failure mode. Real image datasets with ground-truth intermediate frames under dynamic lighting are scarce, so “Frame Interpolation for Dynamic Scenes with Implicit Flow Encoding” focuses on flow EPE for quantitative evaluation and presents extensive visual comparisons. This suggests that photometric variation remains a data and benchmarking bottleneck even when motion estimation is robust (Figueirêdo et al., 2022).

In high-speed optical flow, frame-based processing is bound by sensor frame-rate and by heavier CNN or optimization latencies for high accuracy, while event-based methods can run at kHz but suffer from noisy, sparse intensity-invariant data and lower accuracy. The frame–event fusion results show that the classic accuracy versus latency trade-off can be mitigated, but not eliminated (Lele et al., 2022).

Within Transformer-based interpolation, the original Transformer is commonly used for 2D images; how to develop a Transformer-based framework with consideration of temporal self-attention for video frame interpolation remained an open issue in the formulation of VFIFT. The VFIFT results suggest one practical answer: calculate temporal self-attention in a matched local area with the guidance of flow, rather than use unconstrained global attention (Gao et al., 2023).

Across the literature, the central tension is stable: large motion, occlusions, motion blur, dynamic textures, illumination change, and computational overlap all push frame-flow methods away from simple pairwise correspondence. The resulting systems therefore combine motion priors, local or multi-scale matching, temporal propagation, adaptive warping, or alternative modalities to preserve detail while keeping complexity reasonably low.