---
title: 'Frame Flow: Multi-Domain Motion & Inference'
url: https://www.emergentmind.com/topics/frame-flow
type: topic
---

# Frame Flow: Multi-Domain Motion & Inference

“Frame flow” (*Editor’s term*) denotes a family of formulations in which a model represents, propagates, or conditions on motion between discrete frames. In computer vision, these formulations appear in video frame interpolation, video frame synthesis, multi-frame optical flow estimation, scene flow estimation, and frame–event fusion. The same label also appears in a flow framework for separation logic reasoning about programs that manipulate general graphs and in an SE(3) flow-matching method for fast protein backbone generation [2307.16144] [2304.04886] [2310.05297].

## 1. Core formulations across frame-based modeling

In video frame interpolation, the task is to synthesize one or more intermediate frames \(I_t\), \(0<t<1\), from two input images \(I_0\) and \(I_1\). A common strategy is to estimate an optical-flow field that describes the per-pixel motion between the inputs and then warp and blend the inputs to form the intermediate image. In traditional optical flow, a one-to-one flow from frame \(A\) to frame \(B\), \(F_{A\to B}=(f_x,f_y)\in\mathbb R^{H\times W\times 2}\), assigns to each pixel \((x,y)\) a single 2D offset \((f_x(x,y),f_y(x,y))\). By contrast, ST-MFNet defines a multi-interflow, or many-to-one mapping, in which each output pixel draws from a weighted combination of \(N\) source locations, which can model spatially complex, non-rigid motion such as water, fire, and foliage [2111.15483].

In video frame synthesis, Deep Voxel Flow represents motion by a 3D “voxel-flow” vector \(F(x,y)=(\Delta x(x,y),\Delta y(x,y),\Delta t(x,y))\in\mathbb R^3\), with spatial offsets \((\Delta x,\Delta y)\) and a temporal weight \(\Delta t\in[0,1]\) that blends between two frames. The synthesized pixel is obtained by trilinear interpolation over eight corners of a “virtual voxel” in the 3D volume \(X\), which makes the whole system end-to-end differentiable [1702.02463].

In dynamic stereo, scene flow is the per-pixel 3D motion, equivalently \((D_t(p),F(p))\), where disparity \(D_t(p)\) determines depth \(z_t(p)=f\cdot B/D_t(p)\) and \(F(p)\) is the optical flow. The rigid-flow from frame \(t\to t+1\) for a static point is \(F_{\text{rig}}(p)=w(p;D_t,P_t)-p\), while non-rigid pixels receive a separate optical-flow proposal. This decomposition separates camera ego-motion from independently moving objects [1707.01307].

| Formulation | Core state | Representative use |
|---|---|---|
| Optical flow | \(F_{A\to B}\in\mathbb R^{H\times W\times 2}\) | Frame interpolation, multi-frame flow |
| Deep voxel flow | \((\Delta x,\Delta y,\Delta t)\in\mathbb R^3\) | Interpolation and extrapolation |
| Multi-interflow | \((\boldsymbol\alpha,\boldsymbol\beta,\mathbf w)\in\mathbb R^{H\times W\times N\times 3}\) | Complex non-rigid motion |
| Scene flow | \((D_t(p),F(p))\) | Dynamic stereo and motion segmentation |

A consistent theme is that “frame flow” is not a single representation. It ranges from pairwise 2D correspondence, to 3D voxel warping, to many-to-one weighted sampling, to coupled disparity–motion estimation.

## 2. Flow-guided frame interpolation and frame synthesis

Deep Voxel Flow combines the advantages of flow-based copying and end-to-end learning. The network is trained without human supervision: on UCF-101, the middle frame is dropped, the two outer frames are fed into the network, and the dropped frame is used as supervision. The method uses only self-supervised reconstruction losses plus simple smoothness regularizers; no adversarial or perceptual losses are needed to obtain sharp results. On UCF-101 frame interpolation, Deep Voxel Flow reports PSNR \(=35.8\,\text{dB}\) and SSIM \(=0.96\), compared with \(34.2\,\text{dB}\) and \(0.95\) for EpicFlow and \(32.8\,\text{dB}\) and \(0.93\) for Beyond MSE [1702.02463].

“Video Frame Interpolation with Flow Transformer” proposes Video Frame Interpolation Flow Transformer, which incorporates motion dynamics from optical flows into the self-attention mechanism. Its Flow Transformer Block computes temporal self-attention in a matched local area with the guidance of flow. The \(L\times L\) search window \(N_p\) is not centered at the query location \(p=(i,j)\) but rather at \(c_p=\operatorname{round}(p+F(p))\). VFIFT adopts a three-stage pyramid of FTFB modules at \(1/4\), \(1/2\), and full resolution, and its local flow-guided self-attention has per-block complexity \(O(HW\cdot C\cdot d + HW\cdot L^2\cdot d)\), far below global attention \(O(H^2W^2)\). On Vimeo90K, VFIFT with \(17.3\) M parameters reaches PSNR \(=36.43\,\text{dB}\), SSIM \(=0.9813\); on UCF101 it reaches PSNR \(=35.71\,\text{dB}\), SSIM \(=0.9795\); on DAVIS480p it reaches PSNR \(=28.27\,\text{dB}\), SSIM \(=0.8913\). Runtime on an RTX 3090 is \(0.11\) s per frame, versus \(0.18\) s for VFIformer-S and \(0.24\) s for VFIT-S. Omitting the optical-flow shift drops PSNR by \(0.8\)–\(1.1\) dB [2307.16144].

“Frame Interpolation for Dynamic Scenes with Implicit Flow Encoding” addresses brightness and illumination changes by taking advantage of existing optical flow methods that are highly robust to the variations in the illumination. It uses bidirectional flows from a pre-trained flow network, predicts flows from an intermediate frame to the two input images through a coordinate-based network powered by a hypernetwork, and then uses the estimated flows within an existing blending network. On Xiph 2K, Xiph 4K, and MPI-Sintel, the interpolated-flow EPE is \((3.40, 17.46, 5.16)\), versus FILM’s learned intermediate flows at \((13.97, 37.34, 12.90)\) [2209.13284].

EA-Net integrates the edge information into the frame interpolation task through edge-guided flow estimation and edge-protected frame synthesis. In the flow-estimation stage, it uses three edge-aware mechanisms—Edge-Augmentation, Edge-Concatenation, and Two-Stream—and in the synthesis stage it predicts refined flows and an attention map \(A_0(x)\in(0,1)\), with \(A_1(x)=1-A_0(x)\), to blend the two warped candidates. The final intermediate frame is
\[
\hat I_t(x)=A_0(x)\cdot W(I_0,F^r_{t\to 0})(x)+A_1(x)\cdot W(I_1,F^r_{t\to 1})(x).
\]
On Vimeo90K, EA-Net reports PSNR \(=34.39\,\text{dB}\), SSIM \(=0.9753\); on UCF101, PSNR \(=34.97\,\text{dB}\), SSIM \(=0.9675\); on Adobe240-fps, PSNR \(=31.02\,\text{dB}\), SSIM \(=0.9205\) [2105.07673].

FG-DFPN addresses video frame prediction rather than interpolation, but it shares the same design principle: explicit motion cues capture the dominant, coarse motion, and deformable convolutions learn to correct the residual, fine-grain motion. It receives four consecutive grayscale frames \(I_1\ldots I_4\), predicts four dense flow fields \(F_1\ldots F_4\in\mathbb R^{2\times H\times W}\), warps multi-scale features in feature space, and refines deformable offsets by
\[
\hat{\Delta p}_k^l(p_0)=\alpha^l\cdot\tanh(\Delta p_k^l(p_0)) + G(F_l)(p_0).
\]
Across eight MPEG test sequences, FG-DFPN reports average PSNR \(=32.09\,\text{dB}\), average SSIM \(=0.933\), and runtime per frame \(=125\) ms, compared with DFPN at \(31.19\,\text{dB}\), \(0.914\), and \(165\) ms [2503.11343].

## 3. Multi-frame optical flow and scene flow

VideoFlow redefines the basic optical-flow unit as a tri-frame estimator with full bidirectional fusion. Its TRi-frame Optical Flow module estimates flows from a center frame \(I_t\) to both neighbors \(I_{t-1}\) and \(I_{t+1}\) in a single module, while the MOtion Propagation module bridges multiple TROFs and propagates motion features between adjacent TROFs. The iterative update is
\[
h^{k+1}=\mathrm{Updater}(F_m^k,g,h^k),\qquad
\Delta f^k=\mathrm{FlowHead}(h^{k+1}),\qquad
f^{k+1}=f^k+\Delta f^k.
\]
On Sintel, VideoFlow achieves \(0.991\) average end-point-error on the clean pass and \(1.649\) on the final pass; on KITTI-2015 it achieves an F1-all error of \(3.65\%\). It ranks 1st on all public benchmarks and reduces AEPE on large-motion \(>40\) px pixels from \(11.66\) to \(8.80\) [2303.08340].

StreamFlow targets the inefficiency of recursive multi-frame estimation. Its Streamlined In-batch Multi-frame pipeline computes features and adjacent-pair correlations exactly once per group, the Integrative Spatio-temporal Coherence module introduces spatio-temporal coherence in the encoder with zero new trainable weights, and the Global Temporal Regressor explores temporal relations during decoding. On Sintel with all modules enabled, end-to-end latency is \(84.59\) ms, compared with \(122.18\) ms for a three-frame recursive baseline, and the method reports a \(63.82\%\) enhancement in speed compared with previous multi-frame methods such as VideoFlow and TransFlow. On Sintel test, it reports clean \(=1.04\) and final \(=1.87\), with unmatched EPE \(=6.42\) on clean and \(10.44\) on final [2311.17099].

DeltaFlow extends frame flow into multi-frame scene flow estimation for LiDAR. Instead of concatenating temporal features, it forms a single \(\Delta\)-feature
\[
\mathscr D_{\rm delta}=\frac1N\sum_{n=1}^N \lambda^{n-1}\bigl(\mathscr D_t-\mathscr D_{t-n}\bigr),
\qquad \lambda\in(0,1].
\]
Because the result lives in \(\mathbb R^{V\times C}\) regardless of \(N\), the backbone always sees the same feature shape. The method adds a Category-Balanced Loss and an Instance Consistency Loss to address imbalanced object class distributions and motion inconsistency. On Argoverse 2, \(\Delta\)Flow at 5 frames reports dynamic bucket-normalized EPE mean \(=0.113\), compared with Flow4D at \(0.145\), and three-way EPE mean \(=2.11\) cm, compared with \(2.24\) cm. On Waymo, \(\Delta\)Flow at 2 frames reports \(1.64\) cm, versus \(2.03\) cm for Flow4D, with runtime \(18\) s per sequence versus \(33\) s [2508.17054].

Fast multi-frame stereo scene flow with motion segmentation follows a different decomposition. It estimates disparity using SGM, refines 6-DOF camera motion by direct visual odometry, identifies regions inconsistent with the estimated camera motion, computes per-pixel optical flow only at these regions, and fuses the rigid and non-rigid flow proposals by graph-cut. On KITTI 2015 scene-flow, it reports disparity error \(=6.74\%\), optical-flow error \(=12.00\%\), and scene-flow error \(=15.54\%\), with overall rank \#3. Its CPU implementation runs in \(2\)–\(3\) seconds per frame [1707.01307].

## 4. Auxiliary signals, multimodal fusion, and frame-wise representations

“Fusing Frame and Event Vision for High-speed Optical Flow for Edge Application” combines a frame-based Farneback pipeline with an event-stream “leaky CNN” pipeline. The event pipeline maintains a leaky temporal accumulator
\[
A_{ij}(t)=\alpha A_{ij}(t^-)+e_{ij}(t),\qquad 0<\alpha<1,
\]
applies differential kernels to estimate horizontal and vertical motion channels, and smooths them with a \(3\times 3\) average. A fusion block keeps the last frame-based flow and last event-based flow, computes a per-pixel confidence, and selects either the event-based or the most recent frame-based flow. On MVSEC, fusion provides a \(4\times\) higher effective frame-rate with only \(\sim 19\%\) error degradation, from baseline Farneback AEE \(\approx 0.80\) at \(10\) FPS to \(0.95\) after fusion at \(\approx 40\) FPS. In a high-speed drone-flight scenario, the event camera computes the flow even before the optical camera sees the drone [2207.10720].

FFNeRV uses flow information inside frame-wise neural representations for videos. Instead of mapping \((x,y,t)\to\) RGB, it maps a temporal coordinate to an entire frame and splits the output into an “independent” frame \(I(t)\) and a “flow-aggregated” component \(\bar A(t)\) that warps and blends the independent frames of its four temporal neighbors \(\{t-2,t-1,t+1,t+2\}\). The aggregated frame is
\[
\bar A(t)(x)=\sum_{i\in\mathcal N} w'_M(t+i,t)(x)\cdot \mathrm{Warp}(I(t+i),M(t+i,t))(x),
\]
and the final output is
\[
\hat X_t(x)=w_A^*(x)\cdot \bar A(t)(x)+w_I^*(x)\cdot I(t)(x).
\]
With about \(12.4\)M parameters, FFNeRV reports average PSNR \(=37.04\,\text{dB}\) over seven UVG videos, compared with NeRV at \(35.40\,\text{dB}\) and E-NeRV at \(36.39\,\text{dB}\). For frame interpolation on UVG, FFNeRV reports average PSNR \(=32.34\,\text{dB}\), compared with \(26.23\,\text{dB}\) for NeRV and \(26.89\,\text{dB}\) for E-NeRV. With quantization-aware training and entropy coding, it outperforms H.264 and HEVC and performs on par with state-of-the-art video compression algorithms [2212.12294].

These systems broaden the scope of frame flow. In one case, the frame signal is complemented by asynchronous events; in the other, frame flow is built directly into a neural video representation and rate–distortion pipeline.

## 5. Formal-methods and generative uses of “FrameFlow”

Outside vision, “Make flows small again: revisiting the flow framework” develops a flow framework for separation logic reasoning about programs that manipulate general graphs. A flow graph \(h=(X,E,\mathrm{in})\) induces a continuous operator
\[
(F_h(\sigma))(u)=\mathrm{in}_u+\sum_{v\in X}E(v,u)(\sigma(v)),
\]
and by Kleene’s theorem its least fixed point \(\mathrm{flow}_h=\mu F_h=\sup_{n<\omega}F_h^n(\bot)\) exists uniquely. Contextual equivalence is defined by comparing transfer from inflows to outflows: \(h_1\equiv_{\rm ctx} h_2\) iff for all \(\mathrm{in}'\le \mathrm{in}\), \(tf_{h_1}(\mathrm{in}')=tf_{h_2}(\mathrm{in}')\). Because \((FG,*,\equiv)\) is a separation algebra, it can be integrated into Separation Logic in the standard way, yielding a frame rule for heap updates that are frame-preserving. The paper also gives a fixed-point algorithm for automating footprint inference. In the evaluation on \(1\,272\) updates from ten published concurrent-set implementations, the new algorithm succeeded in all cases where plankton did and sped up footprint inference by about \(30\%\) [2304.04886].

“Fast protein backbone generation with SE(3) flow matching” uses the title FrameFlow for a generative model on protein residue frames. On the manifold \(\mathcal M=SE(3)^N\), the model learns a time-dependent vector field \(v_t(x)\in T_x\mathcal M\) so that the push-forward of a prior \(p_0\) under the ODE
\[
\frac{dX_t}{dt}=v_t(X_t),\qquad X_0\sim p_0,
\]
matches the data distribution at \(t=1\). A protein backbone is represented as a sequence of rigid frames \(T=(r,x)\in SO(3)\times \mathbb R^3\), one per residue. Compared to FrameDiff, FrameFlow requires five times fewer sampling timesteps while achieving two fold better designability. At \(100\) ODE steps, FrameFlow reports designability \(=0.77\), versus \(0.39\) for FrameDiff at \(100\) SDE steps and \(0.42\) for FrameDiff at \(500\) SDE steps; on a V100, FrameFlow at length \(100\) takes about \(5.7\) s, versus about \(128\) s for GENIE at \(1\,000\) steps [2310.05297].

A plausible implication is that the word “flow” retains a common structural role across very different domains: it specifies a state evolution law, a transfer function, or a motion field, while “frame” refers either to temporal video frames, logical framing in separation logic, or residue frames in protein geometry.

## 6. Limitations, failure modes, and recurring research questions

Several recurring limitations appear across frame-flow formulations. In Deep Voxel Flow, scenes with highly repetitive textures or ambiguous motion may cause voxel copying to latch onto wrong regions, and the method cannot hallucinate entirely new content outside the support of the input frames, such as large dis-occlusions and new objects [1702.02463]. In FG-DFPN, extremely fast motions that exceed the maximum deformable receptive field can still produce ghosting, and severe occlusions/unocclusions are only implicitly handled because explicit occlusion masks are not modeled [2503.11343].

Dynamic lighting introduces a different failure mode. Real image datasets with ground-truth intermediate frames under dynamic lighting are scarce, so “Frame Interpolation for Dynamic Scenes with Implicit Flow Encoding” focuses on flow EPE for quantitative evaluation and presents extensive visual comparisons. This suggests that photometric variation remains a data and benchmarking bottleneck even when motion estimation is robust [2209.13284].

In high-speed optical flow, frame-based processing is bound by sensor frame-rate and by heavier CNN or optimization latencies for high accuracy, while event-based methods can run at kHz but suffer from noisy, sparse intensity-invariant data and lower accuracy. The frame–event fusion results show that the classic accuracy versus latency trade-off can be mitigated, but not eliminated [2207.10720].

Within Transformer-based interpolation, the original Transformer is commonly used for 2D images; how to develop a Transformer-based framework with consideration of temporal self-attention for video frame interpolation remained an open issue in the formulation of VFIFT. The VFIFT results suggest one practical answer: calculate temporal self-attention in a matched local area with the guidance of flow, rather than use unconstrained global attention [2307.16144].

Across the literature, the central tension is stable: large motion, occlusions, motion blur, dynamic textures, illumination change, and computational overlap all push frame-flow methods away from simple pairwise correspondence. The resulting systems therefore combine motion priors, local or multi-scale matching, temporal propagation, adaptive warping, or alternative modalities to preserve detail while keeping complexity reasonably low.

Source: https://www.emergentmind.com/topics/frame-flow