STARFlow-V: Efficient Multi-Frame Optical Flow

• The paper demonstrates that STARFlow-V uses a double recurrence mechanism across time and scale to efficiently integrate multi-frame motion context and occlusion reasoning.
• It employs a shared encoder and a compact CNN within the STaR cell to refine flow estimates via coarse-to-fine warping and cost-volume construction, achieving state-of-the-art performance with a reduced parameter count.
• The framework’s joint flow and occlusion estimation, validated through extensive ablation studies, results in improved accuracy in occluded regions and a ~60% reduction in parameters compared to non-recurrent approaches.

STARFlow-V refers to a lightweight multi-frame optical flow estimation framework based on a spatiotemporal recurrent network architecture featuring a double recurrence mechanism—across both time and scale—via the repeated application of an identical STaR (SpatioTemporal Recurrent) cell. STARFlow-V efficiently integrates multi-frame motion context and joint occlusion reasoning, achieving state-of-the-art performance among lightweight methods on canonical benchmarks, while maintaining a significantly reduced parameter count relative to competing architectures (Godet et al., 2020).

1. Architectural Overview

STARFlow-V processes a sliding window of $N$ consecutive frames $\{I_1, I_2,\ldots, I_N\}$, extracting deep feature pyramids with $L$ scales per image pair using a shared encoder. At each time step $t$ and pyramid level $l$ (with $l=1$ as the finest), a recurrent STaR cell receives as input the reference-frame features $f^l_1$, target-frame features $f^l_2$, upsampled flow and occlusion from the next coarser scale, and temporally propagated hidden features $H^{t-1}$ summarizing past motion. The cell outputs a refined optical flow $u^l_t$, an occlusion probability map $o^l_t$, and an updated hidden feature $H^t$ to be passed to the next time instant.

These outputs are recursively refined at successive pyramid scales (coarse-to-fine), and final flow/occlusion estimates are produced at the highest resolution. This double recurrence (spatial and temporal) leverages shared weights for parameter efficiency, enforcing regularization and consistent multi-scale reasoning.

2. The STaR Cell

The core STaR cell operates at a single time-step and scale. Its pipeline includes:

• Warping layer: Upsamples the previous coarse flow $u^{l+1}_t \uparrow$ and warps $f^l_2$ toward the reference frame.
• Cost-volume construction: Computes a correlation volume between $f^l_1$ and warped $f^l_2$ over a small neighborhood.
• Compact CNN block: Processes a concatenated tensor $$[f^l_1, \text{warped } f^l_2, \text{cost-volume}, u^{l+1}_t \uparrow, o^{l+1}_t \uparrow, H^{t-1}]$$ through six layers. The output splits into $\Delta u^l_t$ (flow residual), $\hat{o}^l_t$ (occlusion logits), and $H^{\prime}_t$ (hidden features for recurrence).
• Flow update: $u^l_t = u^{l+1}_t \uparrow + \Delta u^l_t$.
• Occlusion: $o^l_t = \text{sigmoid}(\hat{o}^l_t)$.
• Contextual and bilateral refinement: As in IRR-PWC, an additional module sharpens $u^l_t$ and improves motion boundaries.
• Hidden state update: $H^t$ is produced by projecting $H^{\prime}_t$ to fixed channel size (1x1 conv) and warping via the backward flow.

This mechanism enables efficient flow and occlusion estimation through parameter reuse and feature-level temporal modeling (Godet et al., 2020).

3. Double Recurrence: Temporal and Spatial

Temporal Recurrence: At each frame $t$, hidden features $H^{t-1}$, propagated through time, are warped (using an independently estimated backward flow $v^t$) to the current frame’s coordinates and ingested by the current STaR cell. This design conveys high-level context (including acceleration and occlusions) across frames by learned features rather than raw flow estimates, outperforming methods reliant on raw motion-warping for temporal feedback. In practice, a single model is trained for both forward and backward flows by frame order swapping, so $v^t$ is available at inference.

Spatial Recurrence: The same set of STaR cell weights is shared and applied independently at every pyramid level, with bilinear upsampling of flow/occlusion between scales. This coarse-to-fine Iterative Residual Refinement (IRR) approach drastically reduces parameter count—by about 60% versus non-recurrent approaches—without degrading accuracy.

A summary of the recurrent structure at each level $l$:

Step	Operation	Output
Input	$f^l_1, f^l_2, u^{l+1}_t\uparrow$, $o^{l+1}_t\uparrow$, $W(H^{t-1}, v^t)$	---
Warping	$W(f^l_2, u^{l+1}_t\uparrow)$	warped $f^l_2$
Cost Volume	$\text{Corr}(f^l_1, \text{warped }f^l_2)$	cost volume
CNN	Process all features	$\Delta u^l_t, \hat{o}^l_t, H'_t$
Update	$u^l_t = u^{l+1}_t\uparrow + \Delta u^l_t$	$u^l_t$
Occlusion	$o^l_t = \text{sigmoid}(\hat{o}^l_t)$	$o^l_t$
Hidden Prop.	1x1 conv + warp	$H^t$

4. Joint Occlusion Estimation and Losses

Instead of a separate decoder, STARFlow-V appends a single channel for occlusion logits to the compact CNN output. The sigmoid of this channel gives per-pixel occlusion probabilities $o^l_t \in [0,1]$. This integration incurs minimal additional computational cost while yielding improvements in flow accuracy, especially in occluded regions.

The overall loss combines multi-scale supervision at every time step:

$$L = \frac{1}{N}\sum_{t=1}^N L_t, \quad L_t = \sum_{l=1}^L \alpha_l \left[\|u^l_t-u^{l,GT}_t\|_2 + \lambda\,\mathrm{BCE}_w(o^l_t, o^{l,GT}_t)\right]$$

where $\alpha_l$ are pre-set scale weights, BCE$_w$ is class-balanced binary cross-entropy for occlusion, and $\lambda$ is automatically tuned for flow/occlusion balancing.

5. Implementation Details and Performance

Training proceeds in staged phases: initial pretraining on FlyingChairsOcc, multi-frame training on FlyingThings3D (typically $N=4$ frames), and domain-specific fine-tuning on benchmarks (Sintel, KITTI). Data augmentations include cropping, color jitter, blur, and flips. Backward passes use batch sizes $\sim$4–8, with learning rates decayed at scheduled milestones.

State-of-the-art performance is achieved for lightweight methods:

Method	MPI Sintel Clean	MPI Sintel Final	KITTI 2015 Fl-all	Parameters (M)
STARFlow-ft	2.72 px	3.71 px	7.65 %	~4.77
IRR-PWC	3.84 px	4.58 px	7.65 %	6.36
LiteFlowNet2	3.48 px	4.69 px	7.62 %	6.42
ContinualFlow	3.34 px	4.53 px	10.03 %	14.6
ScopeFlow	3.59 px	4.10 px	6.82 %	6.36

STARFlow-V achieves efficient inference (∼0.22s per $1024×436$ pair on a GTX 1070) with a model size of only ~4.77M parameters. Flow estimation is successful even for longer sequences at test time (up to 5–6 frames exploited post-training, despite $N=4$ during training).

6. Motivations, Ablation, and Design Choices

STARFlow-V’s design undergoes extensive ablation analysis to assess the roles of temporal recurrence by learned features (TRFeat), joint occlusion heads, and spatial recurrence. Key findings include:

• Temporal propagation of learned features outperforms flow-based propagation (as in ContinualFlow), reducing EPE by ~6–10% in occluded regions and enabling use of additional future/past frames with ongoing improvements.
• Joint flow and occlusion estimation, simply via added output channel and BCE loss, yields consistent flow gains (~0.2 px EPE improvement).
• Spatial recurrence (weight sharing across scales) cuts parameter count by ~60% with negligible loss in accuracy.
• The overall multi-scale warping, cost-volume, and refinement structure inherits advantageous properties from prior work (PWC-Net, IRR), maintaining sub-pixel flow precision and sharp boundaries.

These findings demonstrate that lightweight architectures with double recurrence and minimal, integrated occlusion reasoning suffice for near state-of-the-art multi-frame optical flow estimation under strict model size constraints (Godet et al., 2020).

7. Significance and Impact

STARFlow-V exemplifies the effectiveness of parameter sharing and spatiotemporal recurrence in multi-frame optical flow without incurring the substantial cost of heavyweight models. The approach is broadly relevant for real-time applications, multi-frame video processing, and systems with resource constraints, as it provides a strong balance of speed, accuracy, and model compactness. Detailed ablation verifies the value of feature-based temporal feedback in exploiting extended motion context and handling occlusions. The successful unification of flow and occlusion estimation within the same decoder further points toward effective joint modeling strategies in related vision tasks.