DepthFlow: Coupling Depth and Motion

• DepthFlow is an umbrella term in computer vision that couples depth estimation and motion analysis using structured light, light fields, and VOS data synthesis.
• It includes methods that extract depth from motion blur in projected patterns, repurpose optical flow across angular views, and generate synthetic flow for unsupervised video object segmentation.
• By exploiting structural correlations between depth and motion, DepthFlow enhances 3D reconstruction accuracy and improves training data for segmentation models.

DepthFlow is a term used for multiple research programs at the interface of depth estimation and motion analysis rather than for a single canonical method. In the literature, it names at least three distinct formulations: a structured-light method that reconstructs 3D shape from the motion blur of projected patterns, called “light flow” (Furukawa et al., 2017); a light-field depth-estimation pipeline that reuses optical flow across angular views to estimate disparity and then depth (Chen et al., 2020); and a synthetic data generation framework for unsupervised video object segmentation (VOS) that estimates monocular depth from a single image and converts it into synthetic optical flow for training RGB+flow models (Cho et al., 26 Jul 2025). Taken together, these uses suggest a broader theme: DepthFlow methods exploit correlations between scene depth and image-plane motion, motion blur, disparity, or motion-like structure instead of treating depth and flow as entirely separate signals.

1. Terminological scope and principal meanings

The term “DepthFlow” is best understood as overloaded nomenclature within computer vision. In one line of work, it denotes a depth-estimation method for fast-moving objects in structured light. In another, it denotes a fast depth pipeline for light fields. In a third, it denotes a data synthesis strategy for VOS rather than a depth estimator at inference time.

Formulation	Core signal	Target task
Structured-light DepthFlow	Ratio of two light flows from blurred projected patterns	3D shape reconstruction of fast-moving objects
Light-field DepthFlow	Optical flow across angular views interpreted as disparity	Dense depth map and 3D reconstruction from light fields
VOS DepthFlow	Depth-to-flow synthesis from single RGB images	Image-flow-mask triplet generation for unsupervised VOS training

A common misconception is that DepthFlow always denotes “depth from optical flow.” The structured-light formulation instead uses the motion blur of projected patterns as signal, while the VOS formulation explicitly does not aim to reconstruct true physical motion and instead preserves structural cues such as foreground/background contrast, motion boundaries, and spatial layout (Furukawa et al., 2017).

2. Structured-light DepthFlow: depth from projected-pattern motion blur

In “Depth estimation using structured light flow -- analysis of projected pattern flow on an object's surface” (Furukawa et al., 2017), DepthFlow is a structured-light depth-estimation approach that deliberately uses the motion blur of projected structured-light patterns rather than treating blur as an error. The blurred motion of a projected pattern on an object’s surface is termed light flow. The method addresses 3D shape reconstruction of fast-moving objects under conditions in which standard one-shot structured light becomes unstable or fails because the projected pattern cannot be captured clearly.

The theoretical core is that a single light flow is underdetermined. In the general analysis, apparent pattern motion depends on surface depth $z$, surface normal $n$, and depth velocity $\frac{dz}{dt}$ through

$\frac{dp}{dt} = F(z,n)\frac{dz}{dt}.$

With one projector, only one equation is available,

$\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt},$

so depth cannot be solved in the general case. With two projectors,

$$\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt}, \qquad \frac{dp_2}{dt} = F_2(z,n)\frac{dz}{dt},$$

and taking the ratio eliminates the velocity term:

$$\frac{dp_1/dt}{dp_2/dt} = \frac{F_1(z,n)}{F_2(z,n)}.$$

The paper states that at least two light flows are required for depth estimation, while one flow alone is underdetermined.

The practical system uses one camera, two video projectors, and two sets of parallel line patterns, with different colors for separating the two patterns in the captured image. Because the projected patterns are static, synchronization is not required. A single-projector special case is also described, using two non-uniformly spaced line sets. In the two-projector case, the blur of each projected line is measured as the width of the blurred band in the camera image. With $\Delta r$ denoting the width of motion blur in the captured image and $B$ the apparent interval between neighboring lines in the same local region, the light-flow magnitude is approximated by

$\Delta q_h \approx \frac{\Delta r}{B}.$

Depth recovery is then formulated through a calibrated mapping from the ratio of two observed light flows to scene depth. In the simplified derivation, if $n$0, then $n$1. With two projectors, the ratio becomes approximately $n$2, and the paper defines

$n$3

so that depth is recovered by inversion:

$n$4

In implementation, the function $n$5 is sampled for each pixel or image position $n$6 using calibrated projector-camera geometry, and depth is obtained by lookup or interpolation.

The experiments include a planar board, a rotating fan, a thrown ball, and a two-ball configuration for the single-projector variant. The reported outcome is that 3D shapes of fast-moving objects, which are inevitably captured with motion blur, are successfully reconstructed. The method’s limitations are equally explicit: it requires accurate blur measurement, spatial resolution is limited by line spacing, estimation fails when blur bands overlap, projector-camera geometry can become degenerate when $n$7 is nearly flat, and the line-based formulation is sensitive to textures, binarization errors, and boundary outliers.

3. DepthFlow for unsupervised VOS: depth-to-flow synthetic data generation

In “DepthFlow: Exploiting Depth-Flow Structural Correlations for Unsupervised Video Object Segmentation” (Cho et al., 26 Jul 2025), DepthFlow is not a depth estimator at deployment time but a synthetic data generation framework for unsupervised VOS. Its premise is that two-stream RGB+flow VOS models are constrained by the scarcity of complete image-flow-mask datasets, whereas large image saliency datasets provide masks but no flow. The method therefore estimates a depth map from a single source image and converts it into a synthetic optical flow field, thereby transforming image-mask pairs into image-flow-mask triplets.

The pipeline begins with a monocular depth estimator, DPT-Hybrid. Given an input image $n$8, raw depth is min-max normalized:

$n$9

Synthetic motion is then generated through random depth reversal,

$\frac{dz}{dt}$0

random value shifting,

$\frac{dz}{dt}$1

random value scaling,

$\frac{dz}{dt}$2

and normalization,

$\frac{dz}{dt}$3

The resulting two-channel motion map is converted into an RGB optical-flow visualization by

$\frac{dz}{dt}$4

The paper is explicit that the goal is not geometrically correct optical flow. Instead, the synthetic flow preserves structural cues that matter for VOS: spatial ordering of regions, foreground/background contrast, object boundary structure, global motion-like gradients, and relative flow patterns. This distinction is central. The method argues that unsupervised VOS models depend more on structural information embedded in flow maps than on geometric accuracy.

Training uses a simple encoder-decoder two-stream architecture similar to TMO. One encoder processes RGB, another processes flow, features are fused at multiple layers by element-wise addition, a decoder predicts the segmentation mask, and CBAM attention modules are inserted after each fusion step. The training objective uses cross-entropy loss with Adam and learning rate $\frac{dz}{dt}$5. The training protocol consists of pre-training on YouTube-VOS 2018, with all annotated objects merged into a single binary salient object mask, followed by fine-tuning on DAVIS 2016 training data and the synthetic DUTSv2 dataset with mixing ratio 1:3.

The scale expansion is substantial: the paper reports 2,079 triplets with only 30 visual contexts for original real VOS training data, 15,572 triplets with 15,572 visual contexts for synthetic DUTSv2, and 17,651 triplets for the mixed real + synthetic set. Benchmark results are correspondingly strong. On DAVIS 2016 validation, DepthFlow achieves $\frac{dz}{dt}$6, $\frac{dz}{dt}$7, and $\frac{dz}{dt}$8; on FBMS it reaches $\frac{dz}{dt}$9; on YouTube-Objects, mean $\frac{dp}{dt} = F(z,n)\frac{dz}{dt}.$0; and on Long-Videos, $\frac{dp}{dt} = F(z,n)\frac{dz}{dt}.$1. On video SOD benchmarks, it reports average $\frac{dp}{dt} = F(z,n)\frac{dz}{dt}.$2, average $\frac{dp}{dt} = F(z,n)\frac{dz}{dt}.$3, and average $\frac{dp}{dt} = F(z,n)\frac{dz}{dt}.$4.

4. Depth–flow coupling as a broader methodological pattern

The broader literature suggests that DepthFlow is part of a larger family of methods that explicitly couple geometry and motion. “Dense Scene Flow from Stereo Disparity and Optical Flow” represents scene flow in image space through optical flow, stereo disparity, and disparity change, and reconstructs dense scene flow by combining disparity at time $\frac{dp}{dt} = F(z,n)\frac{dz}{dt}.$5, disparity at time $\frac{dp}{dt} = F(z,n)\frac{dz}{dt}.$6, and optical flow, then filling holes with SceneFlowFields interpolation (Schuster et al., 2018). Its key recombination step is

$\frac{dp}{dt} = F(z,n)\frac{dz}{dt}.$7

with disparity change

$\frac{dp}{dt} = F(z,n)\frac{dz}{dt}.$8

The raw recombination is sparse because of occlusions and pixels leaving the image domain, so geometry interpolation uses local planes and motion interpolation uses local affine transformations to obtain a dense field.

“DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency” makes the coupling explicit during training (Zou et al., 2018). It jointly optimizes a Depth Net, Pose Net, and Flow Net on unlabeled monocular video. In rigid regions, predicted scene depth and camera motion synthesize a rigid optical flow:

$\frac{dp}{dt} = F(z,n)\frac{dz}{dt}.$9

A cross-task consistency loss then enforces agreement between rigid flow and learned optical flow on pixels valid for both branches:

$\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt},$0

This formulation is instructive because it shows one way in which depth and flow can supervise one another without external labels.

“FlowDepth: Decoupling Optical Flow for Self-Supervised Monocular Depth Estimation” extends the same logic to dynamic driving scenes (Sun et al., 2024). Its Dynamic Motion Flow Module decouples total optical flow into static and dynamic components,

$\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt},$1

so that moving objects can be warped and the mismatch induced by the static-scene assumption is reduced. It complements this with Depth-Cue-Aware Blur for high-frequency regions and a cost-volume sparse loss for low-texture regions. This suggests that contemporary depth–flow research increasingly uses flow not merely as an auxiliary output but as a mechanism for correcting geometric inconsistencies during self-supervision.

5. Light-field DepthFlow and adjacent focus-based interpretations

In “Fast and Accurate Optical Flow based Depth Map Estimation from Light Fields” (Chen et al., 2020), DepthFlow denotes a fast light-field depth-estimation pipeline. The method uses the two-plane parametrization

$\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt},$2

treats a sequence of rectified sub-aperture images along one angular dimension as an image sequence, applies optical flow between adjacent angular views, interprets the horizontal displacement as disparity, and converts disparity into depth using

$\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt},$3

The pipeline extracts a 3D spatio-angular volume, uses Coarse-to-fine Patch Matching (CPM) for initialization, refines with feature flow, and then aggregates multiple disparity estimates either in the image plane or after conversion into depth. CPM is applied on consecutive pairs $\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt},$4 with reported parameters $\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt},$5, $\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt},$6, and patch size $\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt},$7. Feature flow is favored because its spatio-angular edge-aware filtering yields disparity estimates that are consistent enough for simple median aggregation followed by one-step variational refinement derived from EpicFlow. The alternative 3D aggregation converts each disparity map to depth first and fuses directly in 3D, producing point clouds with more than 21 million points, compared with about 260k points from a single depth map.

A related but differently named line appears in depth-from-focus. “Deep Depth from Focus with Differential Focus Volume” constructs a focus volume over focal distance, then replaces plain stacking with a Differential Focus Volume obtained by first-order differencing along the frame dimension (Yang et al., 2021). The depth estimate is the expectation over focal distances,

$\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt},$8

and uncertainty is the standard deviation of the predicted focus distribution. Although this work is not titled DepthFlow, it reinforces the same broader idea: depth can be recovered from structured changes across an ordered image dimension, whether that dimension is angular viewpoint, focus setting, or motion blur.

6. Later developments, related nomenclature, and conceptual boundaries

Later work expands the depth–flow connection in directions that are adjacent to, but distinct from, the named DepthFlow methods. “DepthFocus: Controllable Depth Estimation for See-Through Scenes” replaces the conventional one-static-depth-per-pixel paradigm with a steerable stereo model conditioned on a scalar depth preference $\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt},$9 (Min et al., 21 Nov 2025). It is built on the stereo backbone S$$\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt}, \qquad \frac{dp_2}{dt} = F_2(z,n)\frac{dz}{dt},$$0M$$\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt}, \qquad \frac{dp_2}{dt} = F_2(z,n)\frac{dz}{dt},$$1, uses conditional Mixture-of-Experts routing,

$$\frac{dp_1}{dt} = F_1(z,n)\frac{dz}{dt}, \qquad \frac{dp_2}{dt} = F_2(z,n)\frac{dz}{dt},$$2

and introduces a newly constructed ~500k stereo-pair synthetic dataset together with new synthetic and real multi-layer benchmarks. The paper reports strong performance on BOOSTER, including reflective/transmissive regions, and uses the LayeredFlow benchmark for generalization studies. This does not redefine DepthFlow itself, but it places depth-conditioned reasoning into layered, see-through scenes where a single depth is insufficient.

“FlowSeek: Optical Flow Made Easier with Depth Foundation Models and Motion Bases” in turn uses a monocular depth foundation model, specifically Depth Anything v2, together with a low-dimensional motion basis to improve optical flow (Poggi et al., 5 Sep 2025). It is trained on a single RTX 3090 GPU, a hardware budget about 8x lower than most recent methods according to the paper, and reports relative improvements of 10% on Sintel Final and 15% on KITTI over SEA-RAFT in cross-dataset generalization. Its motion basis representation is derived from inverse depth and classical rigid-motion subspace structure, indicating another route by which depth priors simplify flow estimation.

The related nomenclature is therefore important. DeFlow is a self-supervised 3D motion estimation system for debris flows using a static camera-LiDAR setup (Zhu et al., 2023). DF-Net is an unsupervised framework for jointly learning monocular depth and optical flow from unlabeled video (Zou et al., 2018). FlowDepth is a self-supervised monocular depth method that decouples optical flow into static and dynamic components (Sun et al., 2024). These systems are conceptually allied to DepthFlow because they couple depth and motion, but they are distinct models with different objectives, sensors, and assumptions.

A precise summary is therefore necessarily plural. In structured light, DepthFlow means recovering depth from the ratio of two blur-induced projected-pattern flows for fast-moving objects (Furukawa et al., 2017). In light fields, it means repurposing optical flow over angular views as disparity estimation (Chen et al., 2020). In VOS, it means synthesizing structurally useful flow from monocular depth to scale RGB+flow training data (Cho et al., 26 Jul 2025). The common thread is not a single architecture or benchmark, but the recurring hypothesis that depth and flow share exploitable structure.