Sensor-Guided Optical Flow

• Sensor-guided optical flow is a technique that integrates sparse sensor-derived hints such as depth and pose into deep networks to enhance motion estimation.
• It modulates correlation volumes in architectures like RAFT to overcome issues like occlusions, low texture, and domain shifts.
• Hybrid approaches merge classical sensor fusion with learned models, benefiting applications from robotic navigation to UAV flight control.

Sensor-guided optical flow denotes a class of methods in which optical-flow estimation is constrained, regularized, or disambiguated by auxiliary sensing rather than inferred from image pairs alone. In the formulation introduced in "Sensor-Guided Optical Flow" (Poggi et al., 2021), sparse yet accurate optical flow hints from an external source are injected into the correlation scores of a state-of-the-art optical flow network, guiding the estimator toward more accurate predictions on both known and unseen domains. In a broader robotics and vision context, related systems use depth, pose, inertial measurements, proprioception, event timing, and on-sensor motion hardware to shape either the flow estimate itself or the downstream state estimate derived from flow (Meronen et al., 2020).

1. Conceptual Foundations

The central problem addressed by sensor-guided optical flow is that purely image-based flow estimation is often fragile under domain shift, low texture, occlusion, independently moving objects, or aggressive resource constraints. The 2021 framework formalizes guidance as an external cue that is sparse in space but accurate enough to bias the network’s internal matching process toward plausible displacements (Poggi et al., 2021).

A common misconception is that sensor guidance requires a physical sensor that directly outputs pixelwise flow. The 2021 framework explicitly states the opposite: no real sensor can provide sparse flow hints directly, and the hints must instead be obtained by combining depth measurements from active sensors with geometry and hand-crafted optical flow algorithms. In that sense, the method is not a replacement for learned optical flow; it is a hybrid estimator in which sensing-derived priors reshape the network’s search space.

The broader literature shows that “guidance” is not limited to external depth. In mobile robotics, a sensorimotor model can condition future optical flow on past action and proprioception, with state variables of the form $$X_t \equiv \{OF_{t-\Delta}, A_{t-\Delta}, V_{t-\Delta}\}$$ and a predictive distribution over future flow (Ribes et al., 2012). In visual-inertial tracking, dense learned flow can be fused with IMU data in an EKF-style update, where the inertial state prediction guides interpretation of the flow field and per-pixel flow uncertainty controls the measurement weighting (Meronen et al., 2020). These variants do not implement the same correlation-volume modulation as the 2021 paper, but they establish sensor guidance as a general design pattern: optical flow becomes a latent variable whose estimation is improved by non-visual information.

2. Correlation-Volume Guidance in Deep Optical Flow

The canonical sensor-guided architecture begins with a modern optical-flow network such as RAFT and inserts the external cue at the all-pairs correlation stage. Two images $I_0, I_1$ are encoded into feature maps $F_0, F_1$ at quarter resolution, a 4D correlation volume $C$ is computed, and each 2D slice $C_p$ is modulated by a Gaussian-shaped mask centered on the hint $H_p$ whenever a valid hint exists. The recurrent update machinery then operates on the modulated volume $C'$ rather than the original $C$.

The modulation is defined as

$$C'_p(x,y) = (1-v_p)\cdot C_p(x,y) + v_p\cdot k\cdot \exp\!\left(-\frac{(x-x_p^*)^2+(y-y_p^*)^2}{2c^2}\right)\cdot C_p(x,y),$$

where $v_p \in \{0,1\}$ indicates whether a hint is available at pixel $I_0, I_1$0, $I_0, I_1$1 is the hint location in displacement space, and $I_0, I_1$2 are hyper-parameters. In the reported experiments, $I_0, I_1$3 and $I_0, I_1$4. The rest of the network is unchanged: a GRU-based recurrent update block consumes the guided correlation representation, and convex upsampling produces the full-resolution flow estimate.

The paper further argues that RAFT’s default correlation at $I_0, I_1$5 resolution is too coarse for sparse 2D modulation, and therefore introduces Quarter-res RAFT, or QRAFT. The 6th convolution in the encoder changes stride from $I_0, I_1$6 so that features are computed at $I_0, I_1$7 rather than $I_0, I_1$8 resolution, and channels are reduced from $I_0, I_1$9 to keep GPU memory in check. Upsampling is modified accordingly by predicting a $F_0, F_1$0 reconstruction mask at quarter resolution instead of an $F_0, F_1$1 mask at one-eighth resolution. Training remains end to end with the standard weighted multi-step endpoint error loss,

$F_0, F_1$2

and no additional hint-consistency loss is introduced (Poggi et al., 2021).

3. Constructing Sparse Flow Hints from Sensors

Because no real sensor outputs the required sparse 2D hints directly, the 2021 framework constructs a “virtual flow sensor” from several components. For ego-motion flow, depth $F_0, F_1$3 is measured at pixels in $F_0, F_1$4 using LIDAR or ToF, the relative pose $F_0, F_1$5 is estimated by solving a PnP problem over image-depth correspondences with RANSAC, and each pixel is back-projected, transformed, and re-projected:

$F_0, F_1$6

The resulting ego-motion flow is then defined as $F_0, F_1$7.

The pipeline filters outliers by a forward-backward check. A backward flow $F_0, F_1$8 is computed, warped forward, and accepted only if

$F_0, F_1$9

To increase the support of the consistency mask, depth is densified by fast marching or bilateral inpainting before warping. Dynamic-object flow is estimated separately by a hand-crafted method, specifically RICFlow, and subjected to the same forward-backward consistency filtering. A Mask-RCNN semantic mask then combines the two sources: background pixels use the ego-motion flow, while object pixels use the hand-crafted flow. On KITTI-142 this pipeline yields a sparse, accurate hint set with density $C$0–$C$1 and EPE $C$2 px (Poggi et al., 2021).

Related robotics systems provide a useful contrast. A ground-related odometry sensor for micro aerial vehicles combines ground-facing optical flow from the Raspberry Pi SoC’s Coarse Motion Estimator with an ultrasonic height sensor and, optionally, a 3-axis MEMS gyro; metric velocities are obtained by a pinhole projection model and robust aggregation via RANSAC (Écorchard et al., 2019). Earlier quadrotor work also demonstrated hovering control using a single-chip optical flow sensor conventionally used for desktop mice, implemented on an 8-bit microprocessor without external sensors or communication with a ground control station (Lim et al., 2012). These systems do not supply correlation-volume hints to a learned network, but they show that sensor-derived flow priors and flow-distance fusion predate the deep hint-injection formulation.

4. Training Protocol and Quantitative Performance

The training schedule for QRAFT follows a staged RAFT-style regime. Pre-training uses FlyingChairs for 100 K steps with batch size $C$3 and crop $C$4, followed by FlyingThings3D for 100 K steps with batch size $C$5 and crop $C$6. Fine-tuning runs for 50 K steps each on Sintel at $C$7 or KITTI-58 at $C$8. When training in guided mode, each iteration samples $C$9 of ground-truth pixels, perturbs their offsets with uniform noise in $C_p$0 px, constructs $C_p$1 and $C_p$2, forms the guided correlation $C_p$3, and proceeds with the standard forward pass.

Even before test-time guidance, QRAFT improves over RAFT in the reported single-GPU runs. With $C_p$4 pretraining, RAFT reports Sintel Clean $C_p$5, Final $C_p$6, Middlebury EPE $C_p$7, KITTI ’12 EPE $C_p$8 with F1 $C_p$9, and KITTI-142 EPE $H_p$0 with F1 $H_p$1, whereas QRAFT reports Sintel Clean $H_p$2, Final $H_p$3, Middlebury EPE $H_p$4, KITTI ’12 EPE $H_p$5 with F1 $H_p$6, and KITTI-142 EPE $H_p$7 with F1 $H_p$8. Under simulated hints at $H_p$9 density and $C'$0 px, the same $C'$1 model improves further to Sintel Clean $C'$2, Final $C'$3, Middlebury EPE $C'$4, KITTI ’12 EPE $C'$5 F1, and KITTI-142 EPE $C'$6 with F1 $C'$7 (Poggi et al., 2021).

The sensor-guided test-time results on KITTI-142 are summarized below.

Pretraining	KITTI-142 EPE without hints	KITTI-142 EPE with sensor hints
C	9.61	5.88
C+T	6.21	4.55
C+T+S	5.02	4.32
C+T+K	2.58	2.08

These figures show two distinct effects. First, quarter-resolution correlation alone improves the baseline architecture. Second, sparse guidance remains beneficial both in cross-domain settings, where the gain is large, and in fine-tuned settings, where the absolute gain is smaller but still measurable. The paper’s qualitative examples further report correction of large errors under occlusions, in low texture, and on independently moving objects (Poggi et al., 2021).

5. Related Sensor-Guided Paradigms

A major neighboring paradigm is dense flow assisted inertial navigation. In "Movement Tracking by Optical Flow Assisted Inertial Navigation," consecutive frames are processed by FlowNet 2.0, Monte Carlo dropout in the final fusion layers yields per-pixel standard deviations $C'$8, and these uncertainties define the covariance of each EKF measurement update. The state includes position, velocity, orientation, IMU biases, accelerometer scale errors, and the previous camera pose; each selected flow pixel contributes a reprojection or triangulation-based constraint. On real-world data acquired by an iPad in a very low-texture corridor, GNSS alone exhibits $C'$9 m drift over a $C$0 m loop, Apple ARKit shows residual $C$1–$C$2 m absolute error, and the flow-assisted EKF reports average $C$3–$C$4 m error (Meronen et al., 2020).

Another paradigm uses action and proprioception as the guidance signal. "Sensory Anticipation of Optical Flow in Mobile Robotics" learns an online Gaussian-mixture model of the joint distribution over past optical flow, past action, and past proprioceptive reading, then predicts future optical flow by conditioning on the active component. The reported empirical result is that including action reduces AEPE by roughly $C$5 over the naïve predictor, nearly independently of the number of GMM components, and that the learned collision signal can predict bumper-trigger events up to $C$6 s before they occur (Ribes et al., 2012).

Event-based vision has produced several additional forms of sensor guidance. DistSurf-OF constructs a distance surface $C$7 from asynchronous events and then treats $C$8 as analogues of intensity gradients in a Horn–Schunck-style variational estimator; with denoising, the reported results include checkerboard RAEE $C$9, AAE $$C'_p(x,y) = (1-v_p)\cdot C_p(x,y) + v_p\cdot k\cdot \exp\!\left(-\frac{(x-x_p^*)^2+(y-y_p^*)^2}{2c^2}\right)\cdot C_p(x,y),$$0, classroom RAEE $$C'_p(x,y) = (1-v_p)\cdot C_p(x,y) + v_p\cdot k\cdot \exp\!\left(-\frac{(x-x_p^*)^2+(y-y_p^*)^2}{2c^2}\right)\cdot C_p(x,y),$$1, AAE $$C'_p(x,y) = (1-v_p)\cdot C_p(x,y) + v_p\cdot k\cdot \exp\!\left(-\frac{(x-x_p^*)^2+(y-y_p^*)^2}{2c^2}\right)\cdot C_p(x,y),$$2, and reductions of AAE by more than $$C'_p(x,y) = (1-v_p)\cdot C_p(x,y) + v_p\cdot k\cdot \exp\!\left(-\frac{(x-x_p^*)^2+(y-y_p^*)^2}{2c^2}\right)\cdot C_p(x,y),$$3 relative to state-of-the-art DVS flow methods in complex scenes (Almatrafi et al., 2020). SOFAS, by contrast, represents events as a 3D Event Contour Extrusion in $$C'_p(x,y) = (1-v_p)\cdot C_p(x,y) + v_p\cdot k\cdot \exp\!\left(-\frac{(x-x_p^*)^2+(y-y_p^*)^2}{2c^2}\right)\cdot C_p(x,y),$$4 space and searches for the velocity-aligned projection that maximizes a sharpness metric $$C'_p(x,y) = (1-v_p)\cdot C_p(x,y) + v_p\cdot k\cdot \exp\!\left(-\frac{(x-x_p^*)^2+(y-y_p^*)^2}{2c^2}\right)\cdot C_p(x,y),$$5; on synthetic ground-truth datasets it reports magnitude errors $$C'_p(x,y) = (1-v_p)\cdot C_p(x,y) + v_p\cdot k\cdot \exp\!\left(-\frac{(x-x_p^*)^2+(y-y_p^*)^2}{2c^2}\right)\cdot C_p(x,y),$$6–$$C'_p(x,y) = (1-v_p)\cdot C_p(x,y) + v_p\cdot k\cdot \exp\!\left(-\frac{(x-x_p^*)^2+(y-y_p^*)^2}{2c^2}\right)\cdot C_p(x,y),$$7 and angle errors $$C'_p(x,y) = (1-v_p)\cdot C_p(x,y) + v_p\cdot k\cdot \exp\!\left(-\frac{(x-x_p^*)^2+(y-y_p^*)^2}{2c^2}\right)\cdot C_p(x,y),$$8–$$C'_p(x,y) = (1-v_p)\cdot C_p(x,y) + v_p\cdot k\cdot \exp\!\left(-\frac{(x-x_p^*)^2+(y-y_p^*)^2}{2c^2}\right)\cdot C_p(x,y),$$9, compared with Lucas–Kanade magnitude errors $v_p \in \{0,1\}$0 and angle errors $v_p \in \{0,1\}$1–$v_p \in \{0,1\}$2 (Stoffregen et al., 2018). A more recent event-camera tracking system uses optical flow to guide 2D–3D corner-edge association for iterative SE(3) pose refinement, reporting lower rotation and translation errors than nearest-neighbor and line-based baselines in both simulated and real-event sequences (Liu et al., 24 Dec 2025).

6. Edge Hardware, Robotics Deployment, and Limitations

Sensor-guided optical flow has immediate consequences for embedded deployment. The VD56G3 optical-flow camera integrates a $v_p \in \{0,1\}$3 MP global-shutter imager with an ASIC that performs feature-based optical flow estimation. The reported operating points include up to $v_p \in \{0,1\}$4 fps at the full resolution of $v_p \in \{0,1\}$5 pixels and up to $v_p \in \{0,1\}$6 fps at a reduced camera resolution of $v_p \in \{0,1\}$7, with optical-flow power between $v_p \in \{0,1\}$8 mW and $v_p \in \{0,1\}$9 mW depending on the mode. The ASIC implements FAST corner detection, BRIEF descriptor generation, Hamming-distance matching, and optional ratio-test filtering, and outputs up to $I_0, I_1$00 simultaneous flow vectors per frame within a VGA-sized window (Kühne et al., 2023).

The same hardware has been integrated into VINS-Mono by replacing the CPU-based feature tracker with ASIC-generated flow vectors. The resulting OF VINS-Mono reports a $I_0, I_1$01 reduction in latency and a $I_0, I_1$02 reduction of compute load over the original implementation, allowing operation up to $I_0, I_1$03 FPS instead of $I_0, I_1$04 FPS on the quad-core ARM Cortex-A72 processor of a Raspberry Pi Compute Module 4. Average end-to-end latency drops from $I_0, I_1$05 ms to $I_0, I_1$06 ms, and total system power with VIO drops from $I_0, I_1$07 W to $I_0, I_1$08 W (Kühne et al., 2024).

Optical-flow-guided control has also been demonstrated in aggressive flight. "Seeing Through Pixel Motion" uses NeuFlow for dense flow estimation at $I_0, I_1$09 Hz on a Jetson, downsamples flow to a $I_0, I_1$10 grid as a low-pass filter, and couples a central flow attention mechanism with an action-guided active sensing strategy that steers the high-resolution crop toward the direction of motion. In simulation, the ground-truth-flow policy achieves $I_0, I_1$11 success at $I_0, I_1$12 m/s and the NeuFlow-based policy achieves $I_0, I_1$13 at $I_0, I_1$14 m/s; in zero-shot sim-to-real transfer on an FPV racing drone, the system reports peak speed $I_0, I_1$15 m/s and more than $I_0, I_1$16 collision-free performance in $I_0, I_1$17 trials per environment across dense forest, sparse forest, tree branches, and cluttered indoor scenes (Hu et al., 2024).

The limitations of sensor-guided optical flow are correspondingly heterogeneous. In the 2021 correlation-modulation framework, the main limitation is upstream: hints must be synthesized from depth, pose, classical optical flow, and semantic masks because no physical sensor provides them directly (Poggi et al., 2021). In on-sensor flow hardware, the output is sparse, there is no sub-pixel or iterative refinement, and loop closure may still require host-side descriptor extraction (Kühne et al., 2024). In flow-based quadrotor control, optical-flow ambiguity under pure rotation and near the Focus of Expansion remains a challenge at very high speeds above $I_0, I_1$18 m/s (Hu et al., 2024). In event-based variational methods, the dominant bottleneck can remain the flow solver rather than the event representation itself (Almatrafi et al., 2020). Taken together, these results indicate that sensor guidance is not a single algorithmic trick but a spectrum of mechanisms for constraining motion estimation under real sensing, compute, and control constraints.