Weakly Supervised Cross-Modal Learning for 4D Radar Scene Flow Estimation

Abstract: Due to the difficulty of obtaining ground-truth data for 4D radar scene flow estimation, previous methods typically rely on either self-supervised losses or cross-modal supervision using 3D LiDAR data, 2D images, and odometry. However, self-supervised approaches often yield suboptimal results due to radar's inherently low-fidelity measurements, while existing cross-modal supervised methods introduce complex multi-task architecture and require costly LiDAR sensors to generate pseudo radar scene flow labels from pretrained 3D tracking models. To overcome these limitations, we propose a task-specific iterative framework for weakly supervised radar scene flow learning, using only images and odometry for auxiliary supervision during training. Specially, we establish two novel instance-aware self-supervised losses by exploiting off-the-shelf 2D tracking and segmentation algorithms to obtain tracked instance masks, which are back-projected into 3D space to provide instance-level semantic guidance; for static regions, we integrate vehicle odometry with radar's intrinsic motion cues to construct a rigid static loss. Extensive experiments on the real-world View-of-Delft (VoD) dataset demonstrate that our method not only surpasses state-of-the-art cross-modal supervised approaches that rely on 3D multi-object tracking on dense LiDAR point clouds but also outperforms existing fully supervised scene flow estimation methods. The code is open-sourced at \href{https://github.com/FuJingyun/IterFlow}{https://github.com/FuJingyun/IterFlow}.

• The paper introduces IterFlow, a novel architecture for 4D radar scene flow estimation, leveraging weak supervision via RGB images and odometry.
• The IterFlow architecture refines predictions through iterative updates, improving accuracy by 34.7% EPE and reducing model complexity significantly.
• The framework outperforms state-of-the-art methods with minimal parameters, showcasing robust performance even in sparse radar environments.

Weakly Supervised Cross-Modal Learning for 4D Radar Scene Flow Estimation

Motivation and Problem Setting

Scene flow estimation is a fundamental task for dynamic 3D environment understanding, integral to autonomous navigation, motion segmentation, and 3D object detection. While LiDAR-based scene flow methods have benefited from dense, high-fidelity inputs, emerging 4D automotive radar offers robustness under adverse weather and direct radial velocity measurements, but the data is notably sparse and noisy. Obtaining pointwise ground-truth scene flow labels for radar is infeasible, motivating recent self-supervised and cross-modal approaches that leverage LiDAR, RGB, and odometry as auxiliary modalities. However, these either suffer from performance limitations due to radar's low fidelity or demand costly LiDAR sensors and overly complex architectures for pseudo-label generation. The paper proposes a novel weakly supervised cross-modal learning regime, using only RGB images and odometry—ubiquitous and economical sensors—for auxiliary supervision during training, with radar the sole modality at inference.

Figure 1: Comparative illustration of self-supervised (SSF), cross-modal supervised (CMS), and proposed weakly supervised cross-modal settings for radar scene flow.

IterFlow Architecture and Loss Design

The core architectural innovation is IterFlow, an iterative network that refines scene flow predictions for sparse radar input via a GRU-based update scheme. Critically, the method uses ball query-based cross-frame feature correlation instead of traditional KNN grouping, reducing mismatches inherent in sparse radar scenes. Each radar frame passes through a shared multi-scale encoder; across K iterations, points are warped using the latest flow estimates, local neighbors are queried via ball query (radius $R$, $L$ sampled neighbors), and correlation features are generated with spatial locality enforced.

Figure 2: Schematic overview of IterFlow architecture and loss computation pipeline, including iterative refinement and instance-guided losses.

Loss formulation is deliberately concise, consisting of three complementary terms:

• Instance-aware Chamfer Loss ($\mathcal{L}_{ic}$): Instance-level masks from 2D tracking and segmentation are back-projected into 3D radar, assigning each point to a semantic instance or background. Chamfer distance is computed exclusively for pairs within the same instance, mitigating cross-instance mismatches.

  Figure 3: Visualization of mismatched chamfer pairs in radar data, highlighting the confusion between foreground and background when instance guidance is absent.

• Instance-level Flow Smoothness Loss ($\mathcal{L}_{is}$): Conventional spatial flow smoothness using KNN neighborhoods is prone to erroneous regularization across foreground and background. Instead, flow consistency is enforced solely within instances, reducing noise propagation.

  Figure 4: Example of erroneous KNN-based smoothing in sparse radar, where dynamic foreground (cyclist) and static background points are confounded.

• Rigid Static Loss ($\mathcal{L}_{stat}$): Odometry is used to supervise static regions by computing absolute radial velocity, effectively separating static and dynamic zones for targeted regularization.

  Figure 5: Diagram for projecting 2D instance labels onto 3D radar points via tracked segmentation masks.

Experimental Evaluation

Comprehensive experiments are conducted on the View-of-Delft (VoD) dataset, which provides synchronized radar, LiDAR, RGB, and odometry, but not ground-truth scene flow annotations. Labels are generated from 3D tracking boxes for evaluation consistency.

Quantitative Comparison

IterFlow, trained with only three loss terms and weak supervision, consistently outperforms state-of-the-art radar-based methods (e.g., CMFlow [ding2023hidden])—which depend on pseudo-labels from dense LiDAR and multi-modal losses—across all metrics: 34.7% EPE improvement, 13.6%/21.4% AccS/AccR gains, and 39.2% error reduction on SRNE. Notably, IterFlow surpasses even the best fully supervised LiDAR-based models (PV-RAFT [wei2021pv]), achieving an 8.3% EPE improvement.

Model complexity analysis reveals IterFlow requires only $113$K parameters and $0.40$ GFLOPs, a $40\times$/$30\times$ reduction in size and compute compared to CMFlow.

Ablation Studies

Loss ablations demonstrate the necessity of the instance-aware and instance-level losses: replacing $\mathcal{L}_{ic}$ with standard chamfer results in a 21.1% EPE degradation. Substituting KNN smoothness for instance-level smoothing also impairs performance. Iterative inference (K=12) substantially boosts accuracy, as evidenced by the ablation on iteration steps and ball query parameters.

Figure 6: Ablation results for scene flow iteration steps and ball query hyperparameters.

Ball query-based correlation improves robustness compared to KNN, as shown by a 42.5% EPE reduction in sparse radar conditions.

Qualitative & Failure Analysis

Qualitative visualizations on VoD validate iterative refinement and instance-guided supervision, yielding flow vectors tightly aligned with ground truth, especially in challenging sparse regions.

Figure 7: Qualitative results on multiple traffic scenes—IterFlow predictions closely match ground truth flow in both dense and sparse areas.

Failure cases are primarily due to extreme radar sparsity or large foreground displacements, confirming that the primary limitation is intrinsic data quality, not insufficient 2D guidance.

Figure 8: Failure case visualization—regions of large scene flow error pinpointed by yellow circles.

Practical and Theoretical Implications

This work demonstrates that leveraging 2D semantic guidance and odometry, without costly LiDAR pseudo-labels, is not only feasible but superior for radar scene flow estimation. The concise architecture and carefully targeted losses show that task-specific weak supervision can outperform both self-supervised and pseudo-label-based cross-modal regimes. The methodology is robust to different 2D tracking and segmentation backbones, generalizes to diverse traffic conditions, and offers real-time inference capability.

On the theoretical side, instance-aware cross-modal loss formulation reduces the negative transfer from label noise and instance confusion, highlighting the importance of semantic correspondence in scene flow learning for sparse data. Iterative refinement leverages recurrent update mechanisms and spatial locality to overcome the limitations of one-shot prediction in adverse regimes.

Future Directions

Further improvements could involve joint optimization of radar and image modalities, tackling non-rigid object motion more explicitly, or integrating temporal consistency across multiple frames. Extending the framework to other sensor modalities (event cameras, multimodal fusion) and additional downstream tasks (SLAM, motion forecasting) remains an open area.

Conclusion

The paper introduces a weakly supervised cross-modal framework for radar scene flow estimation, capitalizing on 2D semantics and odometry for training supervision. IterFlow achieves state-of-the-art performance with significantly lower computational and annotation cost, reshaping the paradigm for radar-based dynamic environment understanding and providing a practical path for scalable deployment in mainstream autonomous systems (2605.18507).