RAVEN: Radar Adaptive Vision Encoders for Efficient Chirp-wise Object Detection and Segmentation

Abstract: This paper presents RAVEN, a computationally efficient deep learning architecture for FMCW radar perception. The method processes raw ADC data in a chirp-wise streaming manner, preserves MIMO structure through independent receiver state-space encoders, and uses a learnable cross-antenna mixing module to recover compact virtual-array features. It also introduces an early-exit mechanism so the model can make decisions using only a subset of chirps when the latent state has stabilized. Across automotive radar benchmarks, the approach reports strong object detection and BEV free-space segmentation performance while substantially reducing computation and end-to-end latency compared with conventional frame-based radar pipelines.

• The paper introduces RAVEN, a unified architecture leveraging state space models and K-pooling to compress fast-time ADC radar samples for efficient detection and segmentation.
• It employs adaptive chirp selection with early-exit inference, achieving improved mAP (+0.849%) and mIoU (+0.672%) while reducing computational load.
• RAVEN demonstrates hardware-agnostic generalization and robust multi-task learning, outperforming SOTA methods across diverse MIMO configurations.

RAVEN: Radar Adaptive Vision Encoders for Efficient Chirp-wise Object Detection and Segmentation

Introduction

The paper introduces RAVEN, a unified architecture for FMCW radar perception that exploits adaptive token compression and chirp-wise selection using state space models (SSMs) for efficient, joint object detection and semantic segmentation from raw radar signals. RAVEN addresses challenges in compressing high-dimensional fast-time ADC radar samples, mitigating multi-task interference, supporting variable MIMO configurations, and enabling early-exit inference for computational efficiency.

Efficient Chirp-wise Representation and Compression

RAVEN leverages per-RX SSMs, based on Mamba-style state spaces, to encode fast-time (ADC) radar samples into compact latent representations that preserve range information. The architecture condenses samples from each antenna into a single token via a learned K-pooling operation, with ablation demonstrating minimal F1-score degradation when compressing to a single token (F1: 0.934 with $K=1$ and 0.937 with $K=16$), supporting the radical compression hypothesis. This design achieves efficient multiplexing of DDM and virtual MIMO array formation, with lightweight token sequences fed into the cross-antenna mixer for downstream processing.

Adaptive Chirp Selection and Early Exit Inference

The model employs an adaptive chirp selection mechanism to dynamically terminate chirp processing based on scene predictability. Unlike fixed-chirp models, RAVEN learns to select the number of chirps required per frame. Analysis of selected chirp counts reveals no significant correlation with target/dominant object velocity, indicating that the policy is driven by task confidence and prediction stability rather than raw motion cues.

Figure 1: (a) Distribution of target velocities; (b) Relationship between adaptive chirp counts and observed velocities, indicating scene dynamics do not dictate chirp selection.

For early exit, RAVEN utilizes a cosine-similarity-based criterion to decide when sufficient information is accrued, yielding higher mAP (+0.849%) and mIoU (+0.672%) compared to entropy-based and alternative risk control strategies, while maintaining comparable computational complexity. This method aligns with recent trends in resource-aware deep model deployment and offers practical acceleration for radar perception stacks.

Multi-task Learning and Joint Objective Optimization

Empirical results demonstrate that joint training of detection and segmentation heads avoids explicit gradient interference; RAVEN surpasses task-specific baselines (Detection: 0.95 vs 0.93 mAP, Segmentation: 90.2% vs 90.1% mIoU), reflecting robust multi-task feature sharing without harming representation quality. The detection head follows anchor-free, BEV-based decoding, predicting confidence and object offsets directly from learned embeddings in a manner akin to anchor-free methods.

Class-wise breakdown on the RaDICaL benchmark confirms consistently high performance across both vehicle and pedestrian categories (e.g., F1: 97.8% vehicle, 95.2% pedestrian; Chamfer distances 0.082–0.085), challenging the notion that radar-based models must be heavily class tuned.

Hardware-Agnostic Generalization and Task Transferability

RAVEN is independent of underlying radar MIMO hardware configuration. The model processes variable-length sequences of channel tokens and generalizes to different MIMO architectures (e.g., 3Tx, 4Rx on RADDet), outperforming SOTA methods such as T-FFTRadNet by significant margins (67.0% F1 vs 60.8%). Notably, high pedestrian detection F1 on RaDICaL attests to its viability for diverse traffic participants and generalizes beyond vehicle-centric datasets.

Architectural Hyperparameters and Implementation Details

Key model components include a multi-head attention-based antenna mixer (dim: 64, 8 heads, expansion 4×), SSM layers (state dim: 16, kernel: 4, expansion: 2), and spatial projection to a high-dimensional BEV grid. All parameter choices are fully documented, supporting reproducibility and facilitating transfer to related settings.

Implications and Future Directions

RAVEN's contributions are manifold: evidence that aggressive fast-time compression is viable; demonstration of adaptive, explainable chirp-wise compute allocation; elimination of task competition in multi-task radar perception; and practical advances in radar-centric autonomy across hardware and object classes. The proposed early-exit approaches and SSM-based modeling schemes suggest promising cross-domain transfer potential within multimodal, sequential sensor fusion for real-time robotics and automation.

Future work could extend these paradigms to 3D radar labels, robust temporal sequence modeling (potentially leveraging recurrent SSMs), and integration of radar with complementary modalities like vision and LiDAR under the same compute- and task-adaptive framework. Expanding benchmarking across environmental domains and sensor configurations may further solidify practical adoption and inform broader autonomous system design.

Conclusion

RAVEN provides a highly efficient and modular foundation for joint radar-based object detection and segmentation, combining aggressive data compression, hardware abstraction, and dynamic inference. Empirical validations confirm both competitive performance and scalability, establishing RAVEN as a reference architecture for next-generation radar perception in resource-constrained and multi-task autonomous driving deployments (2604.04490).