Road-Obstacle Video Segmentation

• Road-obstacle video segmentation is the task of estimating temporally coherent binary masks for obstacles on drivable surfaces, emphasizing continuity to mitigate flickering.
• It leverages both image-based and video-based methodologies, using techniques like optical flow warping, spatio-temporal attention, and cross-modal fusion for robust detection.
• Evaluation involves pixel-, component-, and video-level metrics, highlighting the trade-offs between instantaneous segmentation accuracy and long-term temporal consistency.

Road-obstacle video segmentation is the task of estimating temporally coherent pixel masks for obstacles that lie on the drivable road surface across a sequence of frames, rather than treating each frame independently. In the formulation introduced for dedicated road-obstacle video segmentation, a video is represented as $I^{1:T}=\{I^1,\dots,I^T\}$, with binary ground-truth obstacle maps $A^{1:T}$, where $A^t\in\{0,1\}^{H\times W}$ and the positive label denotes “obstacle on the drivable road” (Rai et al., 16 Sep 2025). The central claim emerging from recent work is that the problem is inherently temporal: consecutive obstacle masks are strongly correlated, and purely image-based methods tend to produce flickering predictions and spurious positives across time (Rai et al., 16 Sep 2025). Closely related lines of work in RGB-D semantic segmentation, negative-obstacle segmentation, and driving-scene VSS supply much of the architectural and evaluation machinery now used for the topic, including cross-modal fusion, feature propagation, optical-flow warping, vanishing-point priors, and temporally aware benchmarking (Sun et al., 2020, Feng et al., 2023, Guo et al., 2024).

1. Task formulation and temporal structure

A canonical formulation models a video segmenter $\Psi$ as operating on a temporal window of $k$ frames, $I^{t:t+k-1}$, and producing predicted masks $\bar A^{t:t+k-1}$, with $\bar A^t\in[0,1]^{H\times W}$ (Rai et al., 16 Sep 2025). In that setting, training can use a per-frame binary cross-entropy term,

$$L_{\mathrm{per}}=-\sum_{\tau=0}^{k-1}\sum_{x,y}\Big[A^{t+\tau}(x,y)\log \bar A^{t+\tau}(x,y)+(1-A^{t+\tau}(x,y))\log(1-\bar A^{t+\tau}(x,y))\Big],$$

optionally augmented by a temporal consistency term,

$$L_{\mathrm{tmp}}=\sum_{\tau=1}^{k-1}\left\|\bar A^{t+\tau}-W_{t+\tau\leftarrow t+\tau-1}\big(\bar A^{t+\tau-1}\big)\right\|_1,$$

where $A^{1:T}$0 is an optical-flow warp (Rai et al., 16 Sep 2025). At inference, each pixel may be scored by maximum softmax or an energy-based anomaly score and then binarized by thresholding (Rai et al., 16 Sep 2025).

This explicit formulation clarifies two technical features of the field. First, road-obstacle video segmentation is not merely semantic video segmentation with a different class vocabulary: the target variable is often binary and road-restricted, which makes temporal false positives particularly damaging. Second, temporal coupling can be realized either through direct losses, through warped feature reuse, or through sequence models with learned spatio-temporal attention.

Earlier driving-scene work already exposed this distinction between per-frame and temporal formulations. The Highway Driving baseline used a three-part system—priming, approximating, and ensemble—in which a high-resolution priming network is run on the first frame or at low frequency, and subsequent frames reuse prior high-resolution information through the ensemble module; temporal consistency is therefore implemented implicitly, without explicit optical flow or recurrent LSTM components (Kim et al., 2020). A general objective for semantic video segmentation was written there as

$A^{1:T}$1

making clear that temporal smoothness and instantaneous segmentation quality can be traded against each other (Kim et al., 2020).

2. Benchmarks, annotation regimes, and data ecology

Benchmark construction for road-obstacle video segmentation combines several annotation regimes: dense semantic video labeling, obstacle-focused anomaly annotation, and RGB-depth road/negative-obstacle corpora. The most explicit road-obstacle video benchmarks were curated from four datasets—SOS, Lost & Found, LidarSOD, and AsRO—after filtering to road-only anomalies (Rai et al., 16 Sep 2025). In parallel, Highway Driving supplies dense 30 Hz semantic annotations for short highway clips, and DRNO contributes a large RGB-depth dataset for drivable roads and negative obstacles based on annotated NPO sequences (Kim et al., 2020, Feng et al., 2023).

Benchmark	Temporal statistics	Labels
Highway Driving	20 clips, 2 s each, 60 frames at 30 Hz, 1,200 annotated frames	10 classes including road, lane, fence, construction, car, truck
SOS	18 sequences, 1,004 annotated frames, $A^{1:T}$2 FPS, 1920 × 1080	road vs out-of-distribution obstacle
Lost & Found (video split)	13 sequences, 1,779 frames, avg length 149.2, 2048 × 1024	fine-grained obstacle annotation restricted to road pixels
LidarSOD	15 sequences, 2,421 frames, avg length 194.6, 1280 × 720	road, off-road, road-obstacle
AsRO	12 sequences, 12,480 frames, avg length 1192.7, 3384 × 2710	road-centered obstacles including cone, pole, tricycle, group, dustbin
DRNO	20 driving sequences; 8,752 RGB-depth pairs at 1242 × 2208, annotated 1 of every 5 frames	background, drivable road, negative obstacles

The Highway Driving dataset is notable because every frame in each 30 Hz clip is densely annotated in chronological order, and on average 97.8% of pixels are assigned to one of ten classes (Kim et al., 2020). Its label space emphasizes high-speed highway semantics—road, lane, sky, fence, construction, traffic sign, car, truck, vegetation, and unknown—rather than anomaly-only detection (Kim et al., 2020). By contrast, the road-obstacle video benchmarks concentrate directly on road-surface obstacles and include video-level statistics such as average clip length, which matter for evaluating consistency over long sequences (Rai et al., 16 Sep 2025).

DRNO adds another dimension to the data ecology by focusing on RGB-depth segmentation under invalid depth, with 748 images containing at least one negative obstacle and the remainder containing only drivable road (Feng et al., 2023). This design foregrounds a recurring issue in obstacle segmentation: the failure mode created by missing or untrustworthy depth measurements at far range.

3. Algorithmic families

The current literature separates into image-based, video-based, and multimodal families. In the 2025 road-obstacle study, image-based baselines include Entropy, Max Softmax, Energy, Max Logit, Void Classifier, Mask2Anomaly, EAM/AEM, and RbA, all applied per frame without temporal modeling; video-based baselines include M2F-Video and DVIS, both of which use learned temporal attention mechanisms across frames (Rai et al., 16 Sep 2025). Two vision-foundation baselines were introduced there: HM2F-Video, which combines a Hiera-B+ encoder with a Mask2Former-Video decoder using 100 mask queries and 9 decoder layers, and CC-SAM2, which freezes SAM 2 and adds a class-conditioned mask decoder trained with cross-entropy against binary ground truth (Rai et al., 16 Sep 2025).

The technical distinction is not simply whether multiple frames are input. M2F-Video and DVIS realize temporal coupling through transformer-style spatio-temporal masked attention and refinement blocks, whereas HM2F-Video and CC-SAM2 inherit large-scale pre-training from video-capable foundation architectures (Rai et al., 16 Sep 2025). This suggests that in road-obstacle video segmentation, temporal stability is increasingly mediated by generic video representation learning rather than by hand-engineered post-processing alone.

A distinct line of development appears in VPSeg, a driving-scene VSS model that uses vanishing-point priors rather than optical flow as the primary source of cross-frame correspondence (Guo et al., 2024). VPSeg processes the target frame together with $A^{1:T}$3 past frames in a context-detail framework. MotionVP performs VP-guided motion estimation and fusion on low-resolution features, DenseVP mines sparse-to-dense fine features around a rectangular VP region, and Contextualized Motion Attention fuses dynamic context with high-resolution detail via

$A^{1:T}$4

The model argues that objects near vanishing points are both less discernible and more constrained in their apparent motion under a forward-facing camera and linear ego-motion (Guo et al., 2024).

The Highway Driving baseline sits at a lower-complexity point in this design space. Its priming network, approximating network, and ensemble network together provide a practical real-time compromise: deep inference is done sparsely, low-resolution coarse segmentation is done frequently, and shallow fusion restores detail at full resolution (Kim et al., 2020). In obstacle-centric settings, such architectures remain relevant because they demonstrate that temporal reuse can be effective even without explicit recurrent or flow modules.

4. RGB-D fusion and obstacle-specific segmentation

Road-obstacle video segmentation has also inherited methods from RGB-D semantic segmentation, especially for small obstacles and negative obstacles. RFNet is a two-branch encoder-decoder in which RGB is the main stream and depth is the subordinate stream, both using ResNet-18 backbones with fusion after every stage, followed by SPP and three lightweight upsampling modules (Sun et al., 2020). Its Attention Feature Complementary fusion computes

$A^{1:T}$5

where $A^{1:T}$6 and $A^{1:T}$7 are RGB and depth features, respectively, and the channel-wise weighting is derived from GAP and $A^{1:T}$8 convolutions (Sun et al., 2020). Unexpected obstacle detection is implemented as an additional semantic class, “small obstacle,” so the head outputs 20 classes rather than using a separate detection head (Sun et al., 2020).

RFNet’s multi-dataset training is technically important because it formalizes how obstacle classes can be merged into a broader road-scene taxonomy. Cityscapes provides 19 standard classes, Lost and Found supplies small-obstacle pixels, and the joint loss partitions classes into a non-conflicting set $A^{1:T}$9 and a conflicting set $A^t\in\{0,1\}^{H\times W}$0: $A^t\in\{0,1\}^{H\times W}$1 with $A^t\in\{0,1\}^{H\times W}$2 for Cityscapes images and $A^t\in\{0,1\}^{H\times W}$3 otherwise (Sun et al., 2020). This is a direct example of obstacle segmentation being embedded into semantic segmentation through label engineering and dataset gating.

AMFNet addresses a different RGB-D issue: fusion with untrustworthy depth features can degrade performance below single-modality baselines (Feng et al., 2023). In AMFNet, each encoder stage receives RGB features $A^t\in\{0,1\}^{H\times W}$4, depth features $A^t\in\{0,1\}^{H\times W}$5, and a binary trust mask $A^t\in\{0,1\}^{H\times W}$6, where $A^t\in\{0,1\}^{H\times W}$7 if $A^t\in\{0,1\}^{H\times W}$8 and $A^t\in\{0,1\}^{H\times W}$9 otherwise. Adaptive-mask generation produces weights $\Psi$0, then defines

$\Psi$1

The reweighted features are fused by addition, then passed through channel attention and spatial attention (Feng et al., 2023). The associated claim is explicit: standard fusion often treats invalid depth as valid, while AMFNet suppresses depth wherever $\Psi$2, effectively falling back to RGB in untrusted regions (Feng et al., 2023).

A third strand is represented by the two-stage RGB-D obstacle avoidance system for road robots (Hua et al., 2019). Stage 1 segments road versus non-road, extracts an ROI polygon from the road mask, and Stage 2 performs three-class segmentation $\Psi$3 within that ROI; optical-flow supervision propagates previous-frame features to the current frame, and motion-blur augmentation improves robustness to camera shake (Hua et al., 2019). The resulting binary obstacle map is then refined by morphology and used for local path planning via a destination-setting heuristic and an Artificial Potential Field (Hua et al., 2019). Although this system is not framed as a benchmark paper, it directly ties temporally stabilized road-obstacle segmentation to downstream collision-free navigation.

5. Evaluation protocols and empirical behavior

The dedicated road-obstacle video benchmarks evaluate models at pixel, component, and video levels (Rai et al., 16 Sep 2025). Pixel-level metrics are AuROC, AuPRC, and FPR$\Psi$4; component-level metrics are sIoU, PPV, and

$\Psi$5

video-level metrics include $\Psi$6, $\Psi$7, and the balanced measure

$\Psi$8

This metric stack makes a substantive point: a model can score competitively on framewise ranking metrics while remaining temporally unstable over a sequence.

On Lost & Found, SOS, LidarSOD, and AsRO, CC-SAM2 attains the lowest FPR$\Psi$9 range, 3.7–25.6%, and the highest $k$0 range, 1.0–54.2, across all four datasets; HM2F-Video is the strongest runner-up, and image-based methods can reach AuROC values up to $k$1 but still show $k$2 (Rai et al., 16 Sep 2025). Qualitatively, CC-SAM2 is reported to yield smooth, stable obstacle masks over tens of frames, whereas image-level methods flicker and hallucinate (Rai et al., 16 Sep 2025). The associated failure modes remain clear: extreme illumination changes such as night or glare still generate false positives, and very small obstacles below 100 px are sometimes missed or spuriously detected (Rai et al., 16 Sep 2025).

The RGB-D literature contributes complementary quantitative evidence. RFNet reports 22 Hz on full-resolution 2048×1024 inputs and 41.6 Hz on half resolution, achieves $k$3 mIoU on the blended validation set of 20 classes, and reaches small-obstacle IoU $k$4, compared with 62.8% for single-RGB SwiftNet (Sun et al., 2020). On Cityscapes validation it achieves $k$5 mIoU while still fusing depth (Sun et al., 2020). AMFNet reports $k$6, $k$7, and $k$8, surpassing the best multi-modal competitor MAFNet at $k$9 on DRNO (Feng et al., 2023).

For semantic video segmentation in driving scenes, Highway Driving provides a classic speed-accuracy trade-off: DRN reaches 62.3% mIoU at normalized runtime $I^{t:t+k-1}$0, PSPNet reaches 62.9% at $I^{t:t+k-1}$1, and the mixed priming-approximating-ensemble baseline reaches 56.3% at $I^{t:t+k-1}$2 or 55.7% at $I^{t:t+k-1}$3, depending on update rate (Kim et al., 2020). VPSeg, evaluated on broader driving-scene VSS rather than obstacle-only masks, attains 77.48 mIoU, 49.42 iIoU, and 41.48 mIA-IoU on ACDC validation, and 82.46 mIoU with 61.79 iIoU on Cityscapes validation (Guo et al., 2024). Its obstacle-class IA-IoU values on ACDC improve for person, rider, bicycle, and traffic-sign relative to SegFormer and MRCFA (Guo et al., 2024). The small-obstacle avoidance system reports outdoor results of 92.1% / 97.2% / 93.8% / 4.2% for $I^{t:t+k-1}$4, $I^{t:t+k-1}$5, ODR, and NOFP under the “+Blur+Flow” setting, with outdoor path-planning Hausdorff distance 0.27 m (Hua et al., 2019).

6. Misconceptions, limitations, and research directions

One recurring misconception is that framewise obstacle segmentation is sufficient if the per-frame classifier is strong. The dedicated video benchmarks argue the opposite: the task is inherently temporal, and exploiting frame-to-frame correlation reduces false positives by up to 80% in FPR$I^{t:t+k-1}$6 and doubles video consistency (Rai et al., 16 Sep 2025). Another misconception is that multimodal fusion is uniformly beneficial. AMFNet explicitly reports that when RGB and depth are fused in the presence of untrustworthy features from invalid depth regions, performance can drop below that of single-modality networks (Feng et al., 2023).

A third misconception is that temporal consistency requires a single canonical mechanism. The literature instead presents several incompatible but effective paradigms. The small-obstacle avoidance system uses optical-flow supervision and warped feature fusion (Hua et al., 2019). Highway Driving uses shallow reuse of prior high-resolution features without explicit optical flow or recurrent LSTM (Kim et al., 2020). VPSeg dispenses with dense flow and builds explicit correspondences from vanishing-point geometry (Guo et al., 2024). Foundation-model baselines such as HM2F-Video and CC-SAM2 rely on learned spatio-temporal memory and decoder attention (Rai et al., 16 Sep 2025). This suggests that the design space is structured less by a binary choice between “temporal” and “non-temporal” models than by how temporal correspondence is parameterized.

The main documented limitations are also heterogeneous. CC-SAM2 still suffers under extreme illumination changes and with very small obstacles below 100 px (Rai et al., 16 Sep 2025). VP-guided correspondence can fail in cluttered scenes with multiple vanishing directions, during rapid lane changes or sharp turns, or for extremely close objects moving tangentially; proposed remedies include a denser angular codebook, a VP-refinement CNN, online adaptation of $I^{t:t+k-1}$7, and fallback local self-attention (Guo et al., 2024). In RGB-D settings, invalid depth remains a central failure mode, motivating trust masks and adaptive weighting (Feng et al., 2023).

Research directions proposed across the literature are relatively consistent. They include flow-guided warping or 3D convolutions for stronger frame coupling, multi-scale temporal decoders for small obstacles, higher-FPS densely annotated benchmarks, and unified ranking scores over the combined pixel-, component-, and video-level metrics (Rai et al., 16 Sep 2025). Additional directions include pre-training on large-scale video data such as VSPW 2021 (Rai et al., 16 Sep 2025), feature-level propagation instead of label-only propagation (Kim et al., 2020), multi-sensor fusion with lidar or radar (Kim et al., 2020), and real-time optimizations such as pruning, quantization, and knowledge distillation (Kim et al., 2020, Feng et al., 2023). In RGB-D road-obstacle systems, lightweight recurrent modules such as ConvLSTM or gated recurrent units, optical-flow-based warping of previous predictions, temporal smoothing of logits, and CRF or bilateral temporal filtering have all been proposed as extensions toward continuous road-obstacle video segmentation (Sun et al., 2020, Feng et al., 2023).

Taken together, the field now spans binary road-anomaly masking, full semantic video segmentation adapted to obstacle regions, RGB-D negative-obstacle modeling, and foundation-model-based sequence segmentation. The strongest common conclusion is that road-obstacle segmentation becomes materially different once temporal coherence is elevated from an afterthought to a first-class modeling objective.