RipVIS: Video Segmentation Benchmark

• RipVIS is a purpose-built video instance segmentation benchmark that accurately identifies rip currents using pixel-level annotations and safety-oriented recall evaluation.
• It offers a comprehensive dataset of 184 videos and 212,328 frames with expert-curated train/validation/test splits from diverse coastal environments.
• The benchmark introduces Temporal Confidence Aggregation (TCA) to enhance segmentation stability and improve recall in dynamic, real-world coastal settings.

RipVIS is a purpose-built video instance segmentation benchmark targeting the safe, reliable identification of rip currents in real coastal scenes. Introduced as a large-scale benchmark explicitly designed for rip current segmentation, it contains 184 videos and 212,328 frames, combines pixel-level masks, explicit train/validation/test splits, recall-focused evaluation centered on $F_2$, and a temporally aware post-processing method called Temporal Confidence Aggregation (TCA). Its stated objective is to help shift rip current analysis from prototype detection toward deployable segmentation tools for lifeguard towers, coastal agencies, fixed cameras, and drones (Dumitriu et al., 1 Apr 2025).

1. Hazard model and benchmark rationale

Rip currents are strong, localized and narrow currents of water that flow outwards into the sea, and they are a major source of beach-related injuries and fatalities. Their automated identification is difficult because they are amorphous, subtle against a dynamic water background, and annotated examples usually require expert knowledge. In visual scenes, they may appear as evolving gaps in wave breaking, streaks of sediment transport, water color changes, offshore sediment flows, turbidity variations, or boundary-controlled currents near structures such as piers and jetties (Dumitriu et al., 1 Apr 2025).

The benchmark is motivated by the mismatch between the physical character of rip currents and standard object-centric vision assumptions. Unlike rigid objects with crisp boundaries, a rip current is an evolving flow region embedded in complex surf dynamics. This makes bounding-box-only formulations incomplete for many operational settings. RipVIS therefore frames the problem as video instance segmentation of a single class, with each “instance” defined as any contiguous rip current region visible in a frame. The emphasis is on precise spatial delineation in each frame rather than coarse localization alone (Dumitriu et al., 1 Apr 2025).

A central design principle is safety-oriented evaluation. The benchmark explicitly prioritizes recall because, in beach safety, missing a rip current is more costly than producing a false alarm. This rationale underlies the prominence of the $F_2$ score in both model comparison and post-processing design (Dumitriu et al., 1 Apr 2025).

2. Dataset design, sources, and annotation protocol

RipVIS is an order of magnitude larger than previous rip-current datasets. It contains 150 videos with rip currents and 34 videos without rip currents, for a total of 184 videos and 212,328 frames. Most sequences are annotated at 5 FPS, although the sampling rate varies from 1–30 FPS depending on camera motion and scene dynamics (Dumitriu et al., 1 Apr 2025).

Subset	Videos	Frames
With rip currents	150	163,528
Without rip currents	34	48,800
Total	184	212,328

The benchmark aggregates material from multiple acquisition regimes. Seventy-six videos were recorded by the authors using drones and mobile phones, 87 were collected from the Internet to capture real-world variability, and 21 came from the de Silva et al. dataset. Viewpoints span four categories: water-level beachfront, elevated beachfront, aerial tilted view, and aerial bird’s-eye view. Geographic coverage includes the USA, Mexico, Costa Rica, Portugal, Italy, Greece, Romania, Sri Lanka, Australia, and New Zealand (Dumitriu et al., 1 Apr 2025).

Annotation density is unusually high for this domain. RipVIS provides pixel-level, polygon-based instance segmentation masks of rip current regions, and bounding boxes are also available for compatibility with detection-style methods. A total of 15,784 frames were manually annotated, yielding 25,298 rip current instances, or approximately 1.6 instances per frame. Interpolated labels verified by experts extend the annotations to 163,528 frames with rip currents while preserving the average instance count (Dumitriu et al., 1 Apr 2025).

The quality-control process is explicit. Annotation involved 30 trained volunteers under two academic experts; experts seeded the first frame of each video, all labels were reviewed, Cohen’s $\kappa = 0.82$, and 28 videos required major re-annotation to maintain standards. The data split was expert-curated to mirror distribution and avoid overfitting: 112 videos and 147,802 frames for training, 36 videos and 32,566 frames for validation, and 36 videos and 31,960 frames for testing (Dumitriu et al., 1 Apr 2025).

3. Task definition and evaluation protocol

The benchmark task is video instance segmentation with a single foreground category, “rip current.” Because multiple rip-current regions may appear in one frame, the formulation is instance-based rather than purely semantic. Evaluation is computed per frame on manually annotated frames, but the dataset structure also supports temporal analysis because interpolated masks and multiple instances per frame are retained (Dumitriu et al., 1 Apr 2025).

RipVIS reports Precision, Recall, AP/AP50 for segmentation, $F_1$, $F_2$, and FPS for inference speed. In the single-class setting, mAP equals AP for rip currents. The benchmark uses the standard precision–recall-curve definition of average precision,

$$\mathrm{AP} = \sum_n (\mathrm{Recall}_n - \mathrm{Recall}_{n-1}) \cdot \mathrm{Precision}_n$$

and gives particular emphasis to

$F_2 = \frac{5 \cdot P \cdot R}{4P + R}.$

The use of $F_2$ is not incidental. The benchmark explicitly argues that recall should be prioritized because false negatives are operationally more costly than false positives in beach monitoring. This choice also shaped later competitions derived from RipVIS, where recall-weighted metrics remained central (Dumitriu et al., 1 Apr 2025).

A common misunderstanding is to treat RipVIS as only a still-image resource because later challenges use extracted frames. In fact, RipVIS was introduced as a video instance segmentation benchmark; still-image competitions such as AIM 2025 RipSeg and NTIRE 2026 RipDetSeg are downstream derivatives that repurpose its annotated frames for image-based evaluation (Dumitriu et al., 18 Aug 2025, Dumitriu et al., 18 Apr 2026).

4. Baselines and Temporal Confidence Aggregation

The benchmark evaluates both two-stage and one-stage instance segmentation models. The reported baselines are Mask R-CNN, Cascade Mask R-CNN, SparseInst, and YOLO11, with YOLO11 evaluated in n, s, m, l, and x variants and SparseInst evaluated with R-50, R-101, and PVTv2-B1 backbones. Training used NVIDIA RTX 4090 (24 GB), while inference speed was measured on an RTX 3060 (12 GB) at 1920×1080. The implementation stack included Ultralytics YOLO v8.3.29, Detectron2 v0.6, Python 3.10.4, PyTorch 1.12.1, and CUDA 12.2 (Dumitriu et al., 1 Apr 2025).

Representative results show a consistent precision–recall trade-off across model families. YOLO11-l achieved the highest precision at 0.812 without TCA and 0.819 with TCA, but recall remained moderate at 0.588 and 0.613. SparseInst R-50 achieved the highest recall at 0.782 without TCA and 0.807 with TCA. The best overall $F_2$ was obtained by SparseInst PVTv2, improving from 0.751 to 0.780 with TCA; the same model also achieved the top $F_1$ improvement from 0.724 to 0.753 and strong AP50 improvement from 0.721 to 0.751 (Dumitriu et al., 1 Apr 2025).

TCA is the benchmark’s principal methodological addition beyond the dataset itself. It is a post-processing step designed to stabilize per-frame segmentation by aggregating pixel confidences across time and enforcing hysteresis. The pipeline downsamples predicted masks, performs instance tracking across frames via IoU matching with the Hungarian algorithm, updates a per-instance temporal heatmap with a moving average of pixel confidence,

$F_2$0

maintains pixel-level present and absence counters, optionally applies Gaussian smoothing, and then performs hysteresis thresholding with dual thresholds $F_2$1 and $F_2$2 before upsampling refined masks to the original resolution (Dumitriu et al., 1 Apr 2025).

The reported effect is systematic rather than anecdotal. TCA consistently improves recall and $F_2$3 across all baselines: SparseInst PVTv2 improves from 0.751 to 0.780 in $F_2$4, Mask R-CNN from 0.593 to 0.625, Cascade Mask R-CNN from 0.648 to 0.670, and YOLO11 variants gain approximately 0.02–0.03 in $F_2$5. Qualitatively, TCA smooths jagged masks, fills intermittent gaps where a rip current briefly fades, and suppresses isolated foam-induced false positives. The benchmark also records its limits: foam and wave-breaking textures can be mistaken for rip channels, shadows and lighting changes induce false positives, turbidity and intermittent visibility cause false negatives, and moving cameras make temporal aggregation harder when parameters are not tuned to motion (Dumitriu et al., 1 Apr 2025).

5. Position within rip-current computer vision research

RipVIS was introduced against a small and heterogeneous prior literature. Earlier efforts included small timex patches associated with Maryan et al., aerial image sets with boxes and unannotated videos from de Silva et al. and YOLO-Rip, and a 2023 dataset identified as the first instance segmentation dataset but constrained to one region. RipVIS distinguishes itself by combining global geographic diversity, video-scale annotation, curated splits, explicit safety-oriented evaluation, and a temporally aware refinement method tailored to amorphous phenomena (Dumitriu et al., 1 Apr 2025).

Another 2025 study introduced a rip current instance segmentation benchmark with 2,466 annotated still images, 1,307 negative images, and 17 drone videos comprising 24,295 frames for testing, and reported YOLOv8 baselines trained only on static images. That work did not assign a formal name to the benchmark in the paper; its public repository is titled “rip_currents.” Relative to that benchmark, RipVIS places much greater weight on video-scale annotation density, expert-verified interpolation, explicit temporal reasoning, and expert-curated train/validation/test splits (Dumitriu et al., 3 Apr 2025).

RipVIS should also be distinguished from RipViz, a separate stationary-video system that treats rip currents as flow anomalies in an otherwise normal ocean flow. RipViz reconstructs an unsteady 2D vector field using Lucas–Kanade optical flow, forms sequences of short pathlines, trains a convolutional-LSTM autoencoder on normal flow, and localizes anomalous origination points as a rip zone. Its reported objective is interpretable overlay visualization rather than supervised benchmark construction (Silva et al., 2023). This suggests that RipVIS and RipViz occupy complementary positions: one is a benchmark for supervised video instance segmentation, the other a hybrid flow-analysis and anomaly-detection system.

6. Challenge ecosystem and benchmark influence

RipVIS rapidly became the basis of organized competitions. The AIM 2025 RipSeg Challenge used all available annotated frames from RipVIS together with training data from prior work, producing a total of 27,718 images for a still-image single-class instance segmentation task. The challenge split comprised 18,386 training images, 4,348 validation images, and 4,984 test images, and ranking used the composite score

$F_2$6

The competition registered 75 participants and received 5 valid test submissions. The top-ranked method, RipEye, combined HRDA, SHADE, and RipGAN; the second-ranked RipSense used SparseInst with a PVTv2-B1 backbone and morphological opening; and the report concluded that generalization across locations, current types, and camera viewpoints remained difficult (Dumitriu et al., 18 Aug 2025).

The NTIRE 2026 RipDetSeg Challenge extended the evaluation further by introducing both detection and instance segmentation on still images derived from the RipVIS ecosystem. It used 32,407 training images, 9,262 validation-format images, and 9,145 held-out test images. Final ranking averaged $F_2$7, $F_2$8, $F_2$9, and $\kappa = 0.82$0 equally. The challenge attracted 159 registered participants and produced 9 valid test submissions. In segmentation, UNO Pixel Pros ranked first with a Final Score of 55.79; in detection, SiGMoid ranked first with 56.33. The report emphasized that even strong teams scored in the mid-50s, underscoring the difficulty of robust rip current understanding under substantial domain shift (Dumitriu et al., 18 Apr 2026).

These downstream competitions formalized an important misconception correction: RipVIS is not limited to a single benchmark protocol. Its annotated masks, global viewpoint variation, and single-class formulation have been repurposed for video instance segmentation, still-image instance segmentation, and joint detection–segmentation evaluation, while preserving the original safety-oriented concern with recall (Dumitriu et al., 18 Aug 2025, Dumitriu et al., 18 Apr 2026).

7. Deployment implications, limitations, and future directions

RipVIS was designed with operational pathways in view. For fixed beach cameras, the benchmark recommends slower-decay TCA variants for stationary viewpoints; for drones and elevated viewpoints, fast-gain, fast-decay variants are recommended for moving cameras. Lightweight YOLO11-n can exceed 100 FPS at 1080p without TCA on mid-range GPUs, while SparseInst PVTv2 offers strong $\kappa = 0.82$1 and AP at approximately 28 FPS without TCA and approximately 18 FPS with TCA on a 3060. The benchmark therefore frames deployment as a safety-versus-latency trade-off rather than a purely accuracy-driven choice (Dumitriu et al., 1 Apr 2025).

The limitations are explicit. Although geographically diverse, the dataset underrepresents flash or traveling rip currents and some camera geometries. Environmental variance, including weather and turbidity, remains challenging. Annotation ambiguity persists because rip currents are amorphous and intermittently visible; despite high inter-annotator agreement, some frames without visually obvious currents are intentionally left unannotated. Cross-dataset tests indicate significant distribution-shift effects, implying that domain adaptation may be necessary for new beaches, bathymetries, or camera heights and angles (Dumitriu et al., 1 Apr 2025).

The benchmark also raises practical safeguards. Videos can include people, so deployment and data sharing should comply with local regulations and protect privacy. The paper recommends human-in-the-loop verification, conservative TCA variants where missing a current is unacceptable, and alert policies that balance false alarms against operational trust. This suggests that RipVIS is intended as an assistive component in beach monitoring rather than a sole decision-maker (Dumitriu et al., 1 Apr 2025).

Future directions identified around the benchmark include end-to-end video instance segmentation architectures with explicit temporal layers or transformers, multi-modal sensing that combines RGB with optical flow, polarization, thermal, or dye-tracer imagery, domain adaptation and self-supervised learning for beach-specific appearance shifts, active learning with experts, and safety-focused training with explicit false-negative penalties and uncertainty estimation. The benchmark website hosts data, code, baseline models, and results, and it encourages contributions of new videos and models to extend the resource over time (Dumitriu et al., 1 Apr 2025).