High-Def Stereo Video Datasets

• High-definition stereo video datasets are detailed collections of synchronized binocular videos with precise geometric and photometric calibration for depth and disparity estimation.
• They employ diverse capture modalities—from consumer devices to specialized rigs—enabling advanced applications such as neural rendering, stereoscopic coding, and XR.
• These datasets provide comprehensive annotations and standardized metrics that drive innovations in stereo matching, HDR imaging, and 3D scene reconstruction.

High-definition stereo video datasets are foundational resources for research in computational stereo, 3D computer vision, neural rendering, immersive media, and extended reality (XR). Such datasets typically consist of rectified binocular video sequences, with precise geometric and photometric calibration, and may include per-frame ground truth in the form of disparity or depth maps. Recent years have seen increasing diversity and size in available datasets, supporting both classical algorithm development and learning-based approaches. The following sections synthesize comprehensive information on key collections, hardware/capture methodologies, ground truth generation, annotation formats, applications, and ongoing challenges in this rapidly evolving domain.

1. Major High-Definition Stereo Video Datasets

Numerous large-scale and high-fidelity stereo video datasets have been released, each emphasizing specific aspects such as sensor modality (e.g., LiDAR, dual RGB, synthetic), environment type (indoor/outdoor), spatiotemporal resolution, and annotation granularity. Selected datasets with explicit technical details:

Dataset	#Clips/Frames	Res/FPS	Capture Domain	Ground Truth
StereoWorld-11M (Xing et al., 10 Dec 2025)	142,520 clips, 11M frames	1920×1080 (orig.), 832×480@12fps (train)	Hollywood Blu-ray (SBS)	Depth (VideoDepthAnything); Disparity (StereoAnyVideo)
SVD (Izadimehr et al., 6 Jun 2025)	320 short/long per device	2200×2200/1920×1080@30fps	iPhone16 Pro, Vision Pro	Dense disparity (StereoSGBM); calibration supplied
StereoV1K (Zhang et al., 2024)	1,000 videos/500k frames	1180×1180@50fps	Real-world, spatial cam	Pseudo-disparity (IGEV), SxS videos
Infinigen SV (Jing et al., 2024)	226 videos/varied frames	1280×720@24fps	Synthetic nature	Rendered disparity, flow, normals
SouthKen SV (Jing et al., 2024)	266 videos/variable	1280×720@30fps	Urban (stereo camera)	Pseudo-disparity, segmentation
Helvipad (Zayene et al., 2024)	40,000 frames	1920×512 (equirect.)	Omnidirectional 360°, LiDAR	Dense LiDAR-projected depth/disparity
SVSR-Set (Imani et al., 2022)	71 videos/20s	1080×1920@30fps	HD stereo camera	Selected framewise disparity
H²-Stereo (Cheng et al., 2022)	1,929 YT+24 camera seqs	960×540@30fps, 1440×1080@200fps	YouTube/camera rig	None (Disparity for training via DispNet/FlowNet)
XR-Stereo (Cheng et al., 2023)	60,000 stereo pairs	640×480@30fps	Synthetic, photo-real, AR/XR	GT disparity, depth, flow, segmentation
WSVD (Wang et al., 2019)	10,788 shots/1.5M frames	≈960×1080@24-60Hz	Web videos (YouTube, etc.)	FlowNet2.0 disparity, masks

This diversity enables evaluation across a wide gamut of scene geometries, lighting, motion regimes, and renders both indoor and outdoor environments tractable for modern stereo algorithms.

2. Capture Modalities and Calibration Protocols

High-definition stereo video datasets deploy varied hardware, ranging from consumer-grade spatial video smartphones to precisely calibrated professional stereo camera rigs and omnidirectional (360°) sensors. Typical aspects of the capture pipeline include:

• Camera Systems and Baseline: Professional stereo cameras (ZED2: baseline ≈ 120 mm (Choudhary et al., 2022), AVP: 63.8 mm, iPhone: 19.2 mm in SVD (Izadimehr et al., 6 Jun 2025)) and custom rigs (Canon RF-S dual-lens, 7.8 mm f/4 STM in StereoV1K (Zhang et al., 2024)) allow for flexible tradeoffs between depth accuracy, field of view (FOV), and device portability.
• Intrinsic/Extrinsic Parameters: Calibrations include focal length ($f$), principal point ($c_x, c_y$), and distortion coefficients (radial, tangential) (Choudhary et al., 2022). Baseline ($B$) values are critical for depth accuracy and are documented for each capture configuration.
• Calibration Methods: Checkerboard-based geometric calibration (Zhang’s method, ZED-SDK) and photometric rectification are employed (Choudhary et al., 2022, Imani et al., 2022). Equirectangular 360° setups (Helvipad (Zayene et al., 2024)) require spherical projection calibration and LiDAR-to-camera extrinsic alignment.
• Synchronization and Temporal Alignment: Hardware triggering and precise timestamping enable synchronization at high frame rates (e.g., 200 Hz in H²-Stereo (Cheng et al., 2022)).

Exposure bracketing and HDR image stacks are often implemented with fixed tripods and adaptive exposure control for scenes with wide dynamic range (Choudhary et al., 2022).

3. Annotations: Disparity, Depth, and Semantic Labels

Most high-definition stereo datasets prioritize ground truth disparity and depth maps, but annotation granularity varies:

• Dense Disparity/Depth Maps:
  • GT by Triangulation: Depth is classically computed as $Z = fB/d$, where $d$ is the horizontal disparity (Choudhary et al., 2022).
  • Pseudo-GT Methods: Datasets without active range sensors generate “pseudo” ground truth using stereo matching networks (e.g., IGEV in StereoV1K (Zhang et al., 2024), FlowNet2.0 in WSVD (Wang et al., 2019), DispNet in H²-Stereo (Cheng et al., 2022)).
  • LiDAR-Supervised: Real-world 360° depth in Helvipad (Zayene et al., 2024) is derived by reprojecting LiDAR point clouds onto equirectangular frames and completing sparse depth via spherical interpolation.
• Additional Annotations: Many datasets provide or recommend computation of optical flow, surface normals, segmentation maps, bounding boxes, and scene mesh files (Cheng et al., 2023, Jing et al., 2024).
• Format: Disparity and depth are stored as 16/32-bit PNG or EXR images, with RGB views as PNG, MP4, or SVO; calibration data use JSON or proprietary containers (Choudhary et al., 2022, Cheng et al., 2023).

Most real-world datasets do not include explicit segmentation/object labels except when incorporating off-the-shelf detection pipelines (Mask2Former, YOLOv8 in SouthKen SV (Jing et al., 2024)).

4. Dataset Structure, Distribution, and Accessibility

Datasets adhere to structured directory layouts by scene and frame index, often with explicit metadata files:

• File/Folder Organization: Typical directory structures segment by dataset root, device/scene, exposure, and view (left/right), with frame-level annotation files stored alongside or in parallel hierarchies (Choudhary et al., 2022, Izadimehr et al., 6 Jun 2025, Zhang et al., 2024).
• Split Protocols: Consistent train/validation/test splits are enforced for benchmarks (StereoWorld-11M: 141,520/1,000 (Xing et al., 10 Dec 2025); StereoV1K: 955/45 (Zhang et al., 2024)). Some real-world datasets (SouthKen SV (Jing et al., 2024)) do not establish fixed splits, supporting qualitative or robustness evaluation.
• Access and Licensing: Public access is standard, with many collections released under Creative Commons (CC BY or CC BY-NC) terms, with explicit non-commercial or academic restrictions (Zhang et al., 2024, Izadimehr et al., 6 Jun 2025, Wang et al., 2019). Download links and code for data loaders are routinely supplied.

5. Evaluation Metrics, Benchmarks, and Protocols

Unambiguous, standardized evaluation is enabled through published protocols and metrics, tailored for stereo and video settings:

• Spatial Consistency: End-Point Error (EPE) in pixels, Bad-Pixel Ratio (>1px), D1-all error (>3px or 5%) (Zhang et al., 2024, Xing et al., 10 Dec 2025, Jing et al., 2024).
• Temporal Consistency: Temporal EPE (TEPE), Optical-flow Warping Error (OPW), Relative Temporal Consistency (RTC), and Temporal Change/Motion Consistency (TCC/TCM) measure per-pixel disparity stability over time (Jing et al., 2024).
• Fidelity: Peak Signal-to-Noise Ratio (PSNR), SSIM, LPIPS are computed between restored/synthesized and ground-truth frames. VBench IQ-score and TF-score quantify image and flicker quality in generative settings (Xing et al., 10 Dec 2025).
• Subjective/Objective Quality: Studies supplement objective metrics with human studies (Stereo Effect, Binocular Consistency, etc.) and structural features (spatial/temporal complexity, colorfulness) (Izadimehr et al., 6 Jun 2025).

Comparison tables, as presented in (Jing et al., 2024, Imani et al., 2022), enable cross-dataset performance analysis and highlight the range of environmental complexity and annotation accuracy.

6. Application Domains and Research Use Cases

High-definition stereo video datasets underpin a spectrum of core 3D vision and immersive media tasks:

• Stereo Matching and Depth Estimation: Benchmarking monocular, stereo, multi-view, and omnidirectional depth estimation methods on challenging, photorealistic, and dynamic content (Cheng et al., 2023, Zayene et al., 2024, Xing et al., 10 Dec 2025).
• HDR Imaging and Fusion: Joint exposure fusion and stereo matching for 3D HDR video and tone mapping in variable lighting conditions (Choudhary et al., 2022).
• Stereoscopic Coding/Compression: Adaptive MV-HEVC and 3D-HEVC codec benchmarking, streaming optimization for AR/VR, and rate-distortion curve computation (Izadimehr et al., 6 Jun 2025, Choudhary et al., 2022).
• Neural and Volumetric Rendering: Training and evaluating monocular- or neural-implicit view synthesis, neural scene flow, and spatiotemporal super-resolution (Xing et al., 10 Dec 2025, Zhang et al., 2024, Imani et al., 2022).
• XR and Mixed Reality: Evaluation of real-time stereo matching for pass-through AR/VR, online spatial video capture, and dynamic scene understanding (Cheng et al., 2023, Izadimehr et al., 6 Jun 2025).

These benchmarks address algorithm robustness under occlusions, motion, lighting variability, and complex geometries not representable by legacy still stereo or lab-based datasets.

7. Open Challenges and Future Directions

Despite significant advances, the field continues to evolve along several axes:

• High Dynamic Range (HDR) and Multi-Exposure: Expansion to larger exposure-stacks ($>5$), and deployment of continuous or multi-sensor bracketing (e.g., quad-Bayer, tri-exposure CMOS) (Choudhary et al., 2022).
• Omnidirectional/360° Stereo: Real-world panoramic benchmarks with dense LiDAR depth annotation (Helvipad (Zayene et al., 2024)) enable research beyond rectilinear stereo, spurring omnidirectional matching networks.
• Large-scale and High-speed Data: The introduction of 11M-frame (StereoWorld-11M (Xing et al., 10 Dec 2025)) and high-speed (200 Hz, H²-Stereo (Cheng et al., 2022)) resources model realistic viewing and content creation scenarios.
• Joint Spatiotemporal Reasoning: Datasets with accurate frame-level calibration/fidelity enable training of networks for joint optimization of stereo matching and temporal coherence.
• Annotations Beyond Disparity: Availability of per-frame flow, surface normals, segmentation, and mesh data further supports multi-task learning.

A plausible implication of these directions is the accelerated closing of the reality gap in robotics/autonomous systems, AR/VR, and 3D understanding—especially as larger, denser, and more diverse annotated corpora become available for open benchmarking and algorithm development.