Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects under Occlusion (2403.19022v2)

Published 27 Mar 2024 in cs.CV

Abstract: Current methods for 2D and 3D object understanding struggle with severe occlusions in busy urban environments, partly due to the lack of large-scale labeled ground-truth annotations for learning occlusion. In this work, we introduce a novel framework for automatically generating a large, realistic dataset of dynamic objects under occlusions using freely available time-lapse imagery. By leveraging off-the-shelf 2D (bounding box, segmentation, keypoint) and 3D (pose, shape) predictions as pseudo-groundtruth, unoccluded 3D objects are identified automatically and composited into the background in a clip-art style, ensuring realistic appearances and physically accurate occlusion configurations. The resulting clip-art image with pseudo-groundtruth enables efficient training of object reconstruction methods that are robust to occlusions. Our method demonstrates significant improvements in both 2D and 3D reconstruction, particularly in scenarios with heavily occluded objects like vehicles and people in urban scenes.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (82)
  1. Amodal 3d reconstruction for robotic manipulation via stability and connectivity. In CoRL, 2021.
  2. Augmented reality meets deep learning for car instance segmentation in urban scenes. In BMVC, 2017.
  3. Asra Aslam. Detecting objects in less response time for processing multimedia events in smart cities. In CVPR, 2022.
  4. Simple online and realtime tracking. In ICIP, pages 3464–3468, 2016.
  5. Omni3d: A large benchmark and model for 3d object detection in the wild. In CVPR, 2023.
  6. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
  7. Yale-cmu-berkeley dataset for robotic manipulation research. IJRR, 2017.
  8. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, 2019.
  9. End-to-end learnable geometric vision by backpropagating pnp optimization. In CVPR, 2020.
  10. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022.
  11. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In ECCV, 2016.
  12. Objaverse: A universe of annotated 3d objects. In CVPR, 2023.
  13. Mot20: A benchmark for multi object tracking in crowded scenes. arXiv preprint arXiv:2003.09003, 2020.
  14. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
  15. Ithaca365: Dataset and driving perception under repeated and challenging weather conditions. In CVPR, 2022.
  16. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
  17. Segan: Segmenting and generating the invisible. In CVPR, 2018.
  18. Learning to detect and track visible and occluded body joints in a virtual world. In ECCV, 2018.
  19. Learning to see the invisible: End-to-end trainable amodal instance segmentation. In WACV, 2019.
  20. A mean field em-algorithm for coherent occlusion handling in map-estimation prob. In CVPR, 2006.
  21. A segmentation-aware object detection model with occlusion handling. In CVPR, 2011.
  22. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
  23. Simple copy-paste is a strong data augmentation method for instance segmentation. In CVPR, 2021.
  24. Video based traffic congestion prediction on an embedded system. In ITSC, 2008.
  25. Humans in 4D: Reconstructing and tracking humans with transformers. In ICCV, 2023.
  26. Google. Google Street View. https://www.google.com/streetview/.
  27. Roca: robust cad model retrieval and alignment from a single image. In CVPR, 2022.
  28. Beyond the line of sight: labeling the underlying surfaces. In ECCV, 2012.
  29. Deep residual learning for image recognition. In CVPR, 2016.
  30. Mask r-cnn. In ICCV, 2017.
  31. SAIL-VOS: Semantic Amodal Instance Level Video Object Segmentation – A Synthetic Dataset and Baselines. In CVPR, 2019.
  32. Cad-deform: Deformable fitting of cad models to 3d scans. In ECCV, 2020.
  33. End-to-end recovery of human shape and pose. In CVPR, 2018a.
  34. Learning category-specific mesh reconstruction from image collections. In ECCV, 2018b.
  35. Gaetano Kanizsa. Organization in vision: Essays on gestalt perception. Praeger Publishers, 1979.
  36. Gsnet: Joint vehicle pose and shape reconstruction with geometrical and scene-aware supervision. In ECCV, 2020.
  37. Deep occlusion-aware instance segmentation with overlapping bilayers. In CVPR, 2021.
  38. Vibe: Video inference for human body pose and shape estimation. In CVPR, 2020.
  39. Pare: Part attention regressor for 3d human body estimation. In ICCV, 2021.
  40. Philipp Krähenbühl. Free supervision from video games. In CVPR, 2018.
  41. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
  42. Learning analysis-by-synthesis for 6d pose estimation in rgb-d images. In ICCV, 2015.
  43. Articulation-aware canonical surface mapping. In CVPR, 2020.
  44. 3d-rcnn: Instance-level 3d object reconstruction via render-and-compare. In CVPR, 2018.
  45. Epnp: An accurate o (n) solution to the pnp problem. IJCV, 2009.
  46. Deep supervision with shape concepts for occlusion-aware 3d object parsing. In CVPR, 2017.
  47. Traffic4d: Single view longitudinal 4d reconstruction of repetitious activity using self-supervised experts. In IV, 2021.
  48. Exploring plain vision transformer backbones for object detection. In ECCV, 2022.
  49. Microsoft coco: Common objects in context. In ECCV, 2014.
  50. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
  51. SMPL: A skinned multi-person linear model. SIGGRAPH Asia, 2015.
  52. Robust category-level 6d pose estimation with coarse-to-fine rendering of neural features. In ECCV, 2022.
  53. The 6th ai city challenge. In CVPRW, 2022.
  54. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
  55. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NeurIPS, 2017.
  56. Amodal instance segmentation with kins dataset. In CVPR, 2019.
  57. Carfusion: Combining point tracking and part detection for dynamic 3d reconstruction of vehicles. In CVPR, 2018.
  58. Occlusion-net: 2d/3d occluded keypoint localization using graph networks. In CVPR, 2019.
  59. Walt: Watch and learn 2d amodal representation from time-lapse imagery. In CVPR, 2022.
  60. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
  61. Octnet: Learning deep 3d representations at high resolutions. In CVPR, 2017.
  62. Learning to look around objects for top-view representations of outdoor scenes. In ECCV, 2018.
  63. Hand keypoint detection in single images using multiview bootstrapping. In CVPR, 2017.
  64. Apollocar3d: A large 3d car instance understanding benchmark for autonomous driving. In CVPR, 2019.
  65. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019.
  66. Putting people in their place: Monocular regression of 3d people in depth. In CVPR, 2022.
  67. Viewpoints and keypoints. In CVPR, 2015.
  68. Structured output regression for detection with partial truncation. In NeurIPS, 2009.
  69. Toward planet-wide traffic camera calibration. In WACV, 2024.
  70. Nemo: Neural mesh models of contrastive features for robust 3d pose estimation. In ICLR, 2021.
  71. Normalized object coordinate space for category-level 6d object pose and size estimation. In CVPR, 2019.
  72. Simple online and realtime tracking with a deep association metric. In ICIP, 2017.
  73. Pointconv: Deep convolutional networks on 3d point clouds. In CVPR, 2019.
  74. Beyond pascal: A benchmark for 3d object detection in the wild. In WACV, 2014.
  75. Learning to track: Online multi-object tracking by decision making. In ICCV, 2015.
  76. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, 2020.
  77. Robust instance segmentation through reasoning about multi-object occlusion. In CVPR, 2021.
  78. Bytetrack: Multi-object tracking by associating every detection box. In ECCV, 2022.
  79. 3d shape estimation from 2d landmarks: A convex relaxation approach. In CVPR, 2015.
  80. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR, 2018.
  81. Semantic amodal segmentation. In CVPR, 2017.
  82. Towards scene understanding with detailed 3d object representations. IJCV, 2015.
Citations (1)
View on Semantic Scholar

Summary

  • The paper presents an automated data synthesis framework that generates robust pseudo-groundtruth labels from time-lapse videos for both 2D and 3D features.
  • The framework enhances object reconstruction under occlusion by synthesizing realistic occlusion scenarios and improving model training efficiency.
  • Experimental results demonstrate significant improvements in tasks like vehicle and human segmentation, keypoint estimation, and pose reconstruction.

An Analytical Overview of "WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects under Occlusion"

The paper "WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects under Occlusion" presents a novel framework designed to address significant challenges in 2D and 3D object understanding, especially under conditions of severe occlusion. The researchers introduce an automatic data generation method leveraging time-lapse imagery to produce a robust dataset, thereby circumventing the extensive requirement for human-labeled ground-truth annotations.

Methodology

The authors build upon the previously established WALT framework, enhancing it by extending capabilities into the 3D domain. This involves synthesizing occlusion scenarios where unoccluded objects are extracted from time-lapse videos and recomposed into backgrounds following their original positions, thereby creating realistic clip-art style images with accurate occlusion configurations. The process generates what is defined as "pseudo-groundtruth" data, predicted by existing methods, for both 2D (e.g., segmentation, keypoints) and 3D (e.g., pose, shape) features.

Key Contributions

  1. Automated Data Synthesis: The framework automates the generation of 2D and 3D supervision data from freely available time-lapse videos without requiring human intervention. This scalability is crucial given the difficulty of manually annotating occluded object parts.
  2. Robustness to Occlusions: The synthetic data significantly strengthens the training of models on both 2D and 3D reconstruction tasks, particularly in challenging urban environments where object occlusion is typical.
  3. Data Efficiency: By leveraging a 3D compositing approach, the method improves the efficiency of training data usage, which is especially beneficial in acquiring quality data under low-data regimes.

Experimental Validation

The authors conducted extensive experiments across a wide variety of scenarios entailing heavy object occlusions. Results demonstrate marked improvements over existing methods in various tasks, including vehicle and human detection, segmentation, and keypoint estimation. Specifically, metrics such as Average Precision (AP) and Percentage of Correct Keypoints (PCK) illustrate the superior performance of models trained with WALT3D-generated data under scenarios featuring significant occlusions.

Potential for Broad Impact

The implications of this research are manifold for both theoretical and practical developments within AI. The automation of realistic dataset generation could facilitate advancements in smart cities and robotic applications by enhancing model robustness to occlusion, critical for autonomous systems reliant on visual data interpretation. Furthermore, this framework suggests future possibilities for refining pseudo-groundtruth data accuracy through enhanced object pose models and the incorporation of further environmental conditions.

Future Directions

Research could explore generalizing the WALT3D methodology to accommodate diverse object categories beyond vehicles and pedestrians. Expanding upon the disparity between synthetic and real-world data appearance, particularly in varying lighting and weather conditions, could be advantageous. Moreover, strategies to integrate and refine parametric object models might address limitations such as handling rare objects without predefined shape models.

In conclusion, WALT3D represents a significant methodological stride in realistic training data generation, providing vital foundational data solutions to advance object perception tasks, particularly under adverse occlusion conditions. This capability is essential to evolving the robustness and capability of autonomous systems within complex real-world environments.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube