Virtually increasing the measurement frequency of LIDAR sensor utilizing a single RGB camera

Abstract: The frame rates of most 3D LIDAR sensors used in intelligent vehicles are substantially lower than current cameras installed in the same vehicle. This research suggests using a mono camera to virtually enhance the frame rate of LIDARs, allowing the more frequent monitoring of dynamic objects in the surroundings that move quickly. As a first step, dynamic object candidates are identified and tracked in the camera frames. Following that, the LIDAR measurement points of these items are found by clustering in the frustums of 2D bounding boxes. Projecting these to the camera and tracking them to the next camera frame can be used to create 3D-2D correspondences between different timesteps. These correspondences between the last LIDAR frame and the actual camera frame are used to solve the PnP (Perspective-n-Point) problem. Finally, the estimated transformations are applied to the previously measured points to generate virtual measurements. With the proposed estimation, if the ego movement is known, not just static object position can be determined at timesteps where camera measurement is available, but positions of dynamic objects as well. We achieve state-of-the-art performance on large public datasets in terms of accuracy and similarity to real measurements.