Predicting 3D shapes, masks, and properties of materials, liquids, and objects inside transparent containers, using the TransProteus CGI dataset (2109.07577v2)

Abstract: We present TransProteus, a dataset, and methods for predicting the 3D structure, masks, and properties of materials, liquids, and objects inside transparent vessels from a single image without prior knowledge of the image source and camera parameters. Manipulating materials in transparent containers is essential in many fields and depends heavily on vision. This work supplies a new procedurally generated dataset consisting of 50k images of liquids and solid objects inside transparent containers. The image annotations include 3D models, material properties (color/transparency/roughness...), and segmentation masks for the vessel and its content. The synthetic (CGI) part of the dataset was procedurally generated using 13k different objects, 500 different environments (HDRI), and 1450 material textures (PBR) combined with simulated liquids and procedurally generated vessels. In addition, we supply 104 real-world images of objects inside transparent vessels with depth maps of both the vessel and its content. We propose a camera agnostic method that predicts 3D models from an image as an XYZ map. This allows the trained net to predict the 3D model as a map with XYZ coordinates per pixel without prior knowledge of the image source. To calculate the training loss, we use the distance between pairs of points inside the 3D model instead of the absolute XYZ coordinates. This makes the loss function translation invariant. We use this to predict 3D models of vessels and their content from a single image. Finally, we demonstrate a net that uses a single image to predict the material properties of the vessel content and surface.

• The paper presents a camera-agnostic XYZ mapping approach that bypasses traditional camera parameter requirements in 3D predictions.
• It employs a novel distance-based loss function, achieving low mean absolute error on both synthetic and real-world datasets.
• Results indicate robust vessel shape modeling and promising material property inference, despite challenges with object scale and translation.

Evaluating 3D and Material Property Prediction Using TransProteus

The paper presents TransProteus, a novel dataset with associated methodologies for predicting 3D structures, masks, and material properties of contents inside transparent vessels, a domain largely unexplored due to complexities associated with the transparency of materials in imaging. The dataset includes procedurally generated synthetic data and real-world images, equipped with 3D models and depth maps as annotations.

Dataset and Methodological Innovations

TransProteus stands out with its procedurally generated dataset of 50,000 CGI images, which include an extensive range of liquids and objects inside transparent containers, enriched with 13,000 distinct objects and 1,450 material textures. These synthetic images incorporate diverse environments and lighting conditions through the use of 500 High Dynamic Range Images (HDRI), enhancing the generalizability of the dataset. The inclusion of 104 real-world images offers a grounded test bed for models trained on synthetic data.

The authors introduce a camera-agnostic approach for 3D model prediction from images, bypassing the need for camera parameters. The model predicts an XYZ map rather than a depth map, allowing for direct point cloud predictions, enabling the bypassing of limitations associated with unknown camera parameters. The loss function used in training is novel, leveraging distance rather than absolute coordinates, offering translation invariance and independence from image orientation, a significant methodological departure from traditional depth prediction models.

Results and Implications

The results demonstrate the efficacy of the proposed solutions, with the XYZ prediction network achieving respectable mean absolute error (MAE) rates across both real and simulated datasets. Notably, the network's predictions for vessel shapes consistently scored well, indicating a robust ability to generalize across data types. Predicting object shapes within vessels proved more challenging when scale and translation errors were considered, highlighting an area for further optimization.

Material properties prediction shows promising results with low MAE values, underscoring the network's ability to compensate for variable conditions such as environmental lighting and vessel material properties. These findings suggest potential applications across fields like chemistry, medicine, and manufacturing where material handling and characterization are crucial.

Future Directions and Challenges

This research opens avenues for expanding the dataset to include more complex multiphase systems and chemically realistic simulations, aiming to extend the dataset's applicability and the method's accuracy. Additionally, leveraging existing graphical resources, such as CGI repositories, could further increase dataset diversity and result accuracy due to improved representational richness.

Despite these achievements, the transition of models trained purely on synthetic data to real-world effectiveness remains an inherent challenge, exacerbated by the complexities of transparent materials. The solution lies in integrating more real-world data and continuously evolving the simulation fidelity to bridge the reality gap.

In conclusion, the TransProteus dataset and methodology provide a significant step forward in transparent material volumetric analysis, introducing solutions that hold promise for broad applicability in automated environments where transparent containers are a common aspect of material handling.