Fit4CAD: A point cloud benchmark for fitting simple geometric primitives in CAD objects (2105.06858v3)

Abstract: We propose Fit4CAD, a benchmark for the evaluation and comparison of methods for fitting simple geometric primitives in point clouds representing CAD objects. This benchmark is meant to help both method developers and those who want to identify the best performing tools. The Fit4CAD dataset is composed by 225 high quality point clouds, each of which has been obtained by sampling a CAD object. The way these elements were created by using existing platforms and datasets makes the benchmark easily expandable. The dataset is already split into a training set and a test set. To assess performance and accuracy of the different primitive fitting methods, various measures are defined. To demonstrate the effective use of Fit4CAD, we have tested it on two methods belonging to two different categories of approaches to the primitive fitting problem: a clustering method based on a primitive growing framework and a parametric method based on the Hough transform.

• The paper presents Fit4CAD, a comprehensive benchmark with 225 high-quality point clouds for evaluating primitive fitting methods on CAD objects.
• It compares two methodologies—Primitive Growing (PG) and Hough Transform (HT)—highlighting HT’s superior accuracy and robustness in point classification and geometric fitting.
• The study employs detailed metrics such as true positive rate, mean fitting error, and coefficient distance to quantify performance and identify challenging cases in primitive segmentation.

The paper "Fit4CAD: A point cloud benchmark for fitting simple geometric primitives in CAD objects" (2105.06858) introduces a new benchmark designed to evaluate and compare methods for detecting and fitting simple geometric primitives (planes, cylinders, cones, spheres, and tori) in point clouds derived from CAD objects. The benchmark addresses the need for standardized datasets and evaluation metrics in the field of reverse engineering, where obtaining digital models from physical objects often involves scanning and processing point cloud data.

The core of the benchmark is the Fit4CAD dataset, consisting of 225 high-quality point clouds generated by sampling CAD models from the Onshape platform and the ABC dataset. The models were carefully filtered and curated, with some manually modified to introduce variations like missing data. The dataset is divided into a training set (190 point clouds) and a test set (35 point clouds). The generation process is detailed, emphasizing the use of existing tools and datasets to ensure the benchmark's expandability. For each point cloud, the benchmark provides a comprehensive ground truth, including:

• The point cloud data itself ($PCi$).
• The list of true primitive segments, mapping point indices to primitives ($PCi\_primitives$). Points not belonging to simple primitives (often from B-spline surfaces in the original model) are labeled as unsegmented.
• Parametric representations for each primitive ($PCi\_parametric$), specified according to defined formats for each primitive type (plane, cylinder, cone, sphere, torus). For instance, a plane is represented by $[a_1, a_2, a_3, b_1, b_2, b_3, c_1, c_2, c_3]$ from its parametric equation:

  x = a1*u + b1*v + c1 y = a2*u + b2*v + c2 z = a3*u + b3*v + c3

• Implicit representations for each primitive ($PCi\_implicit$), also specified in defined formats. For quadric surfaces (cylinder, cone, sphere), this is typically a quadratic form $ax^2+by^2+cy^2+2(dxy+exz+fyz)+2(gx+hy+iz)+l=0$, stored as $[a, b, c, d, e, f, g, h, i, l]$. For planes, it's a linear form $ax+by+cz+d=0$, stored as $[a, b, c, d]$. Torus implicit forms are degree 4 polynomials with coefficients stored in reverse lexicographic order.

To enable quantitative evaluation, the benchmark proposes a set of quality indicators:

• Performance Measures (Point Classification): These assess how well a method's segmentation aligns with the ground truth segmentation at the point level. Metrics derived from the confusion matrix are used, including True Positive Rate (Sensitivity), True Negative Rate (Specificity), Positive Predictive Value (PPV), Negative Predictive Value (NPV), Accuracy (ACC), and the Sørensen-Dice index (DSC, equivalent to F1 score for binary classification). These metrics are calculated by comparing the detected primitive segments to the corresponding ground truth segments.
• Approximation Accuracy: These measure the geometric fidelity of the fitted primitive surfaces to the point data.
  • Mean Fitting Error (MFE): The average Euclidean distance from the points in a detected segment to the fitted surface, normalized by the diagonal of the segment's bounding box.
  • Directed Hausdorff distance: The maximum distance from any point in the detected segment to the fitted surface, also normalized. This metric is sensitive to outliers.
  • Coefficient distance ($L_1$ norm): The $L_1$ distance between the normalized coefficient vectors of the implicit representations of the fitted primitive and the ground truth primitive. This measures the similarity of the estimated surface parameters.

The paper demonstrates the benchmark's utility by evaluating two different primitive fitting methods on the test set:

1. PG (Primitive Growing): A curvature-based method [Qie:2021] that first detects sharp edges using surface variation [Pauly2002] for initial partitioning and then refines regions using slippage analysis [Gelfand:2004]. It applies RANSAC [Schnabel2007] to undersegmented prismatic/revolute/complex regions. This method provides segmentations but not explicit parametric/implicit equations, so only point classification metrics apply. Its computational complexity is roughly $O(N \log N)$ for initialization and $O(M)$ per segment for refinement, where $N$ is the total number of points and $M$ is the number of points in a segment.
2. HT (Hough Transform): A parametric method [beltrametti2012algebraic] that uses the Hough Transform to find primitive parameters in the parameter space, identifying parameters corresponding to accumulator peaks. It refines results by selecting the primitive type with the lowest MFE for each point cluster. This method provides both segmentations and explicit primitive representations, allowing evaluation with all defined metrics. The computational complexity depends on the discretization of the parameter space and the primitive types being sought.

The evaluation results show that the HT-based method generally achieves higher point classification accuracy (higher median DSC, TPR, PPV, ACC) and lower variability across different models compared to the PG method. Both methods struggle with severe undersegmentation when the ground truth contains a large number of small or thin primitives (e.g., PC 9). The HT method is also shown to be more robust to missing data than the PG method in terms of classification metrics. For the HT method, the approximation accuracy metrics (MFE and Directed Hausdorff) are generally low (within 1-2% of bounding box diagonal), indicating good geometric fits. The coefficient distance shows more variability, particularly for tori, suggesting challenges in accurately recovering their implicit parameters compared to other shapes. In terms of computational time, PG can be faster for very small point clouds, but HT is generally faster for larger ones, although both can have significant maximum execution times depending on the complexity of the input.

The authors conclude that Fit4CAD provides a valuable resource for developing and comparing primitive fitting algorithms for CAD point clouds. The benchmark highlights the trade-offs between different approaches and identifies challenging cases, such as models with many small primitives or missing data. Future work includes expanding the dataset with more complex primitives and assembly models and incorporating evaluation of machine learning-based methods, which the dataset is already structured to support. The benchmark resources are made publicly available on GitHub.