ModelNet-R Benchmark Family

• ModelNet-R is a family of benchmark constructs originally derived from ModelNet, notably used for shape retrieval with volumetric CDBN features in 3D ShapeNets.
• It includes a refined version of ModelNet40 that corrects labeling and structural issues as well as a rotated evaluation protocol ensuring SO(3)-robust point-cloud learning.
• Additionally, ModelNet-R denotes a random non-IID split of a CIFAR-100-based federated learning dataset, emphasizing neutral data partitioning without semantic bias.

ModelNet-R is not a single standardized resource. Across arXiv papers, the name has been used for several non-equivalent constructs: a shape-retrieval benchmark derived from the original ModelNet CAD collection; a refined version of ModelNet40 intended to correct labeling, geometric, and class-boundary issues; a rotated-ModelNet40 evaluation protocol for SO(3)-robust point-cloud learning; and a random client-partition variant of a CIFAR-100-based federated-learning dataset (Wu et al., 2014, Saeid et al., 5 Sep 2025, Xiao et al., 2020, Ray et al., 31 May 2025). The term therefore denotes a family of benchmark usages rather than a unique dataset definition.

1. Terminological scope and historical origin

The earliest usage in the supplied literature appears in "3D ShapeNets" (Wu et al., 2014), where ModelNet itself was introduced as a large-scale 3D CAD model dataset with 151,128 mesh models in 660 distinct object categories. Within that framework, ModelNet-R was defined as a retrieval protocol built on top of ModelNet rather than as a separately curated corpus. In later work, the same label was reassigned to substantially different benchmark constructions.

A second usage appears in "Enhancing 3D Point Cloud Classification with ModelNet-R and Point-SkipNet" (Saeid et al., 5 Sep 2025), where ModelNet-R denotes a meticulously refined version of ModelNet40. A third usage appears in "Triangle-Net: Towards Robustness in Point Cloud Learning" (Xiao et al., 2020), which refers to the rotated-ModelNet40 evaluation setting as ModelNet-R. A fourth, unrelated usage appears in "Towards Graph-Based Privacy-Preserving Federated Learning: ModelNet -- A ResNet-based Model Classification Dataset" (Ray et al., 31 May 2025), where ModelNet-R is the random variant of a federated-learning dataset derived from CIFAR-100 and ResNet50 embeddings.

This multiplicity of meanings has direct interpretive consequences. A plausible implication is that any citation to ModelNet-R must be read together with its source paper, task definition, and evaluation protocol; otherwise, reported accuracies, retrieval scores, or systems claims are not commensurate.

2. ModelNet-R as a retrieval benchmark in 3D ShapeNets

In the 3D ShapeNets formulation, ModelNet-R is a shape-retrieval benchmark derived from a volumetric representation of CAD meshes (Wu et al., 2014). Each mesh is embedded in a $30\times 30\times 30$ regular grid and represented by a binary tensor $v \in \{0,1\}^{30\times 30\times 30}$, where a voxel is occupied if it lies inside the mesh volume. In practice, the usable volume is $24\times 24\times 24$ with a 3-voxel padding on all sides. The associated representation is learned by a 5-layer Convolutional Deep Belief Network with binary hidden units and no pooling. The layer sequence is: input voxels; Conv1 with 48 feature-maps and $6\times 6\times 6$ filters at stride 2; Conv2 with 160 feature-maps and $5\times 5\times 5$ filters at stride 2; Conv3 with 512 feature-maps and $4\times 4\times 4$ filters at stride 1; FC-RBM4 with 1,200 hidden units; and Top-RBM5 with 4,000 hidden units plus $K$ label-units.

The retrieval protocol is defined over a gallery $G$ of test-set shapes and a query set $Q$, often with $Q=G$. Relevance is binary and category-based: two shapes are relevant when their labels are identical. Similarity is computed in feature space after extracting either the 4th-layer code $v \in \{0,1\}^{30\times 30\times 30}$0 or the 5th-layer features from the fine-tuned CDBN, and ranking is performed by ascending $v \in \{0,1\}^{30\times 30\times 30}$1 distance, $v \in \{0,1\}^{30\times 30\times 30}$2. Evaluation uses Precision@k, Average Precision, mean Average Precision, and Area Under the Precision-Recall Curve with uniformly sampled recall levels $v \in \{0,1\}^{30\times 30\times 30}$3 and linear precision interpolation.

The reported training regime for the 40-category experiments sampled 100 unique CAD models per category, producing 4,000 meshes. Rotation augmentation around the gravity axis at $v \in \{0,1\}^{30\times 30\times 30}$4 increments yielded 12 orientations per model and thus 48,000 examples, with a split of 38,400 for training and 9,600 for testing. For a 10-category subset overlapping NYU-Depth, 4,899 unique CAD models became 58,788 examples after 12-way rotation, again split 80/20.

The benchmark compared the deep volumetric feature against SPH and LFD:

Split	Descriptor	AUC / mAP
10-category subset	SPH	45.97% / 44.05%
10-category subset	LFD	51.70% / 49.82%
10-category subset	Ours	69.28% / 68.26%
40-category split	SPH	34.47% / 33.26%
40-category split	LFD	42.04% / 40.91%
40-category split	Ours	49.94% / 49.23%

The accompanying discussion attributes these gains to high-level volumetric features that capture both global shape and category-specific part structure. The paper also states that 12-view augmentation produced approximate yaw invariance despite the absence of pooling, and that the learned representation supported generative functions such as shape completion and next-best-view planning in addition to retrieval (Wu et al., 2014).

3. ModelNet-R as a refined version of ModelNet40

In the 2025 point-cloud classification paper, ModelNet-R is a refined version of ModelNet40 intended to correct benchmark deficiencies rather than to define a retrieval protocol (Saeid et al., 5 Sep 2025). The identified limitations in ModelNet40 are inconsistent labeling, multi-object scans carrying single-object labels, two-dimensional or flattened samples lacking true volumetric depth, size mismatches upon normalization, and poor class differentiation in categories such as "flower_pot" versus "vase" and "bowl" versus "cup." The paper states that these issues cause noisy training signals and reduce benchmark reliability.

The refinement pipeline has three stages. First, 2D and multi-object scans are detected and removed. Flattened samples are identified by a depth-variance test using per-point depth variance $v \in \{0,1\}^{30\times 30\times 30}$5, with removal when $v \in \{0,1\}^{30\times 30\times 30}$6; multi-object scans are removed when the connected-component count exceeds 1. Second, relabeling and class-boundary rules are applied. A sample with a handle is relabeled as "cup"; a hemisphere-shaped sample is relabeled as "bowl"; samples containing both plant and pot are assigned to "flower_pot"; samples containing pot but not plant are assigned to "vase"; and samples containing plant but not pot are assigned to "plant." Samples with ambiguous labels or unclear geometry after these rules are removed. Third, high-confusion classes are visually inspected through confusion-matrix analysis, and any sample with no consensus among experts is dropped. The reported net effect is that the five targeted classes become mutually exclusive and semantically meaningful.

Quantitatively, the paper reports that 19 ambiguous, flat, or multi-object samples were removed, leaving 1,266 exemplars across the five refined classes. It further states that variance$v \in \{0,1\}^{30\times 30\times 30}$7 variance$v \in \{0,1\}^{30\times 30\times 30}$8 for these classes. The benchmark setup is reported as using train/test splits identical to ModelNet40, with 9,843 training models and 2,468 test models, 1,024 uniformly sampled points per model, and random rotation, jitter, and scaling during training. Evaluation uses Overall Accuracy,

$v \in \{0,1\}^{30\times 30\times 30}$9

and Mean Class Accuracy,

$24\times 24\times 24$0

The same paper introduces Point-SkipNet and reports the following runtime and memory footprint on an NVIDIA RTX 3080 GPU: 1.47 M parameters, typical training time of approximately 2 hours for 200 epochs, inference of approximately 5 ms per cloud at batch size 32, and peak GPU memory of approximately 3 GB. Its summary table states uniform accuracy gains of +1–4% OA and +1–4% mAcc across leading models when switching from ModelNet40 to ModelNet-R (Saeid et al., 5 Sep 2025).

4. ModelNet-R as rotated-ModelNet40 in Triangle-Net

In the Triangle-Net paper, Xiao and Wachs use ModelNet-R to denote the rotated-ModelNet40 benchmark rather than a cleaned dataset (Xiao et al., 2020). The protocol begins from the ModelNet40 CAD models, comprising 12,311 shapes in 40 categories. Each mesh is randomly down-sampled to a fixed number of points, and a random SO(3) rotation about the object centroid is applied to every shape in both training and testing. The stated purpose is to ensure that the network never sees a canonical orientation.

This benchmark emphasizes robustness to severe sparsity and unconstrained pose. For the global sparsity study, the point budget is varied over 1,024, 256, 64, 32, 16, and 8 points, with $24\times 24\times 24$1 identified as the most challenging setting. Evaluation includes overall classification accuracy over 40 classes and, for retrieval, mean average precision in a top-$24\times 24\times 24$2 nearest-neighbor search using the learned global embedding. In multi-task settings, the paper also reports part-segmentation IoU and voxel reconstruction fidelity.

Triangle-Net itself is based on hyperedge feature extraction over sampled triangles, invariant descriptors $24\times 24\times 24$3, $24\times 24\times 24$4, and $24\times 24\times 24$5, stacked hyperedge-convolution layers with shared MLPs and max aggregation, and a global feature obtained by max-pooling across point features. The classification head is a 3-layer MLP $24\times 24\times 24$6 with ReLU, BatchNorm, and Dropout with $24\times 24\times 24$7. The paper states that Euclidean distances and vector angles are preserved under $24\times 24\times 24$8 for any $24\times 24\times 24$9, and that scale-normalization makes $6\times 6\times 6$0 scale-invariant.

Under arbitrary SO(3) rotations, the reported classification accuracies for Triangle-Net are 81.53% at 64 points, 79.28% at 32 points, 70.35% at 16 points, and 48.19% at 8 points. At $6\times 6\times 6$1, the paper states that Triangle-Net beats PointNet by 35.0% and 3DmFV by 28.1%. The training setup uses fresh random SO(3) rotations during both training and testing, random down-sampling to the target point count, no additional jitter or scale augmentation in the core classification experiments, and a single NVIDIA P100 GPU with approximately 8.5 hours of training when using $6\times 6\times 6$2 hyperedges (Xiao et al., 2020).

5. ModelNet-R as the random variant of a federated-learning dataset

A separate 2025 paper introduces yet another ModelNet-R, unrelated to 3D CAD retrieval or point-cloud recognition (Ray et al., 31 May 2025). In that work, ModelNet is a dataset built on CIFAR-100 and embeddings extracted from a pre-trained ResNet50 model. The three variants are ModelNet-S, ModelNet-D, and ModelNet-R, corresponding to homogeneous, heterogeneous, and random data settings, respectively. ModelNet-R is the random variant.

The construction begins from CIFAR-100 with 100 classes and 600 images per class. The dataset is split into 5,000 disjoint clients. For each client $6\times 6\times 6$3, a subset $6\times 6\times 6$4 of 15 classes is drawn by uniform sampling without replacement from the 100 classes, so every 15-class subset is equally likely and $6\times 6\times 6$5 for any class $6\times 6\times 6$6. If the optional number of images per class $6\times 6\times 6$7 is not specified, the full 600 images per selected class are collected, giving approximately 9,000 images per client. Each image is passed through a fixed ResNet50 feature extractor, producing a 2,048-dimensional pooled feature vector. Client $6\times 6\times 6$8 therefore holds a local dataset $6\times 6\times 6$9 with labels restricted to $5\times 5\times 5$0.

The paper positions ModelNet-R as a neutral, baseline non-IID split with no semantic or statistical bias beyond uniform randomness. It also outlines a privacy-preserving hypothesis in which the federated server sees only anonymized model parameters $5\times 5\times 5$1, although no explicit privacy guarantee such as $5\times 5\times 5$2-DP or $5\times 5\times 5$3-anonymity is specified. A FedAvg-style aggregation rule is assumed,

$5\times 5\times 5$4

with a possible graph-based extension involving a client similarity graph and a Laplacian-regularized term left to future work.

The reported descriptive statistics characterize ModelNet-R as moderately diverse and low in redundancy. The subset class-embedding diversity $5\times 5\times 5$5 peaks around 0.14, between ModelNet-D at approximately 0.15 and ModelNet-S at approximately 0.10. Class-occurrence frequencies are nearly uniform, with most classes appearing approximately 700–800 times and about 750 subset assignments expected on average from $5\times 5\times 5$6. Average pairwise overlap between ModelNet-R subsets is the lowest among the three variants, similarity to the other two variants is around 0.084, median class-overlap between two ModelNet-R subsets is approximately 2, and feature-space coverage lies between ModelNet-D and ModelNet-S (Ray et al., 31 May 2025).

6. Comparative interpretation and nomenclature

The literature therefore uses the same label for four distinct benchmark objects:

Usage of “ModelNet-R”	Construction	Source
Retrieval benchmark	Shape retrieval atop ModelNet using CDBN features and category-based relevance	(Wu et al., 2014)
Refined ModelNet40	Cleaned benchmark addressing mislabeled, flat, multi-object, and ambiguous samples	(Saeid et al., 5 Sep 2025)
Rotated ModelNet40	SO(3)-randomized evaluation protocol for point-cloud robustness	(Xiao et al., 2020)
Random FL variant	CIFAR-100/ResNet50 client-partition dataset with uniform random class subsets	(Ray et al., 31 May 2025)

A common misconception is to treat ModelNet-R as a universally recognized benchmark name. The published record represented here shows instead that the name is overloaded. In one case it is a retrieval protocol; in another it is a refined dataset; in a third it is a rotation-based evaluation condition; and in a fourth it is a random non-IID split for federated learning. This suggests that direct comparison of scores under the heading “ModelNet-R” is meaningful only when the underlying construction, input modality, and task are explicitly aligned.

The ambiguity is especially consequential because the associated metrics differ. The retrieval-oriented ModelNet-R uses AUC and mAP over category relevance; the refined ModelNet-R emphasizes OA and mAcc for classification; the rotated ModelNet-R centers on robustness under arbitrary SO(3) and extreme sparsity; and the federated-learning ModelNet-R is characterized primarily through subset diversity, redundancy, and client-level class allocation. In practice, precise disambiguation requires the cited paper, the modality—voxels, points, or image embeddings—and the protocol definition.