---
title: ModelNet-R Benchmark Family
url: https://www.emergentmind.com/topics/modelnet-r
type: topic
---

# ModelNet-R Benchmark Family

ModelNet-R is not a single standardized resource. Across arXiv papers, the name has been used for several non-equivalent constructs: a shape-retrieval benchmark derived from the original ModelNet CAD collection; a refined version of ModelNet40 intended to correct labeling, geometric, and class-boundary issues; a rotated-ModelNet40 evaluation protocol for SO(3)-robust point-cloud learning; and a random client-partition variant of a CIFAR-100-based federated-learning dataset [1406.5670] [2509.05198] [2003.00856] [2506.00476]. The term therefore denotes a family of benchmark usages rather than a unique dataset definition.

## 1. Terminological scope and historical origin

The earliest usage in the supplied literature appears in "3D ShapeNets" [1406.5670], where ModelNet itself was introduced as a large-scale 3D CAD model dataset with 151,128 mesh models in 660 distinct object categories. Within that framework, ModelNet-R was defined as a retrieval protocol built on top of ModelNet rather than as a separately curated corpus. In later work, the same label was reassigned to substantially different benchmark constructions.

A second usage appears in "Enhancing 3D Point Cloud Classification with ModelNet-R and Point-SkipNet" [2509.05198], where ModelNet-R denotes a meticulously refined version of ModelNet40. A third usage appears in "Triangle-Net: Towards Robustness in Point Cloud Learning" [2003.00856], which refers to the rotated-ModelNet40 evaluation setting as ModelNet-R. A fourth, unrelated usage appears in "Towards Graph-Based Privacy-Preserving Federated Learning: ModelNet -- A ResNet-based Model Classification Dataset" [2506.00476], where ModelNet-R is the random variant of a federated-learning dataset derived from CIFAR-100 and ResNet50 embeddings.

This multiplicity of meanings has direct interpretive consequences. A plausible implication is that any citation to ModelNet-R must be read together with its source paper, task definition, and evaluation protocol; otherwise, reported accuracies, retrieval scores, or systems claims are not commensurate.

## 2. ModelNet-R as a retrieval benchmark in 3D ShapeNets

In the 3D ShapeNets formulation, ModelNet-R is a shape-retrieval benchmark derived from a volumetric representation of CAD meshes [1406.5670]. Each mesh is embedded in a $30\times 30\times 30$ regular grid and represented by a binary tensor $v \in \{0,1\}^{30\times 30\times 30}$, where a voxel is occupied if it lies inside the mesh volume. In practice, the usable volume is $24\times 24\times 24$ with a 3-voxel padding on all sides. The associated representation is learned by a 5-layer Convolutional Deep Belief Network with binary hidden units and no pooling. The layer sequence is: input voxels; Conv1 with 48 feature-maps and $6\times 6\times 6$ filters at stride 2; Conv2 with 160 feature-maps and $5\times 5\times 5$ filters at stride 2; Conv3 with 512 feature-maps and $4\times 4\times 4$ filters at stride 1; FC-RBM4 with 1,200 hidden units; and Top-RBM5 with 4,000 hidden units plus $K$ label-units.

The retrieval protocol is defined over a gallery $G$ of test-set shapes and a query set $Q$, often with $Q=G$. Relevance is binary and category-based: two shapes are relevant when their labels are identical. Similarity is computed in feature space after extracting either the 4th-layer code $h^{(4)}$ or the 5th-layer features from the fine-tuned CDBN, and ranking is performed by ascending $L^2$ distance, $d(q,g)=\|h_q-h_g\|_2$. Evaluation uses Precision@k, Average Precision, mean Average Precision, and Area Under the Precision-Recall Curve with uniformly sampled recall levels $\{0,0.01,\dots,1\}$ and linear precision interpolation.

The reported training regime for the 40-category experiments sampled 100 unique CAD models per category, producing 4,000 meshes. Rotation augmentation around the gravity axis at $30^\circ$ increments yielded 12 orientations per model and thus 48,000 examples, with a split of 38,400 for training and 9,600 for testing. For a 10-category subset overlapping NYU-Depth, 4,899 unique CAD models became 58,788 examples after 12-way rotation, again split 80/20.

The benchmark compared the deep volumetric feature against SPH and LFD:

| Split | Descriptor | AUC / mAP |
|---|---|---|
| 10-category subset | SPH | 45.97% / 44.05% |
| 10-category subset | LFD | 51.70% / 49.82% |
| 10-category subset | Ours | 69.28% / 68.26% |
| 40-category split | SPH | 34.47% / 33.26% |
| 40-category split | LFD | 42.04% / 40.91% |
| 40-category split | Ours | 49.94% / 49.23% |

The accompanying discussion attributes these gains to high-level volumetric features that capture both global shape and category-specific part structure. The paper also states that 12-view augmentation produced approximate yaw invariance despite the absence of pooling, and that the learned representation supported generative functions such as shape completion and next-best-view planning in addition to retrieval [1406.5670].

## 3. ModelNet-R as a refined version of ModelNet40

In the 2025 point-cloud classification paper, ModelNet-R is a refined version of ModelNet40 intended to correct benchmark deficiencies rather than to define a retrieval protocol [2509.05198]. The identified limitations in ModelNet40 are inconsistent labeling, multi-object scans carrying single-object labels, two-dimensional or flattened samples lacking true volumetric depth, size mismatches upon normalization, and poor class differentiation in categories such as "flower_pot" versus "vase" and "bowl" versus "cup." The paper states that these issues cause noisy training signals and reduce benchmark reliability.

The refinement pipeline has three stages. First, 2D and multi-object scans are detected and removed. Flattened samples are identified by a depth-variance test using per-point depth variance $\sigma_z^2$, with removal when $\sigma_z^2 < \tau_{\text{depth}}$; multi-object scans are removed when the connected-component count exceeds 1. Second, relabeling and class-boundary rules are applied. A sample with a handle is relabeled as "cup"; a hemisphere-shaped sample is relabeled as "bowl"; samples containing both plant and pot are assigned to "flower_pot"; samples containing pot but not plant are assigned to "vase"; and samples containing plant but not pot are assigned to "plant." Samples with ambiguous labels or unclear geometry after these rules are removed. Third, high-confusion classes are visually inspected through confusion-matrix analysis, and any sample with no consensus among experts is dropped. The reported net effect is that the five targeted classes become mutually exclusive and semantically meaningful.

Quantitatively, the paper reports that 19 ambiguous, flat, or multi-object samples were removed, leaving 1,266 exemplars across the five refined classes. It further states that variance$(N_{R,c}) \ll$ variance$(N_{40,c})$ for these classes. The benchmark setup is reported as using train/test splits identical to ModelNet40, with 9,843 training models and 2,468 test models, 1,024 uniformly sampled points per model, and random rotation, jitter, and scaling during training. Evaluation uses Overall Accuracy,
$$
\text{OA} = \frac{\#\text{ correct}}{\#\text{ total}} \times 100\%,
$$
and Mean Class Accuracy,
$$
\text{mAcc} = \frac{1}{K}\sum_{c=1}^K \frac{\text{TP}_c}{N_{\text{test},c}} \times 100\%.
$$

The same paper introduces Point-SkipNet and reports the following runtime and memory footprint on an NVIDIA RTX 3080 GPU: 1.47 M parameters, typical training time of approximately 2 hours for 200 epochs, inference of approximately 5 ms per cloud at batch size 32, and peak GPU memory of approximately 3 GB. Its summary table states uniform accuracy gains of +1–4% OA and +1–4% mAcc across leading models when switching from ModelNet40 to ModelNet-R [2509.05198].

## 4. ModelNet-R as rotated-ModelNet40 in Triangle-Net

In the Triangle-Net paper, Xiao and Wachs use ModelNet-R to denote the rotated-ModelNet40 benchmark rather than a cleaned dataset [2003.00856]. The protocol begins from the ModelNet40 CAD models, comprising 12,311 shapes in 40 categories. Each mesh is randomly down-sampled to a fixed number of points, and a random SO(3) rotation about the object centroid is applied to every shape in both training and testing. The stated purpose is to ensure that the network never sees a canonical orientation.

This benchmark emphasizes robustness to severe sparsity and unconstrained pose. For the global sparsity study, the point budget is varied over 1,024, 256, 64, 32, 16, and 8 points, with $n=16$ identified as the most challenging setting. Evaluation includes overall classification accuracy over 40 classes and, for retrieval, mean average precision in a top-$K$ nearest-neighbor search using the learned global embedding. In multi-task settings, the paper also reports part-segmentation IoU and voxel reconstruction fidelity.

Triangle-Net itself is based on hyperedge feature extraction over sampled triangles, invariant descriptors $D_A$, $D_B$, and $D_C$, stacked hyperedge-convolution layers with shared MLPs and max aggregation, and a global feature obtained by max-pooling across point features. The classification head is a 3-layer MLP $(512 \rightarrow 256 \rightarrow 40)$ with ReLU, BatchNorm, and Dropout with $p=0.3$. The paper states that Euclidean distances and vector angles are preserved under $x \mapsto Rx+t$ for any $R \in SO(3)$, and that scale-normalization makes $D_C$ scale-invariant.

Under arbitrary SO(3) rotations, the reported classification accuracies for Triangle-Net are 81.53% at 64 points, 79.28% at 32 points, 70.35% at 16 points, and 48.19% at 8 points. At $n=16$, the paper states that Triangle-Net beats PointNet by 35.0% and 3DmFV by 28.1%. The training setup uses fresh random SO(3) rotations during both training and testing, random down-sampling to the target point count, no additional jitter or scale augmentation in the core classification experiments, and a single NVIDIA P100 GPU with approximately 8.5 hours of training when using $F=4,096$ hyperedges [2003.00856].

## 5. ModelNet-R as the random variant of a federated-learning dataset

A separate 2025 paper introduces yet another ModelNet-R, unrelated to 3D CAD retrieval or point-cloud recognition [2506.00476]. In that work, ModelNet is a dataset built on CIFAR-100 and embeddings extracted from a pre-trained ResNet50 model. The three variants are ModelNet-S, ModelNet-D, and ModelNet-R, corresponding to homogeneous, heterogeneous, and random data settings, respectively. ModelNet-R is the random variant.

The construction begins from CIFAR-100 with 100 classes and 600 images per class. The dataset is split into 5,000 disjoint clients. For each client $i$, a subset $S_i$ of 15 classes is drawn by uniform sampling without replacement from the 100 classes, so every 15-class subset is equally likely and $P(c \in S_i)=15/100$ for any class $c$. If the optional number of images per class $I$ is not specified, the full 600 images per selected class are collected, giving approximately 9,000 images per client. Each image is passed through a fixed ResNet50 feature extractor, producing a 2,048-dimensional pooled feature vector. Client $i$ therefore holds a local dataset $D_i=\{(x_{ij}, y_{ij})\}$ with labels restricted to $S_i$.

The paper positions ModelNet-R as a neutral, baseline non-IID split with no semantic or statistical bias beyond uniform randomness. It also outlines a privacy-preserving hypothesis in which the federated server sees only anonymized model parameters $\tilde W_i^t = A(W_i^t)$, although no explicit privacy guarantee such as $\epsilon$-DP or $k$-anonymity is specified. A FedAvg-style aggregation rule is assumed,
$$
W^t = \sum_{i=1}^N p_i \,\tilde W_i^t,
$$
with a possible graph-based extension involving a client similarity graph and a Laplacian-regularized term left to future work.

The reported descriptive statistics characterize ModelNet-R as moderately diverse and low in redundancy. The subset class-embedding diversity $\mathcal{D}(S)$ peaks around 0.14, between ModelNet-D at approximately 0.15 and ModelNet-S at approximately 0.10. Class-occurrence frequencies are nearly uniform, with most classes appearing approximately 700–800 times and about 750 subset assignments expected on average from $5{,}000 \times 15 / 100$. Average pairwise overlap between ModelNet-R subsets is the lowest among the three variants, similarity to the other two variants is around 0.084, median class-overlap between two ModelNet-R subsets is approximately 2, and feature-space coverage lies between ModelNet-D and ModelNet-S [2506.00476].

## 6. Comparative interpretation and nomenclature

The literature therefore uses the same label for four distinct benchmark objects:

| Usage of “ModelNet-R” | Construction | Source |
|---|---|---|
| Retrieval benchmark | Shape retrieval atop ModelNet using CDBN features and category-based relevance | [1406.5670] |
| Refined ModelNet40 | Cleaned benchmark addressing mislabeled, flat, multi-object, and ambiguous samples | [2509.05198] |
| Rotated ModelNet40 | SO(3)-randomized evaluation protocol for point-cloud robustness | [2003.00856] |
| Random FL variant | CIFAR-100/ResNet50 client-partition dataset with uniform random class subsets | [2506.00476] |

A common misconception is to treat ModelNet-R as a universally recognized benchmark name. The published record represented here shows instead that the name is overloaded. In one case it is a retrieval protocol; in another it is a refined dataset; in a third it is a rotation-based evaluation condition; and in a fourth it is a random non-IID split for federated learning. This suggests that direct comparison of scores under the heading “ModelNet-R” is meaningful only when the underlying construction, input modality, and task are explicitly aligned.

The ambiguity is especially consequential because the associated metrics differ. The retrieval-oriented ModelNet-R uses AUC and mAP over category relevance; the refined ModelNet-R emphasizes OA and mAcc for classification; the rotated ModelNet-R centers on robustness under arbitrary SO(3) and extreme sparsity; and the federated-learning ModelNet-R is characterized primarily through subset diversity, redundancy, and client-level class allocation. In practice, precise disambiguation requires the cited paper, the modality—voxels, points, or image embeddings—and the protocol definition.

Source: https://www.emergentmind.com/topics/modelnet-r