- The paper introduces CAD-Prompted SAM3, which conditions SAM3 on 12-view CAD renderings to segment all visible instances of a geometrically specified industrial object in one forward pass.
- The method uses synthetic domain-randomized training and a two-stage matching strategy, achieving PQ/F1 scores of 0.7385/0.7636 on 3D printing, 0.2799/0.3329 on T-LESS, and 0.4921/0.6090 on ITODD.
- The results show that geometry-based prompts improve robustness when color and texture vary, while similar shapes, clutter, limited data, and the need to enumerate CAD prompts remain important challenges.
CAD-Prompted SAM3 addresses a specific gap in promptable segmentation: industrial and 3D-printed parts whose identity is determined by geometry rather than appearance or language. The paper, authored by Tang, Nagabhirava, and Liu at Carnegie Mellon University's Robotics Institute (2602.20551), reinterprets the CAD model—the canonical specification of an engineered component—as a segmentation prompt. Rather than describing a target object with text or matching against an RGB exemplar, the method conditions SAM3 directly on multi-view renderings of the object's mesh, producing instance masks in a single forward pass.
The motivating observation is that existing prompt modalities are poorly matched to engineered objects. Language prompts struggle with components that lack descriptive names and are distinguished by subtle geometric differences that vision-LLMs resolve poorly. Image exemplars encode color and texture cues that are incidental to part identity: in rapid prototyping and small-batch manufacturing, the same component may appear in different materials, finishes, or colors across production runs. Meanwhile, CAD-based detection systems such as CNOS, MUSE, and NIDS-Net use CAD renderings only post hoc—scoring mask proposals against a library of CAD models in a two-stage pipeline—rather than as a direct conditioning signal.
The paper formalizes CAD-prompted segmentation as follows: given a target mesh M and a query RGB image I, predict binary masks for all visible instances of M. No text, category label, or real exemplar is available at inference time; the mesh is the sole object specification.
Method
The architecture reuses SAM3's image encoder, fusion transformer, detection head, and mask decoder unchanged. The contribution lies in the prompt interface:
- Multi-view rendering: each CAD mesh is rendered from V=12 canonical viewpoints on the viewing sphere using Blender, yielding RGB views with foreground masks.
- Geometry encoding: each view is passed through SAM3's image encoder, and 25 point prompts sampled within the foreground mask are fed through the GeometryEncoder, producing geometry-aware embeddings per viewpoint.
- Cross-image fusion via the text pathway: this is the key design decision. Feeding cross-image exemplar embeddings directly into the detection head (as SAM3 does for same-image exemplars) yields weak alignment due to domain mismatch between rendered and query features. Instead, the authors reinterpret the geometry embeddings as concept tokens and route them through SAM3's text–image fusion transformer. Self-attention among viewpoint tokens aggregates multi-view information before conditioning image features, and the transformer's support for variable-length token sequences allows all twelve views to be injected without architectural modification.
- Single-stage decoding: fused features feed the detection head and mask decoder, producing all instances of the prompted object in one forward pass.
Synthetic training pipeline
Fine-tuning uses roughly 9,000 CAD models from the ABC Dataset, composed into cluttered scenes via Isaac Sim's Replicator with Automated Domain Randomization across 21 parameters spanning PBR materials, HDR lighting, camera intrinsics/extrinsics, scale, pose, and distractors. Materials and colors are assigned independently of mesh identity specifically to block appearance shortcuts and force geometry-based reasoning. Ground-truth masks are free from simulation.
Training proceeds in two stages, motivated by an instability the authors report with standard one-to-one DETR-style matching: Stage 1 applies score-weighted mask supervision (BCE + Dice) over all predictions, each matched greedily to its best-IoU ground-truth mask. This stabilizes early optimization but permits prediction collapse onto single instances when multiple objects are present—an acknowledged failure mode of this objective. Stage 2 switches to one-to-many matching (up to K=5 predictions per ground truth via IoU-based greedy matching) plus a presence loss supervising detection scores, enabling multi-instance coverage. NMS removes redundant predictions at inference.
Results
Evaluation follows a prompt-based protocol—one prompt per ground-truth category per image—and reports Panoptic Quality and instance-level F1. Baselines are Matcher (DINOv2 bi-directional feature matching), PerSAM (SAM embedding similarity), and vanilla SAM3 image-exemplar prompting, which for fairness was given the same cross-image fusion mechanism used by the proposed method since it otherwise does not support cross-domain exemplars.
| Method |
3D Printing PQ / F1 |
T-LESS PQ / F1 |
ITODD PQ / F1 |
| Matcher |
0.6057 / 0.6271 |
0.2637 / 0.3050 |
0.2874 / 0.3520 |
| PerSAM |
0.2846 / 0.5035 |
0.1660 / 0.1880 |
0.2502 / 0.3621 |
| SAM3 image exemplar |
0.1713 / 0.1935 |
0.1710 / 0.1894 |
0.4152 / 0.5070 |
| CAD-Prompted SAM3 |
0.7385 / 0.7636 |
0.2799 / 0.3329 |
0.4921 / 0.6090 |
The strongest result is on the custom 3D printing dataset (8 unseen printed objects, 80 real images, ~3.5 instances per image, two colors per object): PQ improves from Matcher's 0.6057 to 0.7385, a relative gain of roughly 22%. On ITODD—texture-less metallic objects in grayscale—the method reaches PQ 0.4921 versus 0.2874 for Matcher, though it is worth noting that vanilla SAM3 exemplar prompting already achieves 0.4152 there, narrowing the margin attributable purely to CAD conditioning. Gains on T-LESS, where texture-less parts exhibit strong geometric similarity under heavy clutter and occlusion, are more modest (PQ 0.2799 vs. 0.2637 for Matcher), indicating that highly similar geometries in dense clutter remain difficult even with geometry-conditioned prompts.
Limitations and open questions
Several caveats bear directly on the reported numbers. The custom dataset is small (80 images, 8 objects), so the largest headline result carries limited statistical weight relative to the benchmarks. The method still requires rasterized renderings as an intermediate representation; the authors themselves identify direct geometric conditioning—encoding meshes, point clouds, or implicit surfaces into the foundation model's prompt space without canonical rendering—as the natural next step. The two-stage training scheme introduces hyperparameters (K=5, stage-switch timing) whose sensitivity is not analyzed. Finally, the evaluation protocol assumes one prompt per known category, leaving open how the approach scales to scenes containing many candidate CAD models, where prompt enumeration may become impractical, and whether performance degrades for meshes whose real appearances deviate beyond the randomized material distribution seen in training.
Conclusion
The paper establishes CAD-derived multi-view renderings as a viable prompt modality for foundation segmentation models, aligning perception with how industrial components are actually specified. Consistent improvements over appearance-based exemplar methods across a 3D printing dataset and the T-LESS/ITODD benchmarks support the central claim that geometry-conditioned prompting is more robust than appearance-based conditioning when surface properties vary independently of part identity. The main open directions are direct geometric encoding into the prompt space and evaluation at larger CAD-library scales.