---
title: Category-Aware 3D-to-3D (C33D) Overview
url: https://www.emergentmind.com/topics/category-3d-to-3d-c33d
type: topic
---

# Category-Aware 3D-to-3D (C33D) Overview

Category+3D-to-3D (C33D) denotes a family of category-aware or category-unified 3D learning problems in which category information, category-level priors, or shared models are used to improve 3D reasoning across instances with large variation in size, shape, pose, texture, topology, or motion. In the literature summarized here, C33D includes category-unified 3D LiDAR single-object tracking, category-centric reconstruction from images or videos, category-level pose estimation and correspondence, and cross-category 3D generation or composition; the common objective is to replace per-class specialization with representations that remain stable across heterogeneous object instances [2401.11204; 2507.19908; 2103.16552; 2509.02357; 2605.28257].

## 1. Problem scope and task families

C33D does not refer to one single input-output protocol. Rather, it covers several task families in which category-level structure is part of the learning signal. In category-unified 3D point cloud tracking, a single tracker is trained on all categories together and operates on template and search point clouds, rather than using one tracker per class. In category-centric reconstruction, a model learns a shared 3D prior across many instances of a category and reconstructs a new instance from one or a few views. In cross-category generation or composition, the model uses text, images, or a source 3D model together with category cues to synthesize a novel 3D object. In category-level pose and correspondence, the goal is to recover pose, keypoints, or semantically corresponding 3D points across different instances of the same category [2401.11204; 2109.00512; 2212.01103; 2605.28257].

| Task family | Representative formulation | Representative papers |
|---|---|---|
| Category-unified tracking | One model tracks Car, Pedestrian, Van, Cyclist from template/search point clouds | [2401.11204], [2507.19908] |
| Category-centric reconstruction and generation | Single-image or multi-view input is mapped to meshes, radiance fields, triplanes, or implicit surfaces using category priors | [2008.12709], [2106.05662], [2204.10235], [2405.14832], [2503.08005], [2212.01103], [2509.02357] |
| Pose, keypoints, and correspondence | Shared morphable templates or neural meshes induce pose, ordered keypoints, or camera-space 3D correspondences | [2305.16124], [2407.04384], [2605.28257], [2003.07619] |

A recurring distinction is between explicit category conditioning and category-unified learning. Some methods use category labels, captions, or category phrases as input or auxiliary supervision. Others remove class-specific model boundaries and instead learn a single shared representation whose internal structure absorbs cross-category variation. This distinction is central in unified 3D tracking, where naïvely training prior category-specific trackers jointly causes major degradation, and in weakly supervised reconstruction, where category labels can structure the latent space strongly enough to substitute for stronger geometric supervision [2401.11204; 2204.10235].

## 2. Shared representational motifs

Despite the diversity of tasks, several representational motifs recur. One is canonicalization: an image pixel, point, or surface location is mapped to a category-level canonical entity whose identity is preserved across instances. Canonical 3D Deformer Maps formalize this as
$$
X(y; I) = B(\phi(y; I))\, a(I),
$$
where the canonical map $\phi(y; I)$ assigns each pixel to a canonical coordinate, $B(k)$ is a continuous deformation operator indexed by canonical coordinate, and $a(I)$ is an image-dependent deformation code; the resulting surface is a dense category-level deformation model rather than a fixed template mesh [2008.12709]. Morpheus uses a related idea with a shared morphable template mesh: query points are projected to the source mesh, encoded by barycentric coordinates, transferred through the shared topology, and then reprojected to the target instance, so semantic correspondence is induced by shared vertex identity rather than explicit correspondence labels [2605.28257].

A second motif is geometry-adaptive representation learning. AdaFormer, introduced for category-unified 3D single-object tracking, combines a group regression module that learns deformable groups and adaptive receptive fields with a vector-attention mechanism that encourages feature interaction among points inside those deformable groups. The ablations show that removing group regression drops mean performance from **54.0 / 74.6** to **47.1 / 67.6**, and removing both group regression and vector attention drops it to **45.9 / 63.2**, indicating that adaptive grouping is the dominant factor in unified tracking across categories with different size and shape statistics [2401.11204].

A third motif is transfer from pretrained 3D models through lightweight adaptation. TrackAny3D uses a pretrained 3D point cloud model, RECON, with frozen pretrained parameters and inserts parameter-efficient adapters into Transformer blocks. Its unified input concatenates a learnable temporal token with template and search tokens to form
$$
\mathbf{F}_{0} \in \mathbb{R}^{(1+N_t+N_s) \times d},
$$
and its temporal context propagation updates the token by
$$
\mathcal{T}_0^t = \mathcal{T}_0 + \mathcal{T}_{out}^{t-1}.
$$
The same framework adds a Mixture-of-Geometry-Experts (MoGE) layer with $M$ expert FFNs and Top-K routing, so rigid and deformable objects can activate different experts without explicit class routing [2507.19908].

A fourth motif is explicit structured 3D latent space. Direct3D learns a latent triplane representation with a D3D-VAE and models its distribution with a D3D-DiT, while CDI3D uses a tri-plane-based reconstruction model and FlexiCubes after densifying input views by diffusion-based interpolation [2405.14832; 2503.08005]. This suggests that recent C33D-style generation increasingly treats 3D structure as a first-class latent object rather than as an implicit by-product of view synthesis.

## 3. Supervision regimes and optimization strategies

C33D methods span supervised, weakly supervised, self-supervised, and unsupervised regimes. At one extreme, some pipelines use strong multiview or benchmark supervision; at the other, several methods operate with no 3D ground truth, no viewpoint annotations, or no direct correspondence supervision.

Weak supervision is especially prominent in single-image reconstruction. To The Point predicts 2D locations for the vertices of a category template and a visibility vector, then estimates camera pose and non-rigid deformation through a differentiable optimization layer. Its deformation model is
$$
V = T + Bc,
$$
and its reprojection objective is
$$
l(R,t,c) = \sum_{i=1}^{N} v_i \left\|u_i - \hat u_i(C,R,t)\right\|_2^2 + \gamma \|c\|_2^2.
$$
Training uses weak and self-supervised losses including texture/perceptual, Chamfer-style silhouette coverage, mask, cycle, visibility, equivariance, optional keypoint reprojection, and ARAP regularization [2106.05662].

An even weaker regime appears in category-guided SDF reconstruction without any 3D cues. “Planes vs. Chairs” uses only single-view masked images and category labels, with no ground-truth 3D shape, no viewpoint annotations, and no multi-view supervision. The full objective is
$$
\mathcal{L}_{total} = \mathcal{L}_{recon} + \lambda_1 \mathcal{L}_{metric} + \lambda_2 \mathcal{L}_{gan} + \lambda_3 \mathcal{L}_{cam} + \lambda_4 \mathcal{L}_{SDF-SRN},
$$
where the metric term organizes latent shape codes around category centers. On ShapeNet-55, the method reports **F-Score@1.0 = 0.1619**, **F-Score@5.0 = 0.6164**, **F-Score@10.0 = 0.8386**, **CD = 0.541**, compared with **CD = 0.801** without category information and **CD = 1.172** for SDF-SRN; the appendix further states that category labels provide roughly the benefit of **15%–20% two-view annotation** in Chamfer Distance terms [2204.10235].

Self-supervised multiview learning is another major regime. Warp-conditioned ray embedding learns category-level 3D reconstruction from real object-centric videos with SfM/Colmap cameras and Mask R-CNN masks, and explicitly addresses the fact that per-video SfM coordinate systems are not globally aligned [2103.16552]. Unsupervised category-level 3D pose learning from object-centric videos first aligns coarse reconstructed meshes across videos and then trains a single-image pose model from the resulting canonicalized 3D supervision, with no human pose labels, no CAD models, and no RGB-D input [2407.04384]. In weakly supervised 3D grounding, 3DWG uses only scene-query pairs and decomposes learning into category-level and instance-level alignment, with total loss
$$
\mathcal{L} = \lambda_1 \mathcal{L}_{se} + \lambda_2 \mathcal{L}_{PN} + \lambda_3 \mathcal{L}_{phr.} + \lambda_4 \mathcal{L}_{rel.},
$$
thereby treating category ambiguity and instance complexity as separate optimization problems [2505.01809].

## 4. Category-unified tracking and temporal reasoning

Category-unified 3D LiDAR single-object tracking is the most direct C33D setting in the strict sense of 3D point-cloud input and 3D tracking output. The standard formulation uses a template point cloud from an initial or historical frame and a search point cloud from the current frame; the tracker predicts the target’s 3D box in the search region. The primary challenge is that Car, Pedestrian, Van, and Cyclist exhibit strongly different extents, point densities, and offset distributions, so category-specific trackers do not automatically transfer to unified training [2401.11204; 2507.19908].

The first systematic unification result is the AdaFormer line. The paper “Towards Category Unification of 3D Single Object Tracking on Point Clouds” constructs SiamCUT and MoCUT by combining adaptive shape/size-aware representation learning with unified target encoding and shape-aware positive/negative assignment. On KITTI, the unified P2B baseline improves from **42.4 / 60.0** to **54.0 / 74.6** overall, with gains on **Pedestrian: 28.7 / 49.6** to **48.2 / 76.2**, **Van: 40.8 / 48.4** to **63.1 / 74.9**, and **Cyclist: 32.1 / 44.7** to **36.7 / 47.4**. The paper also reports that category-unified versions outperform category-specific baselines for several trackers, including **P2B: mean +11.6 Success / +14.6 Precision**, **PTT: mean +3.7 / +2.2**, **PTTR: mean +3.7 / +3.6**, **OSP2B: mean +1.8 / +2.1**, and **M$^2$Track: mean +2.9 / +1.6** [2401.11204].

TrackAny3D extends this line by transferring a large-scale pretrained 3D model rather than redesigning a tracker from scratch. The implementation uses frozen pretrained RECON parameters, clip length **3 frames**, **128 / 128** points per template and search, adapter bottleneck dimension **72**, **8 experts** with **top-4 active**, MoGE inserted at **every even-numbered layer**, and single-**NVIDIA RTX3090** inference. On KITTI it reports unified **Mean: 67.1 / 85.4**, surpassing MoCUT in the unified setting by **+1.3% overall** and by **+6.0%** on **Car**; on NuScenes it reports **Mean: 54.57 / 66.25**; on Waymo Vehicle tracking, when trained on KITTI and directly tested on Waymo, it reports **64.0 / 73.3**, demonstrating cross-dataset generalization [2507.19908].

These results clarify a common misconception: category-unified tracking is not merely multi-category training. Both papers state that naïvely forcing prior methods into a unified setting causes major degradation. The successful systems therefore modify the representation, routing, target normalization, and temporal modeling rather than only mixing categories in the training set. This suggests that the principal issue is not label space size but geometric conflict across categories.

## 5. Reconstruction, generation, and composition

Category-centric reconstruction from real imagery is a major C33D branch. CO3D provides **1.5 million frames** from **nearly 19,000 videos** across **50 MS-COCO object categories**, together with camera poses, masks, and ground-truth 3D point clouds, and uses these data to benchmark category-centric reconstruction and novel-view synthesis in the wild [2109.00512]. On top of such multiview supervision, WCR and NerFormer learn category-level 3D representations that generalize across instances: WCR uses warp-conditioned image features sampled at projected 3D query points, while NerFormer replaces naïve aggregation with Transformer attention along the ray-depth and source-view axes [2103.16552; 2109.00512].

Single-view reconstruction methods use category priors more explicitly. C3DM reconstructs dense 3D geometry and texture from one image while establishing dense correspondences between instances, using canonical embeddings, a continuous deformation basis, and weak 2D supervision [2008.12709]. To The Point reconstructs 3D by correspondence-driven optimization over a template deformation model, and on CUB with keypoints reports about **0.765 mIoU / 93.6 PCK**, while without keypoints it reports about **0.749 mIoU / 50.9 PCK** [2106.05662]. “Planes vs. Chairs” scales weakly supervised reconstruction to large category sets, including all **55 ShapeNet categories**, by using category-guided metric learning in latent shape space [2204.10235].

Generative C33D-style systems increasingly operate through structured 3D latent spaces or multiview reconstruction pipelines. Direct3D learns a native 3D generative model with a D3D-VAE and D3D-DiT, uses direct geometry supervision via semi-continuous occupancy, and reports a user study in which **Quality** scores are **4.41** for Direct3D versus **2.53** for the best baseline, and **Consistency** scores are **4.35** versus **2.66** [2405.14832]. CDI3D introduces Dense View Interpolation and a tilt camera pose trajectory, then reconstructs with a tri-plane-based mesh model; on GSO it reports **Chamfer Distance: 0.0101**, **Volume IoU: 0.6399**, **F-score: 0.7765**, and texture metrics **PSNR: 18.32**, **SSIM: 0.8230**, **LPIPS: 0.1397** [2503.08005].

Text-guided and cross-category generation further extend the scope of C33D. 3D-TOGO uses a text-to-views stage followed by a pixelNeRF-based views-to-3D stage and evaluates on ABO with **98 categories**; it reports **PSNR: 24.98**, **SSIM: 0.900**, **LPIPS: 0.092**, and **CLIP-score: 22.84**, outperforming text-NeRF and Dreamfields [2212.01103]. The later method explicitly named C33D tackles 3D object composition from a source 3D model and a target category text prompt by disentangling texture consistency and shape accuracy through ATIH, TMDiff, SMDiff, and fusion-guided adaptive inversion. On a dataset of **110 3D-model/text pairs**, it reports the best overall **DINO-I**, **AES**, and **\(F_{\text{sim}}\)**, and user studies with **98 participants** and **490 responses each** give about **72.45%** preference in the image-to-3D comparison and about **73.67%** in the 3D-to-3D comparison [2509.02357].

## 6. Pose estimation, correspondence, and keypoint structure

Another major C33D interpretation treats category-level semantics as geometry that must be aligned across instances. SyntheticP3D and CC3D formulate category-level 3D pose estimation as inverse rendering of a neural mesh with vertex features. On the broader PASCAL3D+ benchmark, SyntheticP3D+CC3D reports **76.3%** at $\pi/6$, **41.4%** at $\pi/18$, and **15.5^\circ** median error without any real annotations; with only **10%** of the real labels it reaches **86.7% / 62.4% / 8.4^\circ**, and with **50%** it reaches **90.7% / 71.4% / 6.9^\circ**, outperforming the best supervised baseline by **10.4%** at the stricter $\pi/18$ threshold [2305.16124].

Unsupervised video-based pose learning offers a different route. The object-centric video method aligns coarse meshes across videos using a weighted combination of geometry and DINOv2 appearance with a cyclic distance formulation. On CO3D alignment it reports **77.0** for **\(30^\circ\) accuracy**, compared with **69.8** for UCD+, and on in-the-wild pose estimation it reports a PASCAL3D+ mean of **69.2** versus **46.0** for ZSP, ObjectNet3D mean **52.4** versus **42.2**, and runtime **0.22 sec/sample** versus **10.92 sec/sample** [2407.04384].

Category-level correspondence extends beyond pose. HouseCorr3D introduces **178k images** across **50 household object categories** and **280 unique object instances**, with amodal labels and symmetry annotations, and Morpheus learns a shared morphable object prior from which correspondences emerge implicitly. The paper reports **3D mean PCK@0.1: 41.5**, **3D modal mean: 43.7**, **3D amodal mean: 40.8**, and on a filtered real subset of ROPE reports **Morpheus 2D@0.1: 44.7** and **Morpheus 3D@0.1: 34.8** [2605.28257]. At a sparser structural level, unsupervised learning of category-specific symmetric 3D keypoints from point sets reports **100% correspondence/repeatability** across all tested categories, **model error below 1%**, **inclusivity around 95% on average**, and about **93% semantic consistency** on ShapeNet Parts [2003.07619].

Taken together, these works indicate that category-level 3D understanding increasingly targets part semantics and camera-space correspondence, not only rigid pose. This is consistent with the claim that pose alone does not establish part-level equivalence across instances with different shapes.

## 7. Benchmarks, boundaries, and recurrent limitations

C33D research depends heavily on benchmark design because the relevant generalization axes are category variation, instance variation, viewpoint, background, and modality. CO3D prioritizes scale in real category-centric videos, whereas NAVI prioritizes annotation fidelity with **10515 alignments**, **36 objects**, **2298 in-the-wild images**, **8217 multiview images**, and manually verified 2D-3D alignments. For 30 images annotated twice, NAVI reports average **3D rotation distance: 1.7 degrees** and average **3D translation distance: 0.97 millimeteres**, which is why the paper describes its camera parameters as near-perfect GT [2109.00512; 2306.09109].

Several papers also clarify the boundaries of the term. CAMO is explicitly described as **not a pure 3D-to-3D motion transfer method**, because the source is **monocular 2D video** even though the output is a temporally coherent sequence of deformed target meshes [2601.02716]. NAVI is a dataset rather than a direct C33D method, and some image-to-3D systems are described as “C33D-style” or “strongly aligned with C33D” rather than direct 3D-to-3D translation [2306.09109; 2503.08005]. This suggests that the field boundary is often organized more by category-level 3D reasoning and 3D target structure than by a single fixed input modality.

Limitations are also recurrent. Morpheus assumes a shared template with fixed connectivity and therefore cannot handle large topological changes such as missing parts; it is also sensitive to pose errors and may oversmooth thin structures [2605.28257]. The unsupervised video-based pose estimator assumes roughly rigid shape and degrades on categories with high intra-class variation or poor video coverage [2407.04384]. CDI3D explicitly depends on the quality of the main views produced in the first stage, and interpolation cannot fully correct poor anchor views [2503.08005]. The broader pattern is that category priors improve generalization, but topology variation, occlusion, sparse evidence, and unstable cross-view alignment remain the decisive failure modes.

Source: https://www.emergentmind.com/topics/category-3d-to-3d-c33d