Category-Aware 3D-to-3D (C33D) Overview

• Category+3D-to-3D (C33D) is a unified framework that utilizes category-aware representations to enhance 3D reasoning for diverse tasks such as tracking, reconstruction, pose estimation, and generation.
• It leverages shared priors, canonicalization, and adaptive grouping to address large variations in object geometry, shape, and scale across categories.
• C33D methods employ varied supervision regimes—from weak to self-supervised—and demonstrate improved performance and generalization over per-class specialized models.

Category+3D-to-3D (C33D) denotes a family of category-aware or category-unified 3D learning problems in which category information, category-level priors, or shared models are used to improve 3D reasoning across instances with large variation in size, shape, pose, texture, topology, or motion. In the literature summarized here, C33D includes category-unified 3D LiDAR single-object tracking, category-centric reconstruction from images or videos, category-level pose estimation and correspondence, and cross-category 3D generation or composition; the common objective is to replace per-class specialization with representations that remain stable across heterogeneous object instances (Nie et al., 2024, Wang et al., 26 Jul 2025, Henzler et al., 2021, Xiong et al., 2 Sep 2025, Sommer et al., 27 May 2026).

1. Problem scope and task families

C33D does not refer to one single input-output protocol. Rather, it covers several task families in which category-level structure is part of the learning signal. In category-unified 3D point cloud tracking, a single tracker is trained on all categories together and operates on template and search point clouds, rather than using one tracker per class. In category-centric reconstruction, a model learns a shared 3D prior across many instances of a category and reconstructs a new instance from one or a few views. In cross-category generation or composition, the model uses text, images, or a source 3D model together with category cues to synthesize a novel 3D object. In category-level pose and correspondence, the goal is to recover pose, keypoints, or semantically corresponding 3D points across different instances of the same category (Nie et al., 2024, Reizenstein et al., 2021, Jiang et al., 2022, Sommer et al., 27 May 2026).

Task family	Representative formulation	Representative papers
Category-unified tracking	One model tracks Car, Pedestrian, Van, Cyclist from template/search point clouds	(Nie et al., 2024, Wang et al., 26 Jul 2025)
Category-centric reconstruction and generation	Single-image or multi-view input is mapped to meshes, radiance fields, triplanes, or implicit surfaces using category priors	(Novotny et al., 2020, Kokkinos et al., 2021, Huang et al., 2022, Wu et al., 2024, Wu et al., 11 Mar 2025, Jiang et al., 2022, Xiong et al., 2 Sep 2025)
Pose, keypoints, and correspondence	Shared morphable templates or neural meshes induce pose, ordered keypoints, or camera-space 3D correspondences	(Yang et al., 2023, Sommer et al., 2024, Sommer et al., 27 May 2026, Fernandez-Labrador et al., 2020)

A recurring distinction is between explicit category conditioning and category-unified learning. Some methods use category labels, captions, or category phrases as input or auxiliary supervision. Others remove class-specific model boundaries and instead learn a single shared representation whose internal structure absorbs cross-category variation. This distinction is central in unified 3D tracking, where naïvely training prior category-specific trackers jointly causes major degradation, and in weakly supervised reconstruction, where category labels can structure the latent space strongly enough to substitute for stronger geometric supervision (Nie et al., 2024, Huang et al., 2022).

2. Shared representational motifs

Despite the diversity of tasks, several representational motifs recur. One is canonicalization: an image pixel, point, or surface location is mapped to a category-level canonical entity whose identity is preserved across instances. Canonical 3D Deformer Maps formalize this as

$X(y; I) = B(\phi(y; I))\, a(I),$

where the canonical map $\phi(y; I)$ assigns each pixel to a canonical coordinate, $B(k)$ is a continuous deformation operator indexed by canonical coordinate, and $a(I)$ is an image-dependent deformation code; the resulting surface is a dense category-level deformation model rather than a fixed template mesh (Novotny et al., 2020). Morpheus uses a related idea with a shared morphable template mesh: query points are projected to the source mesh, encoded by barycentric coordinates, transferred through the shared topology, and then reprojected to the target instance, so semantic correspondence is induced by shared vertex identity rather than explicit correspondence labels (Sommer et al., 27 May 2026).

A second motif is geometry-adaptive representation learning. AdaFormer, introduced for category-unified 3D single-object tracking, combines a group regression module that learns deformable groups and adaptive receptive fields with a vector-attention mechanism that encourages feature interaction among points inside those deformable groups. The ablations show that removing group regression drops mean performance from 54.0 / 74.6 to 47.1 / 67.6, and removing both group regression and vector attention drops it to 45.9 / 63.2, indicating that adaptive grouping is the dominant factor in unified tracking across categories with different size and shape statistics (Nie et al., 2024).

A third motif is transfer from pretrained 3D models through lightweight adaptation. TrackAny3D uses a pretrained 3D point cloud model, RECON, with frozen pretrained parameters and inserts parameter-efficient adapters into Transformer blocks. Its unified input concatenates a learnable temporal token with template and search tokens to form

$$\mathbf{F}_{0} \in \mathbb{R}^{(1+N_t+N_s) \times d},$$

and its temporal context propagation updates the token by

$$\mathcal{T}_0^t = \mathcal{T}_0 + \mathcal{T}_{out}^{t-1}.$$

The same framework adds a Mixture-of-Geometry-Experts (MoGE) layer with $M$ expert FFNs and Top-K routing, so rigid and deformable objects can activate different experts without explicit class routing (Wang et al., 26 Jul 2025).

A fourth motif is explicit structured 3D latent space. Direct3D learns a latent triplane representation with a D3D-VAE and models its distribution with a D3D-DiT, while CDI3D uses a tri-plane-based reconstruction model and FlexiCubes after densifying input views by diffusion-based interpolation (Wu et al., 2024, Wu et al., 11 Mar 2025). This suggests that recent C33D-style generation increasingly treats 3D structure as a first-class latent object rather than as an implicit by-product of view synthesis.

3. Supervision regimes and optimization strategies

C33D methods span supervised, weakly supervised, self-supervised, and unsupervised regimes. At one extreme, some pipelines use strong multiview or benchmark supervision; at the other, several methods operate with no 3D ground truth, no viewpoint annotations, or no direct correspondence supervision.

Weak supervision is especially prominent in single-image reconstruction. To The Point predicts 2D locations for the vertices of a category template and a visibility vector, then estimates camera pose and non-rigid deformation through a differentiable optimization layer. Its deformation model is

$V = T + Bc,$

and its reprojection objective is

$$l(R,t,c) = \sum_{i=1}^{N} v_i \left\|u_i - \hat u_i(C,R,t)\right\|_2^2 + \gamma \|c\|_2^2.$$

Training uses weak and self-supervised losses including texture/perceptual, Chamfer-style silhouette coverage, mask, cycle, visibility, equivariance, optional keypoint reprojection, and ARAP regularization (Kokkinos et al., 2021).

An even weaker regime appears in category-guided SDF reconstruction without any 3D cues. “Planes vs. Chairs” uses only single-view masked images and category labels, with no ground-truth 3D shape, no viewpoint annotations, and no multi-view supervision. The full objective is

$$\mathcal{L}_{total} = \mathcal{L}_{recon} + \lambda_1 \mathcal{L}_{metric} + \lambda_2 \mathcal{L}_{gan} + \lambda_3 \mathcal{L}_{cam} + \lambda_4 \mathcal{L}_{SDF-SRN},$$

where the metric term organizes latent shape codes around category centers. On ShapeNet-55, the method reports [email protected] = 0.1619, [email protected] = 0.6164, [email protected] = 0.8386, CD = 0.541, compared with CD = 0.801 without category information and CD = 1.172 for SDF-SRN; the appendix further states that category labels provide roughly the benefit of 15%–20% two-view annotation in Chamfer Distance terms (Huang et al., 2022).

Self-supervised multiview learning is another major regime. Warp-conditioned ray embedding learns category-level 3D reconstruction from real object-centric videos with SfM/Colmap cameras and Mask R-CNN masks, and explicitly addresses the fact that per-video SfM coordinate systems are not globally aligned (Henzler et al., 2021). Unsupervised category-level 3D pose learning from object-centric videos first aligns coarse reconstructed meshes across videos and then trains a single-image pose model from the resulting canonicalized 3D supervision, with no human pose labels, no CAD models, and no RGB-D input (Sommer et al., 2024). In weakly supervised 3D grounding, 3DWG uses only scene-query pairs and decomposes learning into category-level and instance-level alignment, with total loss

$\phi(y; I)$0

thereby treating category ambiguity and instance complexity as separate optimization problems (Li et al., 3 May 2025).

4. Category-unified tracking and temporal reasoning

Category-unified 3D LiDAR single-object tracking is the most direct C33D setting in the strict sense of 3D point-cloud input and 3D tracking output. The standard formulation uses a template point cloud from an initial or historical frame and a search point cloud from the current frame; the tracker predicts the target’s 3D box in the search region. The primary challenge is that Car, Pedestrian, Van, and Cyclist exhibit strongly different extents, point densities, and offset distributions, so category-specific trackers do not automatically transfer to unified training (Nie et al., 2024, Wang et al., 26 Jul 2025).

The first systematic unification result is the AdaFormer line. The paper “Towards Category Unification of 3D Single Object Tracking on Point Clouds” constructs SiamCUT and MoCUT by combining adaptive shape/size-aware representation learning with unified target encoding and shape-aware positive/negative assignment. On KITTI, the unified P2B baseline improves from 42.4 / 60.0 to 54.0 / 74.6 overall, with gains on Pedestrian: 28.7 / 49.6 to 48.2 / 76.2, Van: 40.8 / 48.4 to 63.1 / 74.9, and Cyclist: 32.1 / 44.7 to 36.7 / 47.4. The paper also reports that category-unified versions outperform category-specific baselines for several trackers, including P2B: mean +11.6 Success / +14.6 Precision, PTT: mean +3.7 / +2.2, PTTR: mean +3.7 / +3.6, OSP2B: mean +1.8 / +2.1, and M$\phi(y; I)$1Track: mean +2.9 / +1.6 (Nie et al., 2024).

TrackAny3D extends this line by transferring a large-scale pretrained 3D model rather than redesigning a tracker from scratch. The implementation uses frozen pretrained RECON parameters, clip length 3 frames, 128 / 128 points per template and search, adapter bottleneck dimension 72, 8 experts with top-4 active, MoGE inserted at every even-numbered layer, and single-NVIDIA RTX3090 inference. On KITTI it reports unified Mean: 67.1 / 85.4, surpassing MoCUT in the unified setting by +1.3% overall and by +6.0% on Car; on NuScenes it reports Mean: 54.57 / 66.25; on Waymo Vehicle tracking, when trained on KITTI and directly tested on Waymo, it reports 64.0 / 73.3, demonstrating cross-dataset generalization (Wang et al., 26 Jul 2025).

These results clarify a common misconception: category-unified tracking is not merely multi-category training. Both papers state that naïvely forcing prior methods into a unified setting causes major degradation. The successful systems therefore modify the representation, routing, target normalization, and temporal modeling rather than only mixing categories in the training set. This suggests that the principal issue is not label space size but geometric conflict across categories.

5. Reconstruction, generation, and composition

Category-centric reconstruction from real imagery is a major C33D branch. CO3D provides 1.5 million frames from nearly 19,000 videos across 50 MS-COCO object categories, together with camera poses, masks, and ground-truth 3D point clouds, and uses these data to benchmark category-centric reconstruction and novel-view synthesis in the wild (Reizenstein et al., 2021). On top of such multiview supervision, WCR and NerFormer learn category-level 3D representations that generalize across instances: WCR uses warp-conditioned image features sampled at projected 3D query points, while NerFormer replaces naïve aggregation with Transformer attention along the ray-depth and source-view axes (Henzler et al., 2021, Reizenstein et al., 2021).

Single-view reconstruction methods use category priors more explicitly. C3DM reconstructs dense 3D geometry and texture from one image while establishing dense correspondences between instances, using canonical embeddings, a continuous deformation basis, and weak 2D supervision (Novotny et al., 2020). To The Point reconstructs 3D by correspondence-driven optimization over a template deformation model, and on CUB with keypoints reports about 0.765 mIoU / 93.6 PCK, while without keypoints it reports about 0.749 mIoU / 50.9 PCK (Kokkinos et al., 2021). “Planes vs. Chairs” scales weakly supervised reconstruction to large category sets, including all 55 ShapeNet categories, by using category-guided metric learning in latent shape space (Huang et al., 2022).

Generative C33D-style systems increasingly operate through structured 3D latent spaces or multiview reconstruction pipelines. Direct3D learns a native 3D generative model with a D3D-VAE and D3D-DiT, uses direct geometry supervision via semi-continuous occupancy, and reports a user study in which Quality scores are 4.41 for Direct3D versus 2.53 for the best baseline, and Consistency scores are 4.35 versus 2.66 (Wu et al., 2024). CDI3D introduces Dense View Interpolation and a tilt camera pose trajectory, then reconstructs with a tri-plane-based mesh model; on GSO it reports Chamfer Distance: 0.0101, Volume IoU: 0.6399, F-score: 0.7765, and texture metrics PSNR: 18.32, SSIM: 0.8230, LPIPS: 0.1397 (Wu et al., 11 Mar 2025).

Text-guided and cross-category generation further extend the scope of C33D. 3D-TOGO uses a text-to-views stage followed by a pixelNeRF-based views-to-3D stage and evaluates on ABO with 98 categories; it reports PSNR: 24.98, SSIM: 0.900, LPIPS: 0.092, and CLIP-score: 22.84, outperforming text-NeRF and Dreamfields (Jiang et al., 2022). The later method explicitly named C33D tackles 3D object composition from a source 3D model and a target category text prompt by disentangling texture consistency and shape accuracy through ATIH, TMDiff, SMDiff, and fusion-guided adaptive inversion. On a dataset of 110 3D-model/text pairs, it reports the best overall DINO-I, AES, and $\phi(y; I)$2, and user studies with 98 participants and 490 responses each give about 72.45% preference in the image-to-3D comparison and about 73.67% in the 3D-to-3D comparison (Xiong et al., 2 Sep 2025).

6. Pose estimation, correspondence, and keypoint structure

Another major C33D interpretation treats category-level semantics as geometry that must be aligned across instances. SyntheticP3D and CC3D formulate category-level 3D pose estimation as inverse rendering of a neural mesh with vertex features. On the broader PASCAL3D+ benchmark, SyntheticP3D+CC3D reports 76.3% at $\phi(y; I)$3, 41.4% at $\phi(y; I)$4, and 15.5^\circ median error without any real annotations; with only 10% of the real labels it reaches 86.7% / 62.4% / 8.4^\circ, and with 50% it reaches 90.7% / 71.4% / 6.9^\circ, outperforming the best supervised baseline by 10.4% at the stricter $\phi(y; I)$5 threshold (Yang et al., 2023).

Unsupervised video-based pose learning offers a different route. The object-centric video method aligns coarse meshes across videos using a weighted combination of geometry and DINOv2 appearance with a cyclic distance formulation. On CO3D alignment it reports 77.0 for $\phi(y; I)$6 accuracy, compared with 69.8 for UCD+, and on in-the-wild pose estimation it reports a PASCAL3D+ mean of 69.2 versus 46.0 for ZSP, ObjectNet3D mean 52.4 versus 42.2, and runtime 0.22 sec/sample versus 10.92 sec/sample (Sommer et al., 2024).

Category-level correspondence extends beyond pose. HouseCorr3D introduces 178k images across 50 household object categories and 280 unique object instances, with amodal labels and symmetry annotations, and Morpheus learns a shared morphable object prior from which correspondences emerge implicitly. The paper reports 3D mean [email protected]: 41.5, 3D modal mean: 43.7, 3D amodal mean: 40.8, and on a filtered real subset of ROPE reports Morpheus [email protected]: 44.7 and Morpheus [email protected]: 34.8 (Sommer et al., 27 May 2026). At a sparser structural level, unsupervised learning of category-specific symmetric 3D keypoints from point sets reports 100% correspondence/repeatability across all tested categories, model error below 1%, inclusivity around 95% on average, and about 93% semantic consistency on ShapeNet Parts (Fernandez-Labrador et al., 2020).

Taken together, these works indicate that category-level 3D understanding increasingly targets part semantics and camera-space correspondence, not only rigid pose. This is consistent with the claim that pose alone does not establish part-level equivalence across instances with different shapes.

7. Benchmarks, boundaries, and recurrent limitations

C33D research depends heavily on benchmark design because the relevant generalization axes are category variation, instance variation, viewpoint, background, and modality. CO3D prioritizes scale in real category-centric videos, whereas NAVI prioritizes annotation fidelity with 10515 alignments, 36 objects, 2298 in-the-wild images, 8217 multiview images, and manually verified 2D-3D alignments. For 30 images annotated twice, NAVI reports average 3D rotation distance: 1.7 degrees and average 3D translation distance: 0.97 millimeteres, which is why the paper describes its camera parameters as near-perfect GT (Reizenstein et al., 2021, Jampani et al., 2023).

Several papers also clarify the boundaries of the term. CAMO is explicitly described as not a pure 3D-to-3D motion transfer method, because the source is monocular 2D video even though the output is a temporally coherent sequence of deformed target meshes (Kim et al., 6 Jan 2026). NAVI is a dataset rather than a direct C33D method, and some image-to-3D systems are described as “C33D-style” or “strongly aligned with C33D” rather than direct 3D-to-3D translation (Jampani et al., 2023, Wu et al., 11 Mar 2025). This suggests that the field boundary is often organized more by category-level 3D reasoning and 3D target structure than by a single fixed input modality.

Limitations are also recurrent. Morpheus assumes a shared template with fixed connectivity and therefore cannot handle large topological changes such as missing parts; it is also sensitive to pose errors and may oversmooth thin structures (Sommer et al., 27 May 2026). The unsupervised video-based pose estimator assumes roughly rigid shape and degrades on categories with high intra-class variation or poor video coverage (Sommer et al., 2024). CDI3D explicitly depends on the quality of the main views produced in the first stage, and interpolation cannot fully correct poor anchor views (Wu et al., 11 Mar 2025). The broader pattern is that category priors improve generalization, but topology variation, occlusion, sparse evidence, and unstable cross-view alignment remain the decisive failure modes.