2D-to-3D Transfer Learning

• Transfer learning from 2D to 3D is a set of methods that repurpose mature 2D feature hierarchies to efficiently bootstrap 3D models across various domains.
• Key techniques include kernel inflation, pseudo-label distillation, and permutation-invariant aggregation, which boost performance and reduce training costs.
• These methods have demonstrated significant improvements in tasks like medical image segmentation and 3D object detection, offering practical solutions for data-scarce environments.

Transfer learning from 2D to 3D refers to a group of methodologies that exploit powerful representations learned from large-scale 2D datasets and networks (typically natural images and supervised models) to initialize, constrain, or supervise the training of 3D models across domains such as medical imaging, geometry processing, robotics, and autonomous driving. These techniques address the severe data scarcity and annotation cost inherent to 3D domains by leveraging mature 2D feature hierarchies and architectures—either by direct weight mapping (kernel inflation, planar stacking), pseudo-label distillation via RGB-D, architectural composites that lift or project, or multi-modal pipelines for cross-domain knowledge transfer.

1. Core Approaches and Methodologies

The principal mechanisms for 2D-to-3D transfer fall into two broad categories: direct architectural transplantation with weight inflation, and semantic pseudo-label distillation.

Weight Inflation and Kernel Mapping: Architectures such as U-Net, ResNet, EfficientNet, and Vision Transformers, pre-trained on 2D images, can be extended to 3D by inflating their convolutional kernels (from $\mathbb{R}^{C_o \times C_i \times k \times k}$ to $$\mathbb{R}^{C_o \times C_i \times k \times k \times k}$$) either by duplicating the 2D weights along the depth axis (“copy inflation”) or by centering in the middle plane (“centering inflation”). Notably, normalization by $1/\sqrt{d}$ is used for energy conservation in planar kernel mapping (Kolarik et al., 2020). Such initializations preserve pretrained feature regularization and accelerate adaptation (Messaoudi et al., 2023, Zhang et al., 2023).

Pseudo-Label Distillation via RGB-D: A strong 2D model (e.g., DPT or ViT trained on ADE20K) is run across large unlabeled RGB images to produce pixel-wise semantic pseudo-labels. With known camera intrinsics and depth, each pixel is “lifted” to a 3D point, transferring semantic supervision directly into 3D space; models are pre-trained on this lifted set, then fine-tuned on limited 3D instances (Yu et al., 2022, Miao et al., 24 Jul 2025).

Architectural Composites and Permutation-Invariant Aggregation: For medical volumes (MRI, CT), 2D slice encoders (pretrained CNNs/ViTs) process each slice independently, followed by permutation-invariant aggregation (mean pooling or self-attention), often with learned positional embeddings encoding spatial order (Gupta et al., 2023).

Cross-Dimensional Fusion and Multi-Modal Pipelines: In “simCrossTrans” (Shen et al., 2022), 3D point clouds are rendered into 2D pseudo-images; pre-trained 2D backbones and heads (ConvNets, ViTs) are directly fine-tuned for 3D object detection tasks.

2. Mathematical Formulations of Weight Transfer and Kernel Inflation

The dominant formalism for inflating 2D kernels to 3D is:

$$W^{3D}[o,i,d,x,y] = \begin{cases} W^{2D}[o,i,x,y], & \forall d\ \frac{1}{k} W^{2D}[o,i,x,y], & \text{(L2-normalized)} \end{cases}$$

where $k$ is the kernel depth. Centering inflation assigns nonzero weights only at the central depth slice; copy inflation duplicates across all $d$. When “planar stacking” is required, normalization by $1/\sqrt{d}$ is used to conserve the Frobenius norm (Kolarik et al., 2020).

Weight transfer to 3D models is equally used for convolutional backbones and newer transformer-based encoders by expanding the patch embedding convolution across the depth dimension and optional tiling or replicating positional embeddings (Zhang et al., 2023). Decoders remain unchanged or are adapted in a mirrored fashion.

3. Cross-Dimensional Transfer in Medical Imaging

Transfer learning from 2D to 3D has been widely adopted in volumetric medical image segmentation and neuroimaging, overcoming data scarcity with minimal re-training. Recent benchmarks establish the superiority of “weight-transfer” and “kernel inflation” against randomly initialized 3D networks; e.g., average Dice scores > 91% for brain tumor segmentation in BraTS 2022 and multi-organ CT segmentation using inflated ViT or EfficientNet backbones (Messaoudi et al., 2023, Zhang et al., 2023). Permutation-invariant slice aggregation with pretrained 2D encoders achieves or surpasses traditional 3D CNNs in regression and classification tasks (brain age, Alzheimer’s detection) (Gupta et al., 2023). Planar 3D transfer stabilizes convergence and yields significant increases in Dice on highly unbalanced MRI segmentation (Kolarik et al., 2020); anisotropic hybrid networks further exploit between-slice fusion with minimal 3D parameter overhead (Liu et al., 2017).

4. Knowledge Transfer by Pseudo-Label Lifting and Semantic Correspondence

Pseudo-labeling pipelines leverage high-fidelity 2D segmentation models to supervise 3D downstream tasks by lifting per-pixel soft distributions (semantic or instance labels) into 3D space via camera depth and geometry. The transferred pseudo-labels function as distillation targets and regularizers for 3D models, yielding robust data-efficient semantic segmentation with substantial gains in mIoU under low-label regimes (Yu et al., 2022, Miao et al., 24 Jul 2025). This is generalizable to large-scale 3D dataset creation (e.g., COCO-3D) and trains generalist 3D LLMs for spatial reasoning. Cycle-based bidirectional projection schemes further enable explicit 2D–3D–2D semantic correspondence, including occlusion handling and mesh-based keypoint transfer (You et al., 2020).

5. Application Domains and Impact

Transfer learning from 2D to 3D is deployed broadly:

• Medical Imaging: Universal initialization for volumetric segmentation, lesion/tumor detection, clinical neuroimaging, enabling high accuracy with small labeled datasets (Messaoudi et al., 2023, Zhang et al., 2023, Gupta et al., 2023, Liu et al., 2017).
• Robotics & Autonomous Driving: Pseudo-label mining and frustum inflation produce large-scale 3D object cuboids for LiDAR-based detection, outperforming fully supervised approaches when mined over tens to hundreds of logs, particularly for rare classes (Wilson et al., 2020).
• 3D Recognition, Shape Analysis, Scene Understanding: Multi-view consistency losses, knowledge transfer pipelines, and feature projection architectures facilitate 3D shape classification, part segmentation, and open-vocabulary reasoning on synthetic and real-world datasets (Yan et al., 2023, T et al., 2024).
• Aero/Fluid Mechanics: Pretrained 2D models (airfoils) utilized for 3D swept-wing prediction and geometry, with parameterized adaptation layers and simple sweep-theory embedding (Li et al., 2022).

6. Quantitative Evidence and Best Practices

Across tasks and domains, 2D-to-3D transfer learning yields marked improvements:

• Dice scores in 3D segmentation increase 1–3 pp over slice-based transfer and 6–7 pp over random 3D initialization (Zhang et al., 2023).
• Data-efficient semantic segmentation on ScanNet: mIoU improvement of 4–10% under limited annotation (Yu et al., 2022).
• Object detection AP50: +13–16 points with simCrossTrans, Swin-T surpasses ConvNet by 9.7 points, depth-only models nearly match RGB SOTA (Shen et al., 2022).
• 3D pseudo-label mining for autonomous vehicles: surpasses fully-supervised models on rare categories given sufficient data volume (Wilson et al., 2020).

Recommended practices include central or averaged inflation, shallow depth windows, permutation-invariant aggregation, positional embeddings for spatial encoding, and fine-tuning under combined losses (Dice, focal, cross-entropy, adversarial, perceptual). permutation-invariant strategies and cross-modal pseudo-labeling are favored for MRI/CT and point cloud domains.

7. Limitations, Challenges, and Future Directions

Transfer learning from 2D to 3D is principally limited by input modality restrictions (availability of RGB-D or camera pose/depth), pseudo-label domain gaps and inherited biases, and geometric expressiveness—especially in pure geometric feature learning or modeling long-range context. Reliance on pre-trained 2D models may propagate their errors, and fine-grained 3D supervision remains sparse or noisy. Promising extensions include joint self-training cycles, multi-modal teachers (combining depth, masks, PLMs), scalable multi-view and video lifting (for full scene reconstruction), embedding of simple physical theories in non-vision domains (e.g., swept-wing aerodynamics), and dynamic or embodied reasoning by generalist models (Miao et al., 24 Jul 2025, T et al., 2024, Li et al., 2022).

In summary, transfer learning from 2D to 3D encompasses a rich suite of weight-mapping, distillation, aggregation, and architectural infusion techniques that collectively bridge the annotation, generalization, and scaling challenges of 3D data-driven modeling by leveraging the established representational strength and scale of 2D deep learning.