3D Adapter: Bridging 2D Pretraining with 3D Insight

• 3D Adapter is a family of mechanisms that inject explicit 3D structure into 2D or multimodal pretrained models for applications like volumetry, multi-view rendering, and scene perception.
• The adapters leverage parameter-efficient modules, temporal and memory-based strategies, or even optical systems to preserve existing 2D priors while enabling robust 3D tasks.
• Practical implementations have shown improvements in diagnostic accuracy, segmentation quality, and generative performance by incorporating explicit geometric feedback and cross-view consistency.

A 3D adapter is not a single canonical architecture but a family of mechanisms that inject three-dimensional structure into systems originally designed for 2D images, single views, or offline processing. In contemporary arXiv usage, the term spans parameter-efficient modules that wrap frozen 2D vision or vision-language backbones for volumetric CT, multi-view feedback branches inside diffusion samplers, memory and temporal modules for online or slice-wise 3D perception, and earlier optical or software assemblies that convert stereo inputs into stereo or multi-view outputs (Yi et al., 22 Jun 2026, Chen et al., 2024, Huang et al., 2024, Zhang et al., 2024, Gong et al., 2023, Canessa et al., 2019, Bartol et al., 2021, Lunazzi et al., 2013).

1. Scope of the term

The literature uses 3D adapter in several distinct but structurally related senses. In all cases, the adapter mediates between an existing representation and a 3D objective: volumetric diagnosis, 3D shape understanding, multi-view generation, online scene perception, or stereo visualization. The common pattern is not a specific layer type, but an auxiliary mechanism that preserves a strong pretrained prior or a conventional optical path while adding explicit 3D structure.

Usage of “3D adapter”	Adapter function	Representative work
Volumetric model wrapping	Wraps a frozen 2D VLM so it can ingest a thick-slice CT volume	Brain-Adapter (Yi et al., 22 Jun 2026)
Multi-view diffusion augmentation	Lifts denoised 2D views into 3D and feeds rendered guidance back into diffusion	MVEdit (Chen et al., 2024), 3D-Adapter (Chen et al., 2024)
Parameter-efficient 3D transfer	Adds lightweight modules to adapt 2D or multimodal backbones to 3D shapes, point clouds, or volumes	TAMM (Zhang et al., 2024), 3DSAM-adapter (Gong et al., 2023), Adapter-X (Li et al., 2024)
Optical or systems conversion	Converts stereo capture or projection into multi-view or stereo display pipelines	MORPHOLO (Canessa et al., 2019), catadioptric smartphone stereo (Bartol et al., 2021), compact stereo projection (Lunazzi et al., 2013)

This terminological breadth matters because different papers optimize different invariants. Some prioritize parameter efficiency, some geometry consistency, some temporal coherence, and some low-latency optical remapping. As a result, “3D adapter” is best read as a role in a system rather than a fixed architecture class.

2. Adapting pretrained 2D models to volumetric and 3D recognition tasks

A major line of work uses adapters to transfer strong 2D pretrained priors into 3D recognition without retraining a volumetric model from scratch. In Brain-Adapter, a frozen 2D biomedical VLM is wrapped so that each CT slice $x_i\in\mathbb{R}^{H\times W}$ is encoded by $E_V$ into slice embeddings $V=[v_1,\dots,v_N]^\top\in\mathbb{R}^{N\times d}$, while the frozen text encoder embeds the whole report and fine-grained diagnostic sentences into $T=[t_0,t_1,\dots,t_K]\in\mathbb{R}^{(K+1)\times d}$. The Text-Conditioned Attention stream treats each $t_k$ as a query over $V$,

$$A_k=\mathrm{Softmax}\!\left(\frac{t_kV^\top}{\sqrt d}\right),\qquad h_k=A_kV,$$

and aligns $h_k$ to $t_k$ with an InfoNCE loss. In parallel, an ABMIL stream aggregates the same slice embeddings into a volume representation $z=\sum_i a_iv_i$, supervised by structured multi-labels distilled from reports by an external LLM, with Asymmetric Loss for class imbalance. A cosine consistency term,

$E_V$0

aligns the global TCA representation and the MIL representation, while the Uncertainty-Aware Refinement module fuses the two streams at inference with $E_V$1 and $E_V$2. With BiomedCLIP as backbone, LoRA rank $E_V$3, and 852 NCCT studies, Brain-Adapter reaches Micro AUC $E_V$4, Macro AUC $E_V$5, and Hamming Loss $E_V$6, outperforming 3D volumetric baselines and standard 2D-MIL baselines, while also improving zero-shot cross-domain performance on CQ500 (Yi et al., 22 Jun 2026).

TAMM addresses an adjacent problem in 3D shape understanding: the limited scale of 3D datasets relative to image and language corpora. Its central claim is that image and language should not be forced into a single undifferentiated alignment space. Stage 1 introduces a CLIP Image Adapter with residual form

$E_V$7

to re-align CLIP visual features to synthetic renderings. Stage 2 then attaches two independent adapters to the 3D encoder, $E_V$8 and $E_V$9, so that a vision-focused subspace aligns to adapted image features and a semantics-focused subspace aligns to CLIP text. The reported image-text retrieval accuracy rises from $V=[v_1,\dots,v_N]^\top\in\mathbb{R}^{N\times d}$0 to $V=[v_1,\dots,v_N]^\top\in\mathbb{R}^{N\times d}$1 after CIA fine-tuning, zero-shot classification on Objaverse-LVIS improves from $V=[v_1,\dots,v_N]^\top\in\mathbb{R}^{N\times d}$2 to $V=[v_1,\dots,v_N]^\top\in\mathbb{R}^{N\times d}$3, and 5-way 10-shot linear probing on ModelNet40 improves from $V=[v_1,\dots,v_N]^\top\in\mathbb{R}^{N\times d}$4 to $V=[v_1,\dots,v_N]^\top\in\mathbb{R}^{N\times d}$5 (Zhang et al., 2024).

3DSAM-adapter applies the same transfer principle to promptable tumor segmentation. It factorizes a $V=[v_1,\dots,v_N]^\top\in\mathbb{R}^{N\times d}$6 volumetric patch embedding into a frozen $V=[v_1,\dots,v_N]^\top\in\mathbb{R}^{N\times d}$7 convolution initialized from SAM and a trainable $V=[v_1,\dots,v_N]^\top\in\mathbb{R}^{N\times d}$8 depth-wise convolution, adds a trainable depth positional table $V=[v_1,\dots,v_N]^\top\in\mathbb{R}^{N\times d}$9, reuses the original self-attention weights over 3D tokens with local sliding-window attention, and inserts a depth-wise 3D spatial adapter into each transformer block. The adapter computes

$T=[t_0,t_1,\dots,t_K]\in\mathbb{R}^{(K+1)\times d}$0

while freezing $T=[t_0,t_1,\dots,t_K]\in\mathbb{R}^{(K+1)\times d}$1 of the model and fine-tuning only $T=[t_0,t_1,\dots,t_K]\in\mathbb{R}^{(K+1)\times d}$2. With a single click per volume, it reports gains of $T=[t_0,t_1,\dots,t_K]\in\mathbb{R}^{(K+1)\times d}$3, $T=[t_0,t_1,\dots,t_K]\in\mathbb{R}^{(K+1)\times d}$4, and $T=[t_0,t_1,\dots,t_K]\in\mathbb{R}^{(K+1)\times d}$5 over domain state-of-the-art models on kidney tumor, pancreas tumor, and colon cancer segmentation, with similar performance on liver tumor (Gong et al., 2023).

A more general parameter-efficient formulation appears in Adapter-X for 3D point-cloud classification. Its Sharing Mixture of Adapters routes sub-tokens to a shared expert bank,

$T=[t_0,t_1,\dots,t_K]\in\mathbb{R}^{(K+1)\times d}$6

with inter-block parameter sharing across the stack. Combined with a block-specific Prompt Generator, the 3D instantiation on PointMAE uses about $T=[t_0,t_1,\dots,t_K]\in\mathbb{R}^{(K+1)\times d}$7 M new parameters, or $T=[t_0,t_1,\dots,t_K]\in\mathbb{R}^{(K+1)\times d}$8 of a $T=[t_0,t_1,\dots,t_K]\in\mathbb{R}^{(K+1)\times d}$9 M backbone, and reports $t_k$0, $t_k$1, and $t_k$2 OA on ScanObjectNN OBJ_BG, OBJ_ONLY, and PB_T50_RS, respectively (Li et al., 2024).

3. Geometry feedback in multi-view diffusion and 3D generation

In generative modeling, the 3D adapter typically serves as an intermediate geometry-enforcing branch between denoising steps. MVEdit is an explicit example of a training-free 3D adapter inserted into an off-the-shelf 2D latent diffusion model. At each timestep, the pipeline alternates between a “spread” pass, where each noisy view latent $t_k$3 is denoised by a frozen UNet, and a “gather” pass, where the denoised views are fit to a 3D representation $t_k$4 through

$t_k$5

Rendered RGBD images are then passed as ControlNet conditions into the next denoising step. The adapter is described as purely the optimize-and-render block plus frozen ControlNets, with no adapter training required. On an RTX A6000, full image-to-3D or text-guided texture synthesis runs in $t_k$6–$t_k$7 minutes; on 248 GSO renders it reports LPIPS $t_k$8, CLIP $t_k$9, and FID $V$0, versus a best competitor FID of approximately $V$1, and on Objaverse text-guided texture generation it reports Aesthetic $V$2, CLIP $V$3, and runtime $V$4 min (Chen et al., 2024).

The paper “3D-Adapter” formalizes a closely related but broader mechanism called 3D feedback augmentation. A plug-in branch attached to a pretrained 2D diffusion U-Net decodes intermediate features into a 3D representation, renders RGBD novel views, encodes those views with a ControlNet-style encoder, and fuses them back into the base U-Net by

$V$5

Two variants are studied: a fast feed-forward Gaussian-splatting version and a training-free neural-field/mesh version. The reported effect is a strong reduction in geometric discrepancy: on text-to-3D over 379 Objaverse objects, FID improves from $V$6 to $V$7 and MDD from $V$8 to $V$9; on image-to-3D over 248 GSO objects, FID improves from $$A_k=\mathrm{Softmax}\!\left(\frac{t_kV^\top}{\sqrt d}\right),\qquad h_k=A_kV,$$0 to $$A_k=\mathrm{Softmax}\!\left(\frac{t_kV^\top}{\sqrt d}\right),\qquad h_k=A_kV,$$1, with smaller gains in PSNR, SSIM, LPIPS, and CLIP-sim (Chen et al., 2024).

MV-Adapter addresses the same consistency problem by modifying self-attention rather than inserting explicit per-step 3D optimization. In each frozen U-Net self-attention block, the original SelfAttn is retained, duplicated into a MultiViewAttn, and optionally duplicated again into an ImageCrossAttn. All three operate in parallel on the same feature $$A_k=\mathrm{Softmax}\!\left(\frac{t_kV^\top}{\sqrt d}\right),\qquad h_k=A_kV,$$2,

$$A_k=\mathrm{Softmax}\!\left(\frac{t_kV^\top}{\sqrt d}\right),\qquad h_k=A_kV,$$3

with zero-initialized $$A_k=\mathrm{Softmax}\!\left(\frac{t_kV^\top}{\sqrt d}\right),\qquad h_k=A_kV,$$4 so that the network initially matches the pretrained model exactly. A unified condition encoder represents camera rays through per-pixel origin and direction maps and can also ingest depth, position, or normal maps. On SD2.1, MV-Adapter trains only $$A_k=\mathrm{Softmax}\!\left(\frac{t_kV^\top}{\sqrt d}\right),\qquad h_k=A_kV,$$5 M parameters compared with approximately $$A_k=\mathrm{Softmax}\!\left(\frac{t_kV^\top}{\sqrt d}\right),\qquad h_k=A_kV,$$6 M for full fine-tuning; on SDXL it uses $$A_k=\mathrm{Softmax}\!\left(\frac{t_kV^\top}{\sqrt d}\right),\qquad h_k=A_kV,$$7 M parameters and $$A_k=\mathrm{Softmax}\!\left(\frac{t_kV^\top}{\sqrt d}\right),\qquad h_k=A_kV,$$8 GB memory compared with $$A_k=\mathrm{Softmax}\!\left(\frac{t_kV^\top}{\sqrt d}\right),\qquad h_k=A_kV,$$9 B parameters and $h_k$0 GB for full fine-tuning. At $h_k$1, it reports text-to-multi-view FID $h_k$2, IS $h_k$3, and CLIP Score $h_k$4, outperforming MVDream and SPAD, and image-to-multi-view PSNR $h_k$5, SSIM $h_k$6, and LPIPS $h_k$7, outperforming Era3D (Huang et al., 2024).

LACONIC recasts the adapter as a 3D layout conditioner for controllable image creation. An unordered set of semantic 3D bounding boxes and an optional floor plan are encoded into tokens, re-expressed in camera coordinates by

$h_k$8

and injected into every cross-attention layer of a frozen Stable Diffusion v1.5 UNet. Text and layout are fused by

$h_k$9

The adapter size is approximately $t_k$0 M–$t_k$1 M parameters, under $t_k$2 of the full model, and the method supports camera control, semantic restyling, and object-level editing by modifying $t_k$3, $t_k$4, $t_k$5, or $t_k$6 in the scene layout (Maillard et al., 4 Jul 2025).

AdaptSplat extends adapter logic to feed-forward 3D Gaussian Splatting. Its sole customization is a $t_k$7 M-parameter Frequency-Preserving Adapter that extracts high-frequency priors from shallow VFM features using a single-level 2D DWT, keeping $t_k$8 and $t_k$9 subbands as a directional prior, injects them into transformer attention as

$z=\sum_i a_iv_i$0

and modulates decoder skip connections by $z=\sum_i a_iv_i$1. On RE10K, the base model reports PSNR $z=\sum_i a_iv_i$2, SSIM $z=\sum_i a_iv_i$3, and LPIPS $z=\sum_i a_iv_i$4; on DL3DV it improves over MVP across all tested view budgets; and on RE10K the fractional anisotropy rises from $z=\sum_i a_iv_i$5 without FPA to $z=\sum_i a_iv_i$6 with FPA (Xing et al., 11 May 2026).

4. Memory, temporal context, and 3D continuity

Another major family of 3D adapters exists to impose continuity across time or adjacent slices. In the memory-based adapter framework for online 3D scene perception, two plug-and-play adapters $z=\sum_i a_iv_i$7 and $z=\sum_i a_iv_i$8 are inserted after the image and point-cloud backbones. A queued 3D memory stores active voxels, updated by

$z=\sum_i a_iv_i$9

while a separate image memory stores a shifted channel subset of projected image features. The point-cloud adapter queries a local sub-memory, applies a zero-initialized sparse 3D convolution, unvoxelizes the result, and adds it residually; the image adapter concatenates current features with the previous 2D memory and applies a zero-initialized 2D convolution. A 3D-to-2D adapter further projects active voxels back to image space to inject global scene context. On ScanNet, the method raises semantic segmentation mIoU from $E_V$00 to $E_V$01, object detection mAP@25/mAP@50 from $E_V$02 to $E_V$03, and instance segmentation from $E_V$04 to $E_V$05 (Xu et al., 2024).

T-Gated Adapter is a medical segmentation analogue built for adjacent-slice reasoning in a 2D VLM. A center slice and $E_V$06 neighbors are encoded in parallel into $E_V$07 tokens, reshaped so that each spatial token location becomes a length-$E_V$08 slice sequence, and processed by a temporal transformer over the slice dimension. A spatial context block then attends over the $E_V$09 within-slice positions, after which an adaptive gate computes

$E_V$10

with $E_V$11 so the gate begins near zero. The loss combines BCE, Dice, and a gating penalty $E_V$12 with $E_V$13. Training on 30 FLARE22 volumes, the method reaches mean Dice $E_V$14 across 13 organs, compared with $E_V$15 for the CLIPSeg baseline; zero-shot performance rises to $E_V$16 on BTCV and $E_V$17 on AMOS22 CT, with the cross-domain drop reduced from $E_V$18 to $E_V$19. On AMOS22 MRI, without MRI supervision, it reaches $E_V$20 mean Dice versus $E_V$21 for DynUNet trained exclusively on CT (Khadka, 9 Apr 2026).

SAM4EM introduces 3D memory attention into a prompt-free SAM adapter for EM stacks. Its slice-wise loop uses the previous slice’s predicted mask to generate prompt embeddings for a coarse Stage I decoder and a refined Stage II decoder. The memory mechanism forms

$E_V$22

and updates each memory slot by exponential moving average,

$E_V$23

The model uses $E_V$24 memory slots and dual-stage LoRA with $E_V$25 and $E_V$26. On the released “Mice-Dataset” and Lucchi, it reports Dice/mIoU of $E_V$27 for Mice-Glia, $E_V$28 for Mice-Mito, $E_V$29 for Mice-Syn, and $E_V$30 for Lucchi, exceeding prompt-free baselines such as H-SAM, SAMed, and UN-SAM (Shah et al., 30 Apr 2025).

These memory and temporal adapters make explicit that 3D awareness is often a coherence problem rather than only a representation problem. The adapter is the mechanism that transports information across slices, frames, or viewpoints while keeping the base extractor largely intact.

5. Optical and systems interpretations

Long before neural adapter tuning became common, the term was already used for optical or software intermediaries that convert 2D or stereo inputs into 3D outputs. MORPHOLO is a C++ library organized into Input Handling, Calibration Manager, Morphing Engine, and Output Renderer/Streamer. Its Morphing Engine synthesizes $E_V$31 intermediate views from a stereo pair using disparity-based morphing or DeepFlow optical-flow morphing, packs them into a quilt, and uses a precomputed LUT to map the quilt to a slanted lenticular display. The native sub-pixel mapping is

$E_V$32

In a real-time pipeline with capture, processing, quilt assembly, LUT application, and HDMI or ffmpeg streaming, the reported wall-clock time is $E_V$33 s for $E_V$34 views using the LUT, implying $E_V$35 Native frames/sec at small quilt sizes (Canessa et al., 2019).

The catadioptric smartphone adapter uses planar mirrors so that front and back cameras form a stereo pair through virtual camera reflections. For a mirror plane with unit normal $E_V$36 and offset $E_V$37, the reflection operator is

$E_V$38

giving a virtual camera orientation $E_V$39 and center $E_V$40. In the prototype, $E_V$41 and $E_V$42 cm imply $E_V$43 cm. With $E_V$44, $E_V$45 cm, $E_V$46 cm, and $E_V$47, the paper reports $E_V$48, $E_V$49 m, and retained common-FOV percentage $E_V$50. Calibration over 14 stereo pairs yields mean reprojection error $E_V$51 pixels, and 3D human-pose reconstruction gives mean absolute error $E_V$52 cm across six anthropometric measurements (Bartol et al., 2021).

“A Compact System for Registering and Projecting Stereo Views” uses mirror adapters both at capture and projection. For capture, mirror widths satisfy

$E_V$53

while in projection a symmetric four-mirror zig-zag path enforces $E_V$54 and $E_V$55, with lateral shift $E_V$56. The system supports polarization-based stereo, anaglyph stereo, and goggles-less holographic stereo. With an original $E_V$57 lumen projector, the reported net luminance is approximately $E_V$58 lm in polarized mode and $E_V$59 lm in anaglyph mode; crosstalk is about $E_V$60 with linear polarizers on a metallized screen and $E_V$61–$E_V$62 with red-cyan anaglyphs; and side-by-side projection halves horizontal resolution (Lunazzi et al., 2013).

These optical systems show that the adapter concept historically included physically embodied geometry conversion, calibration, and beam routing. The later neural literature preserves the same mediating role, although the implementation shifts from mirrors and LUTs to attention heads, memory banks, and rendering loops.

6. Recurrent design principles, misconceptions, and open problems

Across the neural literature, a recurrent design principle is to preserve a strong pretrained backbone and localize 3D specialization in a small set of modules. Brain-Adapter freezes the text encoder and adapts the 2D vision encoder with LoRA rank $E_V$63 (Yi et al., 22 Jun 2026). TAMM trains only the CIA, IAA, TAA, and the 3D encoder in Stage 2 while keeping CLIP frozen (Zhang et al., 2024). MV-Adapter freezes all original U-Net weights and optimizes only the adapter and condition encoder (Huang et al., 2024). AdaptSplat adds a single $E_V$64 M-parameter module, described as $E_V$65 of the full model (Xing et al., 11 May 2026). This suggests that, in current practice, a 3D adapter often functions as a low-capacity structural bias layered on top of a high-capacity 2D prior.

A common misconception is that a 3D adapter is always a learned PEFT module. The literature is more heterogeneous. MVEdit states that its 3D Adapter is training-free and consists of per-timestep 3D reconstruction plus rendering coupled with frozen ControlNets (Chen et al., 2024). The optimization branch of 3D-Adapter likewise requires no additional network training beyond the base models and uses Instant-NGP and DMTet as the 3D substrate (Chen et al., 2024). MORPHOLO is a C++ library organized around morphing, calibration, and LUT-based display conversion, and the smartphone and compact stereo systems are physical mirror assemblies rather than learned networks (Canessa et al., 2019, Bartol et al., 2021).

A second misconception is that simply feeding multiple views or slices to a 2D model yields robust 3D behavior. Several papers explicitly argue the opposite. TAMM identifies both domain shift between rendered and natural images and a conflict between visual and semantic alignment spaces (Zhang et al., 2024). T-Gated Adapter is motivated by the observation that independently processing 2D slices produces noisy and anatomically implausible segmentations that violate continuity (Khadka, 9 Apr 2026). MV-Adapter argues that invasive full fine-tuning of T2I models can degrade image quality and that duplicated parallel attention is needed to inherit pretrained priors while learning new 3D geometry (Huang et al., 2024). The recurring lesson is that 3D consistency is not automatic; it must be encoded by a specific mechanism such as temporal attention, dual-stream agreement, shared 3D rendering, or explicit camera-conditioned attention.

The reported limitations also differ by subfield. TAMM notes that very large 3D backbones remain untested and that complex objects may require better rendering pipelines (Zhang et al., 2024). AdaptSplat reports strong photometric and perceptual gains but does not report Chamfer distance, and its current formulation relies on a specific DINOv3-ConvNeXt backbone and DWT design (Xing et al., 11 May 2026). MV-Adapter is parameter-efficient relative to full fine-tuning but still reports $E_V$66 GB memory on SDXL (Huang et al., 2024). The catadioptric smartphone adapter remains constrained by a small baseline of approximately $E_V$67 cm and a common FOV of about $E_V$68, which increases depth uncertainty at recording distances above $E_V$69 m (Bartol et al., 2021).

A plausible implication is that 3D adapter has become a design pattern rather than a single algorithmic recipe. In that pattern, a pretrained 2D system is retained for its representation quality, while the adapter supplies the missing 3D inductive bias through one of four routes: geometric feedback from rendering, temporal or memory aggregation, cross-modal decomposition, or explicit scene/layout conditioning (Yi et al., 22 Jun 2026, Chen et al., 2024, Xu et al., 2024, Maillard et al., 4 Jul 2025).