LGAA: Modular Adapter for 3D Assets

• LGAA is a plug-in adapter that converts multi-view diffusion features into 2D Gaussian representations for generating relightable 3D assets.
• Its modular design—including a Wrapper, Switcher, and Decoder—preserves pretrained priors and aligns geometry with physically based rendering channels.
• LGAA enables end-to-end synthesis of detailed mesh assets with efficient training on limited data, reducing the need for large 3D datasets.

Lightweight Gaussian Asset Adapter (LGAA) is a plug-in, modular adapter for lifting pre-trained multi-view (MV) diffusion models into generators of relightable 3D assets with physically based rendering (PBR) materials. In DreamLifting, LGAA reuses internal feature maps from latent-diffusion UNets, preserves the priors encoded by frozen MV diffusion backbones, and adapts those features to predict 2D Gaussian Splatting (2DGS) representations with geometry and PBR channels, followed by post-processing into mesh assets (Yin et al., 9 Sep 2025). Within the supplied literature, the term also has a broader methodological resonance: a Gaussian probabilistic adapter for robust parameter-efficient fine-tuning of frozen GNNs provides an adapter-centric blueprint, and self-adaptive compression methods for 3D Gaussian assets provide an asset-centric blueprint, but the explicit designation “Lightweight Gaussian Asset Adapter” is introduced in DreamLifting (Jiang et al., 24 Nov 2025, Zhang et al., 5 Aug 2025).

1. Conceptual definition and problem setting

LGAA addresses a specific gap in 3D asset generation pipelines: many diffusion-based methods emphasize geometry while either baking appearance into vertex colors or relegating texture synthesis to post-processing with image diffusion models. The DreamLifting formulation targets end-to-end generation of relightable assets by unifying geometry and PBR material prediction in a single pipeline, rather than treating material decomposition as a downstream step (Yin et al., 9 Sep 2025).

The motivating observation is that modern MV diffusion models such as MVDream, DreamView, ImageDream, and IDArb are fine-tuned from Stable Diffusion on billions of 2D images and therefore encode strong MV-consistency priors for RGB, geometry-related cues, and, in some cases, PBR decomposition. LGAA exploits these priors directly instead of training a new native 3D generative model that would require a substantially larger 3D dataset and heavier compute (Yin et al., 9 Sep 2025). This suggests that LGAA is best understood not as a standalone backbone, but as an adaptation layer that preserves the representational structure of MV diffusion and redirects it toward 3D Gaussian assets.

In the DreamLifting pipeline, diffusion denoising proceeds normally, feature maps are tapped from selected UNet blocks at a chosen noise timestep, and the adapter stack converts those features into 2DGS with PBR channels for eight views. The resulting Gaussian representation is then converted into a relightable mesh through TSDF fusion, remeshing, UV unprojection, and differentiable refinement (Yin et al., 9 Sep 2025).

2. Modular architecture: Wrapper, Switcher, and Decoder

LGAA is organized into three components: the LGAA Wrapper (LGAA-W), the LGAA Switcher (LGAA-S), and the LGAA Decoder (LGAA-D). Their division of labor is architectural rather than merely implementation-specific: Wrappers preserve and adapt pretrained MV-diffusion features, Switchers align multiple priors, and the Decoder maps adapted features into Gaussian primitives with PBR channels (Yin et al., 9 Sep 2025).

Component	Function	Key mechanism
LGAA Wrapper	Reuses and adapts UNet layers	Frozen copied block plus zero-initialized $1\times1$ convolutions
LGAA Switcher	Aligns multiple diffusion priors	Bidirectional residual exchange via zero-initialized $1\times1$ convolutions
LGAA Decoder	Predicts 2DGS with PBR channels	Tamed VAE decoder with modified input/output layers

The Wrapper surrounds a pretrained UNet block $\mathcal{F}$ without altering its core parameters $\Theta$. If the original block computes

$$\bm{Y} = \mathcal{F}(\bm{X}_{in}, \bm{X}_{res}, \bm{C}; \Theta),$$

the wrapped version keeps the block frozen and inserts zero-initialized $1\times1$ convolutions $\mathcal{ZC}(\cdot)$: $$\begin{array}{lcl} \bm{Y} &=& \mathcal{F}(\bm{X}_{in}, \bm{X}_{res}, \bm{C}; \Theta) \ \bm{Y}' &=& \mathcal{F}\big(\bm{X}_{in}, \bm{X}_{res} + \mathcal{ZC}(\bm{X}_{res}), \bm{C}; \Theta\big) + \mathcal{ZC}(\bm{Y}). \end{array}$$ Because $\mathcal{ZC}$ is initialized to zero, the wrapped block is functionally identical to the original block at initialization. Training then learns small residual corrections on the skip pathway and on the block output, producing a knowledge-preserving adaptation mechanism rather than wholesale fine-tuning (Yin et al., 9 Sep 2025).

The Switcher is used when multiple priors are present, for example an RGB or geometry branch together with a PBR branch. For branch outputs $\bm{Y}_g$ and $1\times1$0, the Switcher computes

$1\times1$1

This preserves branch independence at initialization and introduces progressive cross-branch residual routing during training. The paper interprets this as a lightweight alignment mechanism for geometry and material priors, avoiding the inconsistency that would arise if separate diffusion streams remained decoupled (Yin et al., 9 Sep 2025).

The Decoder is described as a tamed variational autoencoder. DreamLifting reuses the pretrained Stable Diffusion VAE decoder, freezes most of it, unlocks the input layers so they can accept LGAA-processed features, and replaces the RGB output head with a Gaussian head that predicts 2DGS parameters and PBR attributes. No new VAE KL term is added; the VAE is used as a pretrained upsampling and feature-decoding module rather than retrained as a full generative autoencoder (Yin et al., 9 Sep 2025).

3. Gaussian representation and supervised training

The decoder outputs a multi-channel splatter image in which each pixel corresponds to a 2D Gaussian primitive. For each view and pixel $1\times1$2, the predicted primitive is

$1\times1$3

where $1\times1$4 is 3D position, $1\times1$5 is rotation, $1\times1$6 is scale, $1\times1$7 is opacity, $1\times1$8 is view-dependent color, $1\times1$9 is albedo, and $\mathcal{F}$0 are metallic and roughness (Yin et al., 9 Sep 2025). This representation functions as an intermediate between 2D diffusion features and a final 3D mesh with UV-mapped PBR textures.

Training is carried out on a filtered 69k-instance subset of G-Objaverse, with 38 views per instance and supervision derived from RGB, alpha, normal, depth, albedo, metallic, and roughness maps. Each training sample uses four orthogonal and four random views. The principal image loss for RGB and albedo is

$\mathcal{F}$1

with $\mathcal{F}$2, $\mathcal{F}$3, and $\mathcal{F}$4. Additional supervision includes

$\mathcal{F}$5

and

$\mathcal{F}$6

together with the depth distortion loss and normal consistency loss adopted from 2DGS (Yin et al., 9 Sep 2025).

A distinctive part of the methodology is the deferred shading loss. Because direct per-channel supervision can permit ambiguous decompositions such as illumination baked into albedo, DreamLifting computes a differentiable shaded image from predicted albedo, normal, roughness, metallic, and recovered lighting quantities, then supervises that shaded image with the same color loss. The outgoing radiance is decomposed as

$\mathcal{F}$7

with diffuse and specular terms approximated using image-based lighting and split-sum formulations: $\mathcal{F}$8

$\mathcal{F}$9

This loss couples material maps to appearance and is intended to improve the physical plausibility of the predicted PBR decomposition (Yin et al., 9 Sep 2025).

Training proceeds in two stages: 20 epochs of direct G-buffer supervision, followed by 20 epochs of fine-tuning with deferred shading on about 10k high-quality instances with reliable recovered environment maps. The full system is trained with mixed precision on 8 NVIDIA H20 GPUs, with training and fine-tuning taking about three days (Yin et al., 9 Sep 2025).

4. Conditioning, lifting mechanism, and knowledge preservation

LGAA does not introduce a new conditioning interface. Instead, it inherits the conditioning mechanisms of the underlying MV diffusion model and attaches to its intermediate features. For text-conditioned models such as MVDream and DreamView, text embeddings are produced by the Stable Diffusion text encoder and injected through UNet cross-attention together with view embeddings. For image-conditioned models such as ImageDream, image encoders or image-conditioned latent mechanisms shape the same UNet feature space. LGAA therefore operates on representations that already encode prompt, image, and camera-view information (Yin et al., 9 Sep 2025).

The lifting mechanism is consequently architectural rather than symbolic. MV diffusion models remain frozen; their middle and upsampling blocks are wrapped; multiple priors, such as RGB and PBR, are aligned through Switchers; and the resulting features are decoded into Gaussian primitives. The data-efficient aspect follows from this reuse: the paper attributes efficient convergence to the knowledge-preserving scheme and notes that the model is trained on merely 69k multi-view instances (Yin et al., 9 Sep 2025).

A closely related adapter logic appears in uncertainty-aware GNN fine-tuning, where a frozen backbone is augmented with lightweight Gaussian probabilistic adapters. In that setting, a deterministic bottleneck adapter is replaced by a Gaussian module parameterized by mean and variance,

$\Theta$0

sampled through

$\Theta$1

and fused with the frozen backbone through a learnable scale: $\Theta$2 The supplied synthesis explicitly characterizes this as the kind of design one would examine when building an LGAA-like module for large pretrained models (Jiang et al., 24 Nov 2025). This suggests a broader interpretation of “Gaussian adapter” in which lightweight residual modules can be probabilistic, uncertainty-aware, and attached to frozen pretrained backbones, although DreamLifting itself does not use this probabilistic formulation.

5. Mesh extraction and PBR asset realization

LGAA’s direct output is 2DGS with PBR channels, not a mesh. DreamLifting therefore adds a post-processing pipeline to convert the Gaussian representation into a relightable, watertight mesh asset suitable for downstream graphics workflows (Yin et al., 9 Sep 2025).

Geometry extraction begins by rendering albedo and depth from the 2DGS along a circular camera path with elevations $\Theta$3, $\Theta$4, and $\Theta$5, plus top and bottom views. These depth maps are fused into a TSDF volume using Open3D’s ScalableTSDFVolume with voxel size 0.008 and truncation 0.02, and marching cubes extracts an initial mesh. A convex hull is then computed to fill holes, and Continuous Remeshing is run for 100 iterations while matching rendered normal and alpha maps, yielding a watertight and relatively uniform-triangle mesh (Yin et al., 9 Sep 2025).

PBR material construction proceeds through UV unwrapping, texture initialization, and differentiable refinement. UV maps are created with Blender’s Smart UV Project. Albedo, metallic, and roughness maps rendered from the 2DGS are unprojected onto the mesh UVs. Refinement then uses Mitsuba 3 and Dr.Jit, following the Objects-With-Lighting setup, rendering orthogonal views under recovered environment lighting and optimizing the mesh textures so that the rendered results match the corresponding MV diffusion outputs or 2DGS renders (Yin et al., 9 Sep 2025). The paper reports a total post-process time of under 30 seconds on an RTX 4090.

Within the broader supplied literature, a second asset-oriented interpretation of LGAA appears in Gaussian compression. SA-3DGS describes a self-adaptive pipeline that learns a per-Gaussian importance score $\Theta$6, prunes Gaussians using $\Theta$7 and opacity, compresses spherical-harmonic color coefficients with importance-aware codebooks, and repairs compressed attributes with a lightweight residual MLP: $\Theta$8 The accompanying synthesis explicitly frames this as relevant to designing a “Lightweight Gaussian Asset Adapter” that converts full 3DGS assets into compact, portable Gaussian representations (Zhang et al., 5 Aug 2025). A plausible implication is that “LGAA” spans two adjacent problem classes in the supplied sources: generative lifting from MV diffusion to 3D Gaussian assets, and post-training adaptation or compression of Gaussian assets for storage and deployment.

6. Empirical performance, ablations, and limitations

DreamLifting evaluates LGAA in both text-conditioned and image-conditioned settings. On 1,000 unseen prompts from G-Objaverse, MVDream 2.1 with LGAA achieves FID 36.38, $\Theta$9 41.47, and CLIP 33.70, compared with 3DTopia-XL at FID 64.96, $$\bm{Y} = \mathcal{F}(\bm{X}_{in}, \bm{X}_{res}, \bm{C}; \Theta),$$0 47.19, and CLIP 29.26. Against TRELLIS, LGAA yields worse pure FID, 36.38 versus 27.08, but higher CLIP, 33.70 versus 31.25; the supplied comparison also notes that TRELLIS does not perform PBR separation and instead uses vertex colors (Yin et al., 9 Sep 2025). For image-conditioned generation, ImageDream with LGAA improves over 3DTopia-XL on RGB PSNR, 17.04 versus 13.36; LPIPS, 0.227 versus 0.334; albedo PSNR, 17.62 versus 13.42; metallic MSE, 0.049 versus 0.122; and roughness MSE, 0.015 versus 0.037 (Yin et al., 9 Sep 2025).

Ablations are central to the paper’s interpretation of LGAA’s design. Replacing frozen wrapped blocks with unfrozen clone layers, denoted “w/o LGAA-W,” degrades FID from 36.38 to 54.52 and $$\bm{Y} = \mathcal{F}(\bm{X}_{in}, \bm{X}_{res}, \bm{C}; \Theta),$$1 from 41.47 to 74.31. Removing the Switcher yields worse geometry–material alignment and notably worse $$\bm{Y} = \mathcal{F}(\bm{X}_{in}, \bm{X}_{res}, \bm{C}; \Theta),$$2. Replacing the tamed VAE decoder with PixelShuffle, denoted “w/o LGAA-D,” introduces grid artifacts in albedo and lowers quality (Yin et al., 9 Sep 2025). These observations support the paper’s emphasis on modularity, preservation of pretrained knowledge, and gentle cross-prior alignment.

The authors also state two limitations. First, because the pretrained MV diffusion models are frozen and only the adapters are trained, the training data should follow conventions similar to the original MV diffusion fine-tuning data, such as object-centric framing and comparable field-of-view and viewpoint distributions. Second, supervision is purely pixel-level, through images and G-buffers, with no explicit volumetric regularization or internal 3D prior enforcing object-interior consistency (Yin et al., 9 Sep 2025). This suggests that LGAA’s current strengths lie in relightable asset synthesis and data-efficient adaptation, while internal structural consistency remains an open area for extension.

Within the broader technical landscape represented by the supplied sources, LGAA occupies a distinctive position. Unlike native 3D diffusion systems that require large 3D corpora, it lifts existing 2D MV priors; unlike vertex-color Gaussian or mesh methods, it explicitly predicts PBR channels; and unlike Gaussian compression methods such as predictive splat forests or importance-aware codebooks, it is primarily a generative adapter rather than a post hoc storage format (Cao et al., 2024, Zhang et al., 5 Aug 2025). At the same time, those neighboring lines of work indicate that lightweight Gaussian assets can be studied from multiple angles: modular adaptation of frozen pretrained models, uncertainty-aware Gaussian residual modeling, and compression-aware Gaussian asset packaging.