Transfer VAE Training (TVT) Overview

• TVT is a staged training paradigm that transfers pretrained latent interfaces, decoders, or cores to a subsequent training phase for improved model performance.
• It encompasses diverse strategies such as aggregated β-VAE training, quantize-then-rectify conversion, and latent-resolution transfer, addressing challenges in both image and video domains.
• By decoupling representation learning from task-specific reconstruction, TVT effectively balances latent Gaussianity with reconstruction fidelity, enabling advances in applications like shape completion and super-resolution.

Transfer VAE Training (TVT) is a staged training paradigm in which a variational autoencoder, or a model derived from one, is not optimized only through a single end-to-end objective. Instead, a latent distribution, decoder interface, architectural core, or pretrained VAE is first learned or reused, and that intermediate structure is then transferred to a second training phase. In the literature, the label has been used for several distinct procedures: a two-stage $\beta$-VAE that transfers an aggregated Gaussian posterior to a decoupled decoder, conversion of a pretrained continuous VAE into a VQ-VAE by training only a quantizer and rectifier, adaptation of diffusion-model VAEs to new latent resolutions or to 3D video latents while preserving compatibility with pretrained backbones, and progressive resolution growth from a low-resolution VAE core on sparse, imbalanced data (Li et al., 2022, Zhang et al., 14 Jul 2025, Yi et al., 27 Jul 2025, Zhao et al., 2024, Utyamishev et al., 2019).

1. Conceptual scope and recurring structure

The surveyed literature suggests that TVT is not a single canonical algorithm but a family of transfer-based VAE procedures. What remains common is the explicit reuse of a previously trained latent interface, decoder, or core representation, followed by a second optimization stage in which some components are frozen and others are adapted. In that sense, TVT sits between classical transfer learning and conventional end-to-end VAE training: the transferred object is not only weights, but often a latent geometry or a decoder-compatible representation.

A compact summary of the main instantiations is given below.

Instantiation	Transferred object	Fixed during transfer
Aggregated $\beta$-VAE (Li et al., 2022)	Learned Gaussian posterior $z \sim q_{\theta}(z|x)$	Stage-1 encoder
ReVQ (Zhang et al., 14 Jul 2025)	Pretrained VAE latent space and decoder tolerance	Encoder $E$ and decoder $D$
Real-ISR TVT (Yi et al., 27 Jul 2025)	8$\times$ SD-VAE latent interface into a 4$\times$ VAE	Stage-2 decoder; pretrained UNet largely reused
CV-VAE (Zhao et al., 2024)	SD image-VAE latent space into a 3D video VAE	Image-VAE regularization branches
Progressive VAE (Utyamishev et al., 2019)	Low-resolution core weights	Previously trained inner layers

A recurring implication is that TVT is chiefly a method for separating roles that are entangled in ordinary VAE optimization. In some cases the separation is between KL regularization and reconstruction, in others between continuous latent modeling and discrete tokenization, and in others between latent compatibility and task-specific reconstruction fidelity.

2. Decoupled $\beta$-VAE training via aggregated posterior transfer

In the formulation introduced for 3D skull reconstruction and shape completion, TVT is a two-stage $\beta$-VAE procedure that transfers a learned Gaussian posterior to a decoupled decoder (Li et al., 2022). The starting point is the standard ELBO,

$$\mathcal{L}(\theta,\phi; x)=\mathbb{E}_{q_{\theta}(z|x)}[\log p_{\phi}(x|z)]-\mathrm{KL}(q_{\theta}(z|x)\,\Vert\,p(z)),$$

and its $\beta$0-VAE variant,

$\beta$1

For Gaussian posterior $\beta$2 with $\beta$3 and prior $\beta$4, the reverse KL is

$\beta$5

The paper further gives a gradient-level account:

$\beta$6

and an information-theoretic decomposition,

$\beta$7

These formulas are used to explain why large $\beta$8 improves Gaussianity and latent continuity but degrades reconstruction fidelity.

Stage 1 learns $\beta$9 with strong KL pressure. The implementation uses a 3D conv net with 6 convolutional layers, two linear heads producing $z \sim q_{\theta}(z|x)$0 and $z \sim q_{\theta}(z|x)$1, a 6-layer 3D deconvolutional decoder, latent dimensionality $z \sim q_{\theta}(z|x)$2, reparameterization $z \sim q_{\theta}(z|x)$3, prior $z \sim q_{\theta}(z|x)$4, Dice loss for reconstruction, and KL weight $z \sim q_{\theta}(z|x)$5. Training runs for 200 epochs in MONAI. A baseline is trained at $z \sim q_{\theta}(z|x)$6 for 200 epochs to illustrate antagonism. Stage 2 freezes the Stage-1 encoder, samples $z \sim q_{\theta}(z|x)$7, and trains a new decoder with independent parameters for 1200 epochs using only reconstruction loss,

$z \sim q_{\theta}(z|x)$8

The reported dataset is SkullFix, with 100 complete skulls and 100 cranial defects provided by the dataset, plus 100 facial defects additionally created from complete skulls, for a total of 300 training samples. Volumetric CT-derived skull masks are downsampled from $z \sim q_{\theta}(z|x)$9 to $E$0. Reconstruction quality is measured by Dice similarity coefficient. Mean reconstruction DSC on test samples is 0.9153, 0.9215, and 0.9199 for cranial, facial, and complete skulls with $E$1, versus 0.9076, 0.9094, and 0.9127 for the aggregated VAE at $E$2. Mean completion DSC is 0.9189 and 0.9072 for cranial and facial defects with $E$3, versus 0.8978 and 0.7934 with $E$4. The paper states that the aggregated VAE “maximally satisfies” the Gaussian assumption and that the decoupled decoder recovers reconstruction performance comparable to $E$5 end-to-end training.

The same work also uses latent guidance for shape completion. Let $E$6, $E$7, and $E$8 denote centroids for complete, cranially defective, and facially defective skulls. Then

$E$9

and, for a test latent $D$0,

$D$1

This is used to connect existing AE-based shape-completion heuristics to VAE-style generative manipulation.

3. Converting a pretrained VAE into a VQ-VAE: quantize-then-rectify TVT

A second use of the TVT idea appears in "Quantize-then-Rectify: Efficient VQ-VAE Training," where a pretrained continuous VAE is converted into a discrete tokenizer by freezing the original encoder and decoder and training only a quantizer and a latent-space rectifier (Zhang et al., 14 Jul 2025). The encoder $D$2 produces a latent feature map $D$3, and the frozen decoder $D$4 is treated as a black box for reconstruction evaluation. The inserted modules are a channel multi-group quantizer $D$5 and a lightweight post rectifier $D$6, with rectified latent

$D$7

The training loss is latent-space $D$8 only,

$D$9

The transfer mechanism is explicitly tied to the pretrained VAE’s noise tolerance. Two empirical criteria are reported. First, when Gaussian noise of variance $\times$0 is injected into the VAE latent, reconstructed images retain good perceptual quality; above this threshold, reconstructions degrade markedly. Second, a latent MSE below $\times$1 yields reconstructions acceptable to the human eye. These motivate the criterion

$\times$2

with empirical choice $\times$3.

Quantization is performed by channel multi-group quantization. With $\times$4, channels are partitioned into $\times$5 groups, each with its own codebook $\times$6. Group-wise nearest-neighbor assignment is

$\times$7

Backpropagation uses the straight-through estimator. ReVQ does not introduce explicit commitment or codebook losses. To prevent codebook collapse, non-activation reset is applied at the end of each epoch by resetting inactive codes near highly activated codes with a small perturbation $\times$8.

The reported implementation starts from a pretrained DC-AE. For 512 tokens, the hyperparameters are $\times$9, batch size $\times$0, quantizer learning rate $\times$1, rectifier learning rate $\times$2 of the quantizer learning rate, 100 epochs, and a single RTX 4090. The rectifier is an EfficientViT block with 3 layers for 512 tokens and 4 layers for 256 tokens. On ImageNet validation, ReVQ with 512 tokens and $\times$3 achieves rFID $\times$4, PSNR $\times$5, SSIM $\times$6, and LPIPS $\times$7. The 256-token settings report rFID $\times$8, PSNR $\times$9, SSIM $\beta$0, LPIPS $\beta$1 for $\beta$2, and rFID $\beta$3, PSNR $\beta$4, SSIM $\beta$5, LPIPS $\beta$6 for $\beta$7. Complete training on a single NVIDIA RTX 4090 takes approximately 22 hours, whereas the paper states that MaskBit requires 4.5 days on 32 A100 GPUs and that TiTok-S-128 reportedly uses 1600 GPU hours. The reported reduction is 40$\beta$8–150$\beta$9 in GPU hours.

In this setting, TVT denotes the transfer of a pretrained continuous decoder interface into a discrete-token regime, rather than transfer between two ELBO-optimized VAEs. The essential idea is nonetheless the same: the pretrained VAE supplies a stable latent target and a fixed decoder tolerance envelope, and only the new modules that mediate the latent representation are retrained.

4. Latent-resolution transfer for real-world image super-resolution

In real-world image super-resolution, TVT is used to transfer Stable Diffusion’s 8$\beta$0 downsampled VAE into a 4$\beta$1 one while preserving compatibility with a pretrained UNet (Yi et al., 27 Jul 2025). The motivating observation is explicit: SD 2.1-base compresses a $\beta$2 image to a $\beta$3 latent via 8$\beta$4 spatial downsampling, and this aggressive compression removes high-frequency content and small structures such as small text and fine textures.

The method is again two-stage. In Stage 1, a new 4$\beta$5 decoder $\beta$6 is trained from output features of the original 8$\beta$7 encoder $\beta$8:

$\beta$9

The objective is

$$\mathcal{L}(\theta,\phi; x)=\mathbb{E}_{q_{\theta}(z|x)}[\log p_{\phi}(x|z)]-\mathrm{KL}(q_{\theta}(z|x)\,\Vert\,p(z)),$$0

$$\mathcal{L}(\theta,\phi; x)=\mathbb{E}_{q_{\theta}(z|x)}[\log p_{\phi}(x|z)]-\mathrm{KL}(q_{\theta}(z|x)\,\Vert\,p(z)),$$1

with adaptive weighting

$$\mathcal{L}(\theta,\phi; x)=\mathbb{E}_{q_{\theta}(z|x)}[\log p_{\phi}(x|z)]-\mathrm{KL}(q_{\theta}(z|x)\,\Vert\,p(z)),$$2

In Stage 2, the decoder is fixed and a 4$$\mathcal{L}(\theta,\phi; x)=\mathbb{E}_{q_{\theta}(z|x)}[\log p_{\phi}(x|z)]-\mathrm{KL}(q_{\theta}(z|x)\,\Vert\,p(z)),$$3 encoder $$\mathcal{L}(\theta,\phi; x)=\mathbb{E}_{q_{\theta}(z|x)}[\log p_{\phi}(x|z)]-\mathrm{KL}(q_{\theta}(z|x)\,\Vert\,p(z)),$$4 is trained by

$$\mathcal{L}(\theta,\phi; x)=\mathbb{E}_{q_{\theta}(z|x)}[\log p_{\phi}(x|z)]-\mathrm{KL}(q_{\theta}(z|x)\,\Vert\,p(z)),$$5

with

$$\mathcal{L}(\theta,\phi; x)=\mathbb{E}_{q_{\theta}(z|x)}[\log p_{\phi}(x|z)]-\mathrm{KL}(q_{\theta}(z|x)\,\Vert\,p(z)),$$6

The paper states that no explicit distribution matching or KL/feature alignment term is introduced between the 8$$\mathcal{L}(\theta,\phi; x)=\mathbb{E}_{q_{\theta}(z|x)}[\log p_{\phi}(x|z)]-\mathrm{KL}(q_{\theta}(z|x)\,\Vert\,p(z)),$$7 and 4$$\mathcal{L}(\theta,\phi; x)=\mathbb{E}_{q_{\theta}(z|x)}[\log p_{\phi}(x|z)]-\mathrm{KL}(q_{\theta}(z|x)\,\Vert\,p(z)),$$8 latents; alignment is achieved implicitly through the two-stage transfer.

The proposed compact VAE-D4 uses 3 stages with channels 128/256/256 and removes the middle stage present in the original SD VAE-D8. On Urban100 reconstruction, the reported numbers are: VAE-D8, 1.8 T FLOPs, 83.7 M parameters, PSNR 25.29, SSIM 0.7656, LPIPS 0.0659; VAE-D4, 1.1 T FLOPs, 15.2 M parameters, PSNR 32.35, SSIM 0.9290, LPIPS 0.0230. The compute-efficient UNet replicates the pretrained UNet’s first and last layers to construct a high-resolution branch, downsamples 128$$\mathcal{L}(\theta,\phi; x)=\mathbb{E}_{q_{\theta}(z|x)}[\log p_{\phi}(x|z)]-\mathrm{KL}(q_{\theta}(z|x)\,\Vert\,p(z)),$$9128 latents to 64$\beta$0064 internally, and reports 0.73 T FLOPs versus 1.35 T FLOPs for processing 128$\beta$01128 inputs directly.

The full Real-ISR system uses SD 2.1-base, AdamW, learning rate $\beta$02, batch size 16, LoRA rank 4, and $\beta$03 iterations on 4 A100 GPUs. VAE-D4 encoder and decoder training use batch size 256, $\beta$04 iterations, and 8 A100 GPUs. End-to-end TVT is reported at 1.97 T FLOPs and 1.72 B parameters in one step, compared with OSEDiff at 2.27 T FLOPs and 1.77 B parameters, and S3Diff at 2.62 T FLOPs and 1.33 B parameters.

Quantitative results are reported on DIV2K-val, RealSR, DRealSR, and RealCE. On RealSR, TVT reports PSNR 25.81, SSIM 0.7596, LPIPS 0.2587, CLIPIQA 69.89, MUSIQ 3.770, and Q-Align 0.6829, all marked as best. On DRealSR, TVT reports PSNR 28.27, SSIM 0.7899, FSIM 0.8534, LPIPS 0.2900, CLIPIQA 65.56, and MUSIQ 3.641, again marked as best. On DIV2K-val, TVT reports PSNR 24.23, SSIM 0.6292, FSIM 0.8410, LPIPS 0.2773, CLIPIQA 68.67, and MUSIQ 3.920, with PSNR, SSIM, and FSIM marked as best and LPIPS, CLIPIQA, and MUSIQ as second best. For scene text image super-resolution on RealCE, TVT reports PSNR 19.69, SSIM 0.7030, LPIPS 0.1644, ACC 0.3096, and NED 0.6621, with SSIM, LPIPS, ACC, and NED marked as best.

This version of TVT is therefore not a decoder-only transfer in the sense of aggregated $\beta$05-VAE training. It is a latent-resolution adaptation procedure whose principal constraint is compatibility with a pretrained diffusion UNet.

5. Compatible video VAEs and transfer from image latent spaces

A further development extends TVT from image VAEs to video VAEs by explicitly regularizing a 3D latent space to remain compatible with a pretrained image VAE (Zhao et al., 2024). CV-VAE starts from the SD image VAE and inflates it to 3D: 2D convolutions are replaced by 3D convolutions, 3D kernels are initialized by copying 2D kernels into the temporal center and zero-initializing the remaining temporal positions, and reflection padding in time allows the model to process single images. Spatial compression matches SD’s VAE, latent channels remain $\beta$06, and temporal compression is set to $\beta$07.

The paper trains the video autoencoder with perceptual reconstruction, KL, adversarial loss, and latent compatibility regularization:

$\beta$08

The key TVT term is decoder-based regularization:

$\beta$09

with optional encoder-based regularization

$\beta$10

The total objective is

$\beta$11

Decoder-only regularization is reported as best, and among mapping functions $\beta$12, Random performs best.

The architecture uses an efficient 2D+3D design in which each ResBlock retains half the convolutions as 2D and half as 3D, reducing parameters and FLOPs by about 30% relative to all-3D with negligible reconstruction drop. Temporal tiling with one-frame overlap supports long videos, and the decoder upsamples time by a factor of 4 so that decoding $\beta$13 latent frames yields $\beta$14 output frames.

Training uses AdamW with learning rate $\beta$15 and cosine decay, float32 precision, 16 A100 GPUs, and 200K steps. Batches are mixed from four dataloaders: 256$\beta$16256 images with batch 8 and sampling ratio 40%, 512$\beta$17512 images with batch 2 and sampling ratio 10%, 9$\beta$18256$\beta$19256 videos with batch 1 and sampling ratio 25%, and 17$\beta$20192$\beta$21192 videos with batch 1 and sampling ratio 25%. Evaluation is reported on COCO-Val and WebVid-Val.

For reconstruction, VAE-SD2.1 reports COCO 26.6/0.773/0.127 and WebVid 28.9/0.810/0.145 in PSNR/SSIM/LPIPS; VAE-OSP reports 27.0/0.791/0.142 and 26.7/0.781/0.166; CV-VAE (2D+3D) reports 27.6/0.805/0.136 and 28.5/0.817/0.143; and CV-VAE (all-3D) reports 27.7/0.805/0.135 and 28.6/0.819/0.145. As a drop-in replacement in SD2.1 without fine-tuning, FID changes from 57.3 to 57.6, CLIP from 0.312 to 0.311, and PIC from 0.354 to 0.360. In video generation, SVD baseline generates 25 frames, whereas SVD with CV-VAE-V decodes 97 frames; after fine-tuning only the last U-Net layer, UCF101 reports FVD/KVD/PIC = 681/13.1/0.858 and MSR-VTT reports 295/2.26/0.734.

This formulation makes explicit a central TVT theme: the transferred object can be a latent-space compatibility relation rather than a specific decoder or encoder. The pretrained image VAE defines the target latent semantics, and the video VAE is regularized to inhabit that space.

6. Progressive growth, adjacent transfer-style VAEs, and open issues

An earlier realization of TVT appears in progressive VAE training on highly sparse and imbalanced routing data (Utyamishev et al., 2019). The procedure begins with a low-resolution core model trained on 8$\beta$228 data. In the paper’s architecture, the core corresponds to layers 7–15 of the full model plus an 8%%%%123$\beta$124%%%%1 output conv layer. Once trained, all core layers are frozen. New outer encoder layers consisting of convolution and downsampling, together with symmetric decoder layers consisting of upsampling and deconvolution or transpose-conv, are attached for the next resolution. Only the newly attached layers are trained at each stage, and the process is repeated through 16$\beta$2516, 32$\beta$2632, and 64$\beta$2764.

The work emphasizes a custom sparsity- and connectivity-aware reconstruction loss,

$\beta$28

where

$\beta$29

$\beta$30

with $\beta$31 and $\beta$32. The paper also uses “preliminary route-free” training to avoid collapse to all-zeros outputs. On ISPD’98 ibm02.modified at 64$\beta$3364 resolution, the reported routability is 96% for the progressive VAE, compared with 0% for a simple VAE and 2.7% for a GAN trained for approximately 10 hours on GTX1080.

A related, though differently named, two-stage transfer-style conditional VAE is VT-STOWER for non-parallel text style transfer (Xu et al., 2021). It uses a Transformer VAE with external style embeddings in Stage I and a pivot-words masking enhancement phase in Stage II, during which style embeddings are frozen. The style-controlled latent is

$\beta$34

and Stage II fine-tunes the VAE with masked inputs rather than introducing adversarial or cycle-consistency losses. This suggests that transfer-style staging extends beyond decoder transfer or latent compatibility and can also be applied to conditional generation with explicit control variables.

The surveyed literature also delineates several limitations. In aggregated $\beta$35-VAE TVT, no formal convergence proof for the decoupled decoder is provided, overlapping latent clusters can hinder class-specific manipulation, and broad posteriors with large latent dimensionality make decoder training harder (Li et al., 2022). In ReVQ, extremely high compression such as $\beta$36 tokens is not achieved, frozen encoder–decoder pairs remain sensitive to domain shift, and the paper attributes degradation at extreme compression to rectifier capacity and the intrinsic bound of the frozen VAE architecture (Zhang et al., 14 Jul 2025). In Real-ISR TVT, severe degradations remain challenging and the method can still be heavy for resource-constrained devices (Yi et al., 27 Jul 2025). In CV-VAE, compatibility requires fixing the latent channel dimension to $\beta$37, and over-regularization can underfit temporal dynamics (Zhao et al., 2024).

A common misconception is that TVT must remain within a single ELBO framework. The record is more heterogeneous. Some instantiations preserve a KL-regularized latent Gaussian objective; others replace end-to-end ELBO optimization with latent-space MSE, LPIPS, adversarial reconstruction, or stage-wise reconstruction-only training. Another misconception is that TVT denotes only one transfer direction. The documented cases include encoder-to-decoder transfer, image-VAE-to-video-VAE transfer, low-resolution-core-to-high-resolution-model transfer, and continuous-VAE-to-discrete-tokenizer transfer. The literature therefore suggests that TVT is best understood as a design pattern for structured, staged reuse of VAE representations, rather than as a single standardized algorithm.