OneVAE: Unified Video VAE Framework

• The paper introduces a unified video VAE that models continuous and discrete latent spaces using a joint optimization strategy.
• It leverages Feature-Space Quantization and progressive transfer from pre-trained continuous VAE priors to accelerate convergence and enhance PSNR.
• Structural innovations like multi-token quantization and first-frame enhancement improve token capacity and reconstruction fidelity under high compression.

Searching arXiv for the specified OneVAE paper and closely related VAE context. OneVAE is a unified video variational autoencoder framework that models both continuous and discrete latent representations within a single architecture. It is motivated by a tension in video representation learning: discrete video tokens are desirable because they could align with text tokens to facilitate concise and unified multimodal LLMs, but prior discrete video VAEs suffered from unstable training, long training time, and degraded reconstruction quality relative to continuous VAEs. OneVAE addresses this by leveraging pre-trained continuous VAE priors, using Feature-Space Quantization (FSQ), and introducing a joint discrete-continuous optimization scheme together with structural modifications for multi-token quantization and first-frame enhancement (Zhou et al., 13 Aug 2025).

1. Problem formulation and design objective

The core problem addressed by OneVAE is the construction of a video VAE that preserves the training stability and reconstruction quality associated with continuous latent representations while also producing discrete tokens suitable for token-based multimodal modeling. In the formulation summarized for the paper, discrete tokens are attractive because they could align with text tokens, but this requires significant spatiotemporal compression compared to continuous video representation. Previous discrete video VAEs are described as exhibiting unstable training, long training time, and degraded reconstruction quality, whereas continuous VAEs are characterized as easier to train and superior in performance (Zhou et al., 13 Aug 2025).

OneVAE is therefore positioned not as a replacement of one paradigm by the other, but as a unification of the two. The framework is explicitly described as capable of modeling both continuous and discrete latent representations within a single architecture, and the method is named to reflect that connection. A central implication is that discrete video tokenization need not be pursued independently of continuous VAE priors; instead, the latter can be used to improve optimization and final reconstruction in the former.

2. Unified architecture and joint discrete-continuous optimization

The architecture consists of an encoder that maps video input into a latent space, followed by two training paths. The continuous path uses a standard VAE decoder and is trained with a continuous reconstruction loss. The discrete path passes latent features through FSQ for discretization and reconstructs with a discrete decoder. During training, the model toggles between the discrete and continuous reconstruction objectives: with probability $R_{dis}$, the model uses discrete reconstruction; otherwise it uses continuous reconstruction (Zhou et al., 13 Aug 2025).

The joint loss is given as

$$L = \begin{cases} L_d, & \text{if } r < R_{dis} \ L_c, & \text{otherwise} \end{cases}$$

where $L_d$ is the discrete path reconstruction loss, $L_c$ is the continuous path loss, and $r$ is a random number in $(0,1)$ (Zhou et al., 13 Aug 2025).

In the paper summary, this dual-path training is described as allowing mutual reinforcement between continuous and discrete paths, leading to improved training stability and higher reconstruction quality. The same summary states that the unified model can perform both continuous and discrete video reconstruction without compromising either quality. For the continuous VAE itself, the notation given is

$$E(x) = \mathcal{N}(\mu, \sigma^2), \quad z \sim E(x),$$

and

$z = E(x), \quad \hat{x} = D(z).$

This setup is significant because it redefines the relation between continuous and discrete VAEs as an optimization coupling rather than a strict architectural fork. A plausible implication is that the discrete path benefits from continual exposure to the geometry of the continuous latent space during training, rather than only through initialization.

3. Progressive transfer from a pre-trained continuous VAE and the role of FSQ

A principal contribution of OneVAE is progressive training with a pre-trained continuous prior. The method starts from a pre-trained continuous VAE, inserts a quantization layer based on FSQ, and then fine-tunes the network. This preserves learned priors and accelerates convergence. The abstract states that FSQ could effectively preserve pre-trained continuous VAE priors compared to other quantization methods, and that, by leveraging continuous VAE priors, the method converges several times faster than training from scratch and achieves superior performance at convergence (Zhou et al., 13 Aug 2025).

FSQ is contrasted with vector quantization. The summary describes VQ as looking up the nearest codebook vector,

$$q_i = \arg\min_{c_i \in \mathcal{C}} \|z_i - c_i\|_2,$$

whereas FSQ is defined as

$q_i = \text{Round}(\text{Bound}(z_i)).$

Unlike VQ, FSQ simply rounds each dimension of the bounded latent feature. This is presented as being closer to continuous encodings and therefore more compatible with the latent structure of a pre-trained continuous VAE (Zhou et al., 13 Aug 2025).

The empirical result reported for fixed quantization in the latent space of a continuous VAE is that discrete FSQ achieves PSNR 26.46 versus VQ 22.82. The same summary reports that training convergence is sped up by $$L = \begin{cases} L_d, & \text{if } r < R_{dis} \ L_c, & \text{otherwise} \end{cases}$$0 compared to standard FSQ training-from-scratch, and that progressive FSQ training improves final PSNR by about 1 dB over baselines (Zhou et al., 13 Aug 2025).

These claims situate FSQ not merely as a quantizer but as the specific mechanism that makes transfer from continuous to discrete latent structure operational. The paper’s argument is that preserving the prior matters as much as discretizing the representation.

4. Structural modifications: multi-token quantization and first-frame enhancement

OneVAE introduces two architectural modifications aimed at recovering reconstruction quality under strong compression. The first is a multi-token quantization mechanism. Instead of quantizing each latent spatial position with a single token, the feature vector at each position is decomposed into multiple sub-vectors along the channel dimension, and each sub-vector is quantized independently using shared codebooks. Collectively, multiple tokens per position encode richer detail with finer granularity, while the mechanism is described as increasing representational capacity without increasing the overall token count (Zhou et al., 13 Aug 2025).

The paper summary attributes nearly 1 dB improvement in PSNR to multi-token quantization for the same spatial compression ratio. A more specific result in the quantitative summary reports that multi-token quantization pushes PSNR from 30.67 to 31.80 (Zhou et al., 13 Aug 2025).

The second modification addresses causal video VAEs, where the first frame lacks temporal context and therefore reconstructs poorly under high compression. OneVAE strengthens first-frame reconstruction by assigning lower compression, i.e. more tokens, specifically to the first frame while keeping the total token budget fixed. Subsequent frames then benefit from using this higher-quality first frame as reference during decoding. The abstract states that this markedly improves the performance of $$L = \begin{cases} L_d, & \text{if } r < R_{dis} \ L_c, & \text{otherwise} \end{cases}$$1 discrete VAEs, and the detailed summary reports that first-frame enhancement raises PSNR from 27.40 to 28.11 (Zhou et al., 13 Aug 2025).

Taken together, these modifications show that the paper treats token budget allocation as structurally nonuniform rather than spatially and temporally homogeneous. This suggests that, in high-compression causal settings, reconstruction quality depends not only on global compression ratio but also on where capacity is allocated.

5. Quantitative results and reported benchmarks

The reported empirical evaluation emphasizes reconstruction fidelity, perceptual quality, and video realism. The summary states that the unified model outperforms discrete-only or continuous-only baselines in PSNR, SSIM, LPIPS, FVD on Panda70M (Zhou et al., 13 Aug 2025).

Selected results reported in the supplied summary are organized below.

Method	Compression / token type	Reported metrics
Cosmos Token	$$L = \begin{cases} L_d, & \text{if } r < R_{dis} \ L_c, & \text{otherwise} \end{cases}$$2, discrete	PSNR 26.80, SSIM 0.8531, FVD 350.87
OneVAE	$$L = \begin{cases} L_d, & \text{if } r < R_{dis} \ L_c, & \text{otherwise} \end{cases}$$3, discrete	PSNR 27.40, SSIM 0.8690, FVD 336.48
OneVAE-FE	$$L = \begin{cases} L_d, & \text{if } r < R_{dis} \ L_c, & \text{otherwise} \end{cases}$$4 (+FirstFrm), discrete	PSNR 28.11, SSIM 0.8772, FVD 213.04
Cosmos Token	$$L = \begin{cases} L_d, & \text{if } r < R_{dis} \ L_c, & \text{otherwise} \end{cases}$$5, discrete	PSNR 27.17, SSIM 0.8638, FVD 303.37
OneVAE-MT	$$L = \begin{cases} L_d, & \text{if } r < R_{dis} \ L_c, & \text{otherwise} \end{cases}$$6 (MultiToken), discrete	PSNR 31.80, SSIM 0.9399, FVD 78.35
CV-VAE	$$L = \begin{cases} L_d, & \text{if } r < R_{dis} \ L_c, & \text{otherwise} \end{cases}$$7, continuous	PSNR 31.12, SSIM 0.9142, FVD 100.72
OneVAE	$$L = \begin{cases} L_d, & \text{if } r < R_{dis} \ L_c, & \text{otherwise} \end{cases}$$8, continuous	PSNR 32.48, SSIM 0.9335, FVD 59.16

The summary also reports a decoder expansion result for a $$L = \begin{cases} L_d, & \text{if } r < R_{dis} \ L_c, & \text{otherwise} \end{cases}$$9 discrete VAE: expanding the decoder gives PSNR 25.05 versus 24.86 baseline (Zhou et al., 13 Aug 2025).

These numbers are used in the paper summary to support two distinct claims. First, OneVAE improves discrete video VAE optimization and reconstruction relative to prior discrete baselines. Second, the unified formulation remains competitive on the continuous side as well, rather than sacrificing continuous performance to obtain discrete tokens.

6. Conceptual significance, scope, and distinction from similarly named VAE work

The principal conceptual claim of OneVAE is that discrete and continuous video VAEs can be trained as two coupled modes of one network. In the supplied summary, this is presented as the first time that competitive performance on both continuous and discrete representations is achieved within a single network (Zhou et al., 13 Aug 2025). The framework therefore occupies a specific position in video generative modeling: it is neither a purely continuous compression model nor a purely tokenization-oriented discrete model, but a bridge between the two.

A common misconception would be to interpret OneVAE as simply a discrete VAE initialized from a continuous model. The paper summary indicates a broader scope. In addition to progressive transfer, the method includes joint discrete-continuous optimization, multi-token quantization, and first-frame enhancement. The intended result is a unified model rather than a one-way conversion pipeline.

Another source of confusion concerns nomenclature. The separate paper "DOT-VAE: Disentangling One Factor at a Time" addresses unsupervised disentanglement by augmenting the latent space with a disentangled space $L_d$0, an entangled latent $L_d$1, a Wake-Sleep-inspired two-step algorithm, and one-factor-at-a-time interventions (Patil et al., 2022). That objective is distinct from OneVAE’s video-tokenization and reconstruction setting. DOT-VAE focuses on disentangling interpretable independent factors of variation, whereas OneVAE focuses on unifying continuous and discrete video latent representations. The shared lexical element “One” therefore does not indicate a shared technical agenda.

Within the scope stated in the paper summary, OneVAE’s significance lies in reframing discrete token learning as a problem of preserving and exploiting continuous VAE priors. This suggests a broader methodological lesson: in video representation learning, optimization geometry inherited from continuous models may be as important as the eventual discreteness of the latent code.