RVAE-ST: Recurrent VAE with Subsequent Training

• The paper introduces RVAE-ST, a sequential generative model that integrates LSTM/GRU-based recurrent architectures into the VAE framework with a curriculum learning strategy.
• The model employs a recurrent encoder-decoder structure with gradual sequence length increases and adversarial regularization to improve ELBO and latent space matching.
• Empirical results show significant gains in tasks like anomaly detection, speech enhancement, and time series forecasting, highlighting its parameter efficiency and robust performance.

Recurrent Variational Autoencoder with Subsequent Training (RVAE-ST) is a class of sequential generative models that integrate recurrent neural architectures—most commonly LSTMs or GRUs—within the Variational Autoencoder (VAE) framework. The distinguishing feature is an adapted training regimen, termed "subsequent training" or curriculum learning, in which model sequence length or domain specificity is gradually increased or transferred. RVAE-ST models offer strong inductive bias for temporal data, parameter efficiency for long sequences, and robust generative or discriminative performance when complemented by principled subsequent training or domain adaptation strategies (Fulek et al., 8 May 2025, Kim et al., 2021, Huang et al., 2021, Leglaive et al., 2019).

1. Model Structure and Formulation

RVAE-ST models are defined by a recurrent encoder–decoder VAE backbone, adapted to sequential contexts:

• Encoder: Receives a time series $x_{1:T}$ and processes it via a multi-layer recurrent neural network (RNN) (typically LSTM or GRU). The final hidden state(s) are mapped to the parameters $(\mu, \log \sigma)$ of a variational posterior $$q_\phi(z | x_{1:T}) = \mathcal{N}(z; \mu, \operatorname{diag}\sigma^2)$$, yielding a global latent embedding or, in some variants, a sequence of per-time-step latents (Fulek et al., 8 May 2025, Huang et al., 2021).
• Decoder: Reconstructs the observed sequence from sampled latent(s) $z$. In the vector-to-sequence regime, $z$ is repeated at each time step and combined with a deep RNN decoder and a time-distributed output layer, with weights shared across time ($W \in \mathbb{R}^{H \times c}$), enforcing exact weight-tying and approximate time-shift equivariance.
• Generative Model: For global-latent models:

$$p(x_{1:T}, z) = p(z) \prod_{t=1}^T p(x_t | h_t(z)),$$

where $h_t$ is the RNN state at $t$ given $z$. For structured-latent RVAE variants, the emission and recurrent update are performed for each $z_t$ (Huang et al., 2021).

• Inference Model: $q_\phi(z | x_{1:T})$ (vector) or $q_\phi(z_{1:T} | x_{1:T})$ (sequence), possibly bidirectional, encodes the entire sequence.

2. Subsequent Training Schemes

Subsequent Training ("ST", Editor's term), a critical component of RVAE-ST, includes curriculum-based training, domain adaptation, or adversarial fine-tuning:

• Curriculum Learning for Sequence Length: Begin by training the model on short subsequences, gradually increasing the subsequence length until the target $L_\text{max}$ is reached. This addresses convergence difficulties and optimizes for very long-range dependencies (Fulek et al., 8 May 2025). The process can be formalized as:

  L = L_0 while L <= L_max: train_on_sequences_of_length(L) L += ΔL

  Empirically, this yields significant ELBO gains compared to direct long-sequence training.
• Transfer (Domain Adaptation) Training: For semi-supervised tasks, alternate optimization steps over labeled source data and (labeled or unlabeled) target data. In unsupervised target settings, pseudo-normal samples are iteratively mined based on low reconstruction error for denoising or anomaly detection (Kim et al., 2021).
• Adversarial Training: Embark on a second phase that regularizes the aggregate posterior to match the prior via a discriminator (e.g., WGAN objective), enhancing generative sharpness and latent utilization (Huang et al., 2021).

3. Loss Functions and Optimization

The cornerstone objective is the Evidence Lower Bound (ELBO):

$$\mathcal{L}(x_{1:T}) = \mathbb{E}_{q(z|x_{1:T})}[\log p(x_{1:T}|z)] - D_\mathrm{KL}(q(z|x_{1:T}) \| p(z)).$$

This decomposes to a reconstruction term (mean-squared error, binary cross-entropy, or data-specific likelihood) and a KL divergence regularizer. Typical loss scaling enforces consistency with data log-likelihood (e.g., $$\alpha \cdot \mathrm{SSE} + \beta \cdot \mathrm{KL}$$ with $\alpha \sim 500/T, \beta \sim 0.1$ for regression scenarios) (Fulek et al., 8 May 2025).

For models employing adversarial regularization, an additional discriminator-based loss is introduced:

$$L_\mathrm{dis} = \mathbb{E}_{z \sim p_\theta} [D_\psi(z)] - \mathbb{E}_{z \sim q_\phi} [D_\psi(z)],$$

with RVAE parameters updated to minimize the reverse, $-L_\mathrm{dis}$. Optimization is performed using Adam or RMSProp with typical learning rates $10^{-3}$ ($\mathrm{ELBO}$ steps) and parameter clipping for the discriminator (Huang et al., 2021).

4. Empirical Performance and Benchmarking

RVAE-ST models have been systematically evaluated across unsupervised generation, sequence modeling, anomaly detection, and speech enhancement tasks:

• Synthetic and Real Time-Series: On stationary and quasi-periodic data (e.g., Electric Motor, ECG, Sine), RVAE-ST achieves state-of-the-art generative metrics, including lowest Contextual Fréchet Distance (Context-FID) and highest average ELBO. On more irregular datasets (ETTm2, MetroPT3), RVAE-ST remains highly competitive, typically ranking among the top two models compared to GANs, diffusion models, and transformers (Fulek et al., 8 May 2025).
• Anomaly Detection (Botnet Traffic): RVAE-ST transfer learning raises detection true positive rates from 0.683 (no transfer) to 0.918 (with label) and 0.899 (unsupervised), with only minor increases in false positive rates, demonstrating its efficacy as a transductive learner for domain adaptation (Kim et al., 2021).
• Sequence Enhancement (Speech): In speech denoising, RNN-based RVAE outperforms frame-independent VAEs due to modeling of temporal dependencies, and fine-tuning at test-time using a variational EM procedure yields further improvements in SI-SDR and ESTOI (Leglaive et al., 2019).
• Latent Space Matching: The adversarial RVAE-ST closes the gap between reconstruction loss and ELBO and reaches sharper aggregate posteriors (Huang et al., 2021).

5. Architectural and Implementation Features

• Encoder/Decoder: Deep RNN stacks (e.g., 4 LSTM layers, $H=256$; or 2-layer Bi-GRU $H=512$ per direction) with fixed-size global latent ($d=20$–$100$) for vector-to-sequence models.
• Output Layer: Time-distributed linear mappings applied identically at each step.
• Data Preprocessing: Min–max scaling or per-speaker normalization to standardized ranges.
• Batching and Training: Batch sizes adapted to fit memory constraints for long sequences (batch 32–64 for $T=1000$), with patience-based early stopping for both curriculum and standard training (Fulek et al., 8 May 2025).

6. Theoretical Properties and Inductive Bias

RVAE-ST inherits two crucial inductive biases:

1. Time-Shift Equivariance: The strict sharing of transition and output weights across time steps, combined with global-latent repetition, ensures that for long enough sequences, the model becomes approximately equivariant under index shifts $x_t \mapsto x_{t+1}$, favoring stationary signal reconstruction (Fulek et al., 8 May 2025).
2. Parameter Efficiency: RVAE-ST architectures maintain a parameter count independent of sequence length, a notable advantage over transformer or convolution-dominated models for ultra-long time series.

The subsequent training paradigm demonstrably sharpens ELBO bounds (statistically significant gains in all benchmarks) and strengthens generalization under both domain shift and non-stationarity.

7. Variants and Extensions

While canonical RVAE-ST employs vector-to-sequence latent structure and LSTM/GRU backbones, several notable variants arise:

• Per-Time-Step Latents: Introduction of $z_t$ (Huang et al., 2021, Leglaive et al., 2019).
• Adversarial Regularization: WGAN-style aggregate posterior matching (Huang et al., 2021).
• Unsupervised Transfer: Pseudo-normal mining with iterative reconstruction error thresholds (Kim et al., 2021).
• Fine-Tuned Decoders: Test-time variational EM cycles to adapt to target domain corruptions (Leglaive et al., 2019).

Standard extensions (hierarchical latent variables, more expressive likelihoods, alternative bounds such as IWAE or FIVO, and stronger discriminators) have been suggested.

---

References

• "Generative Models for Long Time Series: Approximately Equivariant Recurrent Network Structures for an Adjusted Training Scheme" (Fulek et al., 8 May 2025)
• "Improving Botnet Detection with Recurrent Neural Network and Transfer Learning" (Kim et al., 2021)
• "Regularized Sequential Latent Variable Models with Adversarial Neural Networks" (Huang et al., 2021)
• "A Recurrent Variational Autoencoder for Speech Enhancement" (Leglaive et al., 2019)