---
title: 'VAMPIRE: Efficient Variational Pretraining'
url: https://www.emergentmind.com/topics/vampire-model
type: topic
---

# VAMPIRE: Efficient Variational Pretraining

VAMPIRE (VAriational Methods for Pretraining In Resource-limited Environments) is a lightweight, semi-supervised text classification framework that pretrains a variational autoencoder (VAE) on in-domain, unlabeled data to produce document features for downstream classification under data- and resource-constrained regimes. Unlike computationally intensive neural language models, VAMPIRE is designed for rapid, effective deployment where labeled data and compute are scarce, achieving strong empirical results on multiple benchmarks [1906.02242].

## 1. Unigram Document VAE: Generative Model, Inference, and ELBO

VAMPIRE’s core is a bag-of-words VAE formulated over document count vectors. Each document $x$ is represented as $c \in \mathbb{N}^V$, with $V$ the vocabulary size. The generative process samples $z \in \mathbb{R}^K$ from a spherical Gaussian prior $p(z) = \mathcal{N}(z; 0, I)$, then applies a softmax to yield topic proportions $\theta = \mathrm{softmax}(z) \in \Delta^K$. Word probabilities $\eta$ are computed via $\eta = \mathrm{softmax}(b + B\theta) \in \Delta^V$, and the likelihood factorizes as:
\[
\log p(x|z) = \sum_{j=1}^V c_j \log \eta_j
\]
The variational posterior $q(z|x)$ is a diagonal Gaussian with parameters output by a multi-layer perceptron (MLP) encoder on $c$:
\[
h = \mathrm{MLP}(c),\quad \mu = W_\mu h + b_\mu,\quad \log\sigma = W_\sigma h + b_\sigma,\quad \sigma = \exp(\log\sigma)
\]
A sample $z = \mu + \sigma\odot\epsilon$ is drawn with $\epsilon\sim\mathcal{N}(0,I)$. The variational lower bound (ELBO) is:
\[
\mathcal{L}(x) = \mathbb{E}_{q(z|x)} [\log p(x|z)] - \mathrm{KL}(q(z|x) \| p(z))
\]
In practice, the expectation is approximated via a single sample.

## 2. VAE Feature Extraction for Downstream Classification

After pretraining, the VAE is frozen. Features for each document are extracted from both the encoder MLP’s intermediate activations $h^{(1)}, ..., h^{(n)}$ and the final topic vector $\theta$. The document embedding is a learnable weighted sum:
\[
r = \lambda_0 \theta + \sum_{k=1}^n \lambda_k h^{(k)}
\]
where $\{\lambda_0,\ldots,\lambda_n\}$ are softmax-normalized scalars. This concatenation of information captures both topic-level and generic feature structure for classification.

## 3. Model Architecture and Training Protocols

Key hyperparameters in VAMPIRE include: $V=30{,}000$ (vocabulary), $K\approx80$–$125$ (latent dim., via random search), encoder depth $n=2$–$3$, hidden layers $\approx80$–$128$ units (tanh or ReLU activations). Decoder activations are linear prior to softmax.

Regularization methods:
- $z$-dropout: dropout rate of $0.4$–$0.5$ on $z$
- Batch normalization on reconstruction logits
- KL-annealing: KL term scheduled from $0$ to $1$ over $\sim$1,000 updates (linear/sigmoid)

Optimization uses Adam (learning rate sampled in $[1\cdot10^{-4},1\cdot10^{-2}]$, typically $2\cdot10^{-4}$), batch size $64$, and up to $50$ epochs. Early stopping tracks topic coherence (NPMI) on held-out data (patience = $5$ epochs), not marginal likelihood.

## 4. Classifier Integration and Semi-supervised Learning Regime

The downstream classifier is a sequence-to-vector encoder $f_{\text{s2v}}(x)$ implemented as a Deep Averaging Network (DAN). DAN embeds tokens (random/GloVe/contextual), averages them, and applies a 1-layer MLP with dropout.

The final classifier input is the concatenation $[r;\ \mathrm{DAN}(x)]$, passed through the DAN’s MLP for prediction. Loss is standard cross-entropy, supervised only on labeled examples $\mathcal{D}_L$. The VAE remains frozen during classifier training. For semi-supervised learning, VAMPIRE leverages large in-domain unlabeled corpora $\mathcal{D}_U$ for VAE pretraining, but does not employ iterative pseudo-labeling as in self-training baselines.

## 5. Empirical Performance and Comparative Results

VAMPIRE demonstrates strong performance on low-resource classification tasks. Test accuracies with only $200$ labeled examples are:

| Dataset       | VAMPIRE         | DAN (sup.)   | DAN+self-training |
|---------------|-----------------|--------------|-------------------|
| AG News       | $82.2\%$ (±2.0) | $68.5\%$     | $73.8\%$          |
| Yahoo         | $74.1\%$ (±0.8) | $67.7\%$     | $68.5\%$          |
| IMDB          | $83.9\%$ (±0.6) | $68.8\%$     | $77.3\%$          |
| Toxic Tweets  | $59.9\%$ (±0.9) | $54.5\%$     | $57.5\%$          |

With $500$–$2{,}500$ labels, VAMPIRE continues to outperform classical baselines and approaches the accuracy of large, fine-tuned language models such as ELMo/BERT on in-domain data. Model size is approximately $3.8$M parameters, training in $7$ minutes on a single GPU, compared to $159$M parameters and $>12$ hours for a Transformer LM on the same corpus.

## 6. Design Rationale and Distinguishing Characteristics

VAMPIRE’s advantages derive from:
- Spherical Gaussian latent prior with topic-based decoding via a softmax transformation
- Bag-of-words document encoding and unigram reconstruction, yielding interpretable, topic-like document representations
- Extraction of both high-level ($\theta$) and hidden-layer ($h^{(k)}$) encoder states as transferrable document features
- KL-annealing and NPMI-based early stopping for robust pretraining without overfitting
- Semi-supervised protocol that avoids the complexity of iterative pseudo-labeling

This design enables significant gains in accuracy and computational efficiency under resource constraints [1906.02242].

## 7. Limitations and Future Directions

VAMPIRE is limited to bag-of-words document representations, and thus does not encode word order or syntactic phenomena directly. Performance degrades when sufficient labeled data and compute permit pretraining and fine-tuning of large contextual models. Extension to richer document architectures and adaptation to languages with less clear bag-of-words semantics remain plausible avenues for future research. The VAMPIRE codebase is made available by the original authors for further study and experimentation [1906.02242].

Source: https://www.emergentmind.com/topics/vampire-model