VaultGemma 1B: Open DP Transformer LLM

• VaultGemma 1B is a decoder-only transformer language model with 1B parameters, designed with strict differential privacy to protect sensitive data.
• It employs efficient DP-SGD training with full global attention and a reduced 1024-token context, enabling scalable per-example gradient clipping.
• Benchmark results show its performance approaches that of non-private models while eliminating memorization risks and ensuring open research access.

VaultGemma 1B is a 1-billion-parameter, decoder-only transformer LLM in the Gemma family, engineered from scratch with strict differential privacy (DP) guarantees. It is the first open-weight LLM at this scale to demonstrate that state-of-the-art DP training can yield competitive utility, closely matching or exceeding the performance of non-private models from earlier generations. VaultGemma 1B was pretrained on the same 13 trillion-token, English-centric corpus as Gemma 2, with comprehensive filtering and redaction to reduce memorization and privacy risks. The model is released with public weights, training code, privacy accounting scripts, and detailed recipes for privacy-preserving downstream use, under a rigorous $$(\varepsilon \le 2.0,\, \delta \le 1.1 \times 10^{-10})$$ guarantee (Sinha et al., 15 Oct 2025).

1. Architectural Design and Model Specification

VaultGemma 1B adopts a classic decoder-only transformer architecture, with key modifications to facilitate efficient and effective DP training. The architecture is summarized as follows:

• Parameter count: ≈1 billion trainable parameters
• Hidden dimension ($d_\text{model}$): 1152
• Number of layers: 26
• Feed-forward dimension (GeGLU): 13,824
• Attention heads: 4 multi-query heads (1 key-value head), head size 256
• Pre-norm: RMSNorm before each sub-layer, no post-norm
• Attention: Full global attention in all layers, context length 1024 tokens
• Vocabulary: 256,128 (trained via SentencePiece; byte-level, split-digit, whitespace-preserving)
• Embedding tying: Shared input/output embeddings

Two architectural features are particularly motivated by DP: (a) the reduced maximum context length (1024 tokens) lowers the compute cost per example and enables efficient, large-batch training; (b) full global attention, as opposed to windowed or blockwise attention, simplifies the DP mechanism by enabling uniform per-example gradient clipping across all layers.

2. Pretraining Data, Tokenization, and Batch Construction

VaultGemma 1B was pretrained on the identical 13 trillion-token dataset as Gemma 2 27B, consisting predominantly of English-language data with the following approximate composition:

Corpus Component	Proportion (%)
Common Crawl–derived web text	60
Culled books and Wikipedia	15
Public source code repositories	15
Scientific/news articles	10

Data were filtered in multiple stages: unsafe/toxic content removal, downstream evaluation set decontamination, and PII redaction. Tokenization uses a unified SentencePiece model with a 256K vocabulary, including byte-level, split-digit, and whitespace features.

Each input sequence for pretraining ("packed" 1024-token sequence) may concatenate multiple documents or split long documents across examples. To balance sample diversity and sampling variance, source documents are drawn with weights proportional to estimated quality. A document may appear up to seven times per epoch, but most are repeated no more than three times.

3. Differential Privacy Guarantees and Mechanisms

VaultGemma 1B is trained end-to-end with a $$(\varepsilon \le 2.0, \delta \le 1.1 \times 10^{-10})$$ guarantee, where the "privacy unit" is a packed 1024-token sequence. Core elements of the DP-SGD mechanism include:

• Per-example gradient clipping: For each example $i$, compute $g_i(\theta)$ and clip to norm $C$,

$$\text{clip}(g_i, C) = g_i \cdot \min(1,\, C / \|g_i\|_2)$$

• Noise addition: Add Gaussian noise calibrated by clip norm $C$ and noise multiplier $\sigma$,

$$\tilde{g} = \frac{1}{L} \sum_{i=1}^L \text{clip}(g_i, C) + \mathcal{N}(0, \sigma^2 C^2 I)$$

with $d_\text{model}$0

• Subsampling: Truncated Poisson Subsampling with a fixed inclusion probability, followed by truncation/padding per batch to expected size $d_\text{model}$1
• Privacy accounting: Utilizes the ABLQ$d_\text{model}$2 (zero-out adjacency) method with a Poisson-Logarithmic Divergence (PLD) accountant, implemented in Google’s DP library, to track cumulative $d_\text{model}$3 over the course of 100,000 updates

The choice of full global attention and reduced sequence length is specifically designed to enable efficient and theoretically robust per-example clipping and private noise addition across large batches.

4. Training Regimen and Hyperparameter Configuration

VaultGemma 1B was trained on a 2 × 16 × 16 TPUv6e mesh (2048 chips), using 2048-way data parallelism.

Key hyperparameters:

• Global batch size: Expected 517,989 sequences per update (≈524,288 pre-truncation/padding)
• Gradient accumulation: 64 micro-batches per update, each sized at 4 examples per core
• Optimizer: DP-SGD (Adam variant with DP noise injection)
• Noise multiplier: $d_\text{model}$4 (as above)
• Clip norm: $d_\text{model}$5 (Gemma‐standard, set for DP tuning)
• Learning rate: Chosen by modeling the final loss as a quadratic function of the rate, searched over seven candidate values
• Total updates: 100,000, consistent with the compute–noise trade-off from DP scaling laws (final loss within 1% of mathematical predictions)

5. Empirical Performance, Utility, and Memorization

VaultGemma 1B performance was benchmarked against its non-private sibling Gemma3 1B (PT) and GPT-2 1.5B across standard zero- and few-shot tasks:

Benchmark	VaultGemma 1B	Gemma3 1B (PT)	GPT-2 1.5B
ARC-Challenge	26.45	38.31	39.78
ARC-Easy	51.78	71.34	51.10
HellaSwag	39.09	61.04	47.91
PIQA	68.00	77.37	70.51
SIQA	46.16	49.28	—
BoolQ	62.04	68.75	61.80
TriviaQA (5-shot)	11.24	39.75	6.00

The DP model’s scores are typically 10–30 points below its non-private version but approach or surpass the performance of GPT-2 1.5B on several tasks. This illustrates that differentially private training at billion-parameter scale can achieve utility clearly competitive with non-private architectures from earlier periods.

Empirical memorization tests—using 1 million subsampled training examples with prefix–suffix extraction—detected no exact or approximate memorization in VaultGemma 1B. In contrast, all non-DP Gemma variants showed measurable memorization, underscoring the privacy effectiveness of the approach.

6. Deployment, Release, and Recommendations

VaultGemma 1B’s 1B-parameter footprint and tied embeddings yield a manageable 4–5 GB memory usage in FP32. The 1024-token fixed context window and universal attention design allow for straightforward inference, including efficient batching and quantized deployment (INT8) on standard GPUs and TPUs.

The full model weights, tokenizer, training code, and privacy-accounting scripts are released under an open-source license, available through the Google Research VaultGemma repository. Practitioners are instructed to observe the $d_\text{model}$6 privacy guarantee during any fine-tuning or deployment, and to apply DP-SGD with the same configuration to any further training on private data to preserve end-to-end privacy.

7. Broader Significance and Future Directions

VaultGemma 1B provides a practical instantiation of competitive open-weight LLMs trained under rigorous DP constraints. Its development demonstrates the feasibility of DP at billion-scale, introduces scalable and reproducible DP training procedures, and delivers implementation recipes for the broader community. The model serves as both a high-utility research artifact and a foundation for building larger, even more effective private models, as suggested by the associated DP scaling laws and public implementation practices (Sinha et al., 15 Oct 2025).