EPSVec: Private Synthetic Text Generation

• EPSVec is a method that steers a frozen language model using sanitized dataset vectors to capture the gap between private and public distributions.
• The method privatizes a one-time, compact dataset-level signal to enable efficient text synthesis without incurring additional privacy cost per generated token.
• EPSVec overcomes challenges like data efficiency, compute cost, and privacy-budget coupling by employing paired mean differences and fixed-shot prompting.

Searching arXiv for the specified paper to ground the article. EPSVec is a method for differentially private synthetic text generation that steers a frozen LLM using sanitized dataset vectors, defined as directions in activation space that capture the distributional gap between a private corpus and a public or synthetic prior. The method is introduced in "EPSVec: Efficient and Private Synthetic Data Generation via Dataset Vectors" (Banayeeanzade et al., 31 Jan 2026). Its central design is to privatize once a compact dataset-level signal and then perform standard decoding as pure post-processing, thereby decoupling the privacy budget from the number and length of generated samples. The reported motivation is to address three bottlenecks in prior private text generation methods: data efficiency, compute cost, and privacy-budget coupling to token generation.

1. Problem formulation and design rationale

Private synthetic data generation for text is presented as being constrained by three practical bottlenecks: data efficiency, compute cost, and privacy-budget coupling to the number of generated tokens (Banayeeanzade et al., 31 Jan 2026). Training-based methods such as DP-SGD fine-tuning or DP LoRA are described as requiring large private corpora, careful hyperparameter tuning under DP noise, and substantial GPU hours. Inference-time methods such as PATE-style aggregation of next-token distributions and DP private prediction are described as needing large private batches at every decoding step, which makes long-form generation and large-scale sampling expensive and tightly couples privacy cost to the number of tokens generated.

EPSVec addresses these shortcomings by privatizing once a compact dataset-level signal—dataset vectors—that captures the distributional gap between the target private corpus and a public or synthetic prior in the LLM’s activation space. After the one-time release of sanitized vectors, all subsequent decoding is pure post-processing of DP outputs and therefore incurs no additional privacy cost. The reported consequences are strong compute efficiency, because generation runs at the cost of standard decoding with no per-token DP aggregation; data efficiency, because vectors are estimated from modest private sample counts such as 500 per class; fidelity, because steering nudges the model toward subtle dataset-level properties such as style, subtopic mixtures, and lexical distributions; and privacy-budget decoupling, because arbitrarily many samples and arbitrary sequence lengths can be generated at zero additional privacy cost.

The paper further states that fidelity and diversity are improved by pairing dataset-vector steering with fixed-shot prompting in pretrained base models, selected via a small DP budget. Pretrained models are reported to have broader stylistic support and diversity than instruction-tuned variants, but to need stable scaffolding; fixed-shots are used to provide that scaffold while preserving privacy. This suggests that EPSVec is not only a privacy mechanism but also a representation-space control method for aligning a pretrained generator with a private dataset distribution.

2. Representation-space formulation

The frozen LLM is denoted by $f$, with $L$ transformer layers and hidden dimension $d$ (Banayeeanzade et al., 31 Jan 2026). For an input text $x$ with $T(x)$ tokens, token-level hidden states at layer $\ell$ are denoted by $a_{\ell,t}(x) \in \mathbb{R}^d$, with $t \in \{1,\dots,T(x)\}$. EPSVec uses mean-pooled sequence representations per layer:

$$h_{\ell}(x) = (1 / T(x)) \sum_{t=1}^{T(x)} a_{\ell,t}(x) \in \mathbb{R}^d.$$

The private dataset for a target attribute or class $y$ is denoted by $L$0, and a public or synthetic reference dataset is denoted by $L$1. In the reported implementation, $L$2 is constructed by prompting $L$3 with a fixed-shot template $L$4, selected privately, to produce a negative synthetic set $L$5 of the same size as $L$6. Private samples are denoted by $L$7 and synthetic samples by $L$8, and they are paired by index $L$9.

A canonical dataset-vector definition is the difference between per-layer mean representations of the private and public or synthetic distributions:

$d$0

$d$1

EPSVec estimates this distributional gap through paired mean differences:

$d$2

$d$3

followed by unit-norm normalization:

$d$4

The paper states that this estimator cancels coarse shared attributes, because both sets are conditioned on $d$5 via prompting, and emphasizes subtler dataset-specific differences such as lexical distributions, formatting, and structure. EPSVec constructs vectors from multiple layers; the main experiments use $d$6 layers in Llama-3.1-8B, specifically layers 18–21. Sequence-level mean pooling is used rather than per-token vectors. Whitening and PCA are noted as not being used in the core pipeline.

This formulation places EPSVec in the family of activation-space steering methods, but with the steering signal estimated from private-versus-synthetic distributional contrasts rather than from direct prompting alone. A plausible implication is that the method is designed to encode properties that are difficult to specify explicitly in prompts while still remaining compatible with standard decoder-only generation.

3. Differential privacy mechanism and accounting

For each layer $d$7, the per-pair difference $d$8 is clipped to $d$9 norm $x$0:

$x$1

The clipped average over $x$2 private pairs is then perturbed with Gaussian noise:

$x$3

and normalized to unit norm (Banayeeanzade et al., 31 Jan 2026).

Under substitution adjacency, the $x$4-sensitivity of the clipped mean is bounded by $x$5. Applying the Gaussian mechanism yields the condition

$x$6

The theorem reported in the paper states that, for any $x$7 and $x$8, if the per-layer noise scale satisfies this inequality, then extracting $x$9 is $T(x)$0-DP. By basic composition over $T(x)$1 layers, releasing $T(x)$2 is $T(x)$3-DP.

A central property of EPSVec is privacy-budget decoupling by post-processing. Once $T(x)$4 are released under DP, all subsequent operations—scaling, combining, injection during decoding, rejection sampling, and filtering—are pure post-processing and add no privacy cost by DP invariance. The paper therefore states that sampling any number of sequences of any length incurs zero additional privacy loss.

If only a fraction $T(x)$5 of the private corpus is used to build vectors, with $T(x)$6, the overall pipeline, including fixed-shot selection, satisfies

$T(x)$7

Here $T(x)$8 is the budget for fixed-shot extraction and $T(x)$9 is the vector-construction budget. The paper attributes this to privacy amplification by subsampling and basic composition.

Fixed-shot selection is itself privatized. EPSVec builds a synthetic candidate pool $\ell$0 by zero-shot prompting the pretrained model, embeds both private and candidate texts via Qwen-3-Embedding-8B, forms a coverage histogram over nearest neighbors, adds Gaussian noise calibrated to $\ell$1, and returns the top-$\ell$2 exemplars $\ell$3. The reported setting uses $\ell$4 and $\ell$5 shots per attribute.

The implementation is reported to use Opacus to compute exact Gaussian noise calibrations, and moments or RDP accounting is described as unnecessary because release is a single-shot Gaussian mechanism per layer with basic composition. The paper also describes a sensitivity variant for the direct difference-of-means estimator:

$\ell$6

which yields

$\ell$7

EPSVec instead uses paired differences with $\ell$8.

4. Steering and generation pipeline

At decoding step $\ell$9, the current hidden state at layer $a_{\ell,t}(x) \in \mathbb{R}^d$0 is denoted by $a_{\ell,t}(x) \in \mathbb{R}^d$1. EPSVec applies additive steering using the sanitized, normalized dataset vectors:

$a_{\ell,t}(x) \in \mathbb{R}^d$2

for a per-layer steering strength $a_{\ell,t}(x) \in \mathbb{R}^d$3 (Banayeeanzade et al., 31 Jan 2026). This is applied at every decoding step. If injection occurs at the final hidden layer $a_{\ell,t}(x) \in \mathbb{R}^d$4 with output head $a_{\ell,t}(x) \in \mathbb{R}^d$5, then

$a_{\ell,t}(x) \in \mathbb{R}^d$6

In the main experiments, multi-layer injection is used, with late transformer layers steered; for Llama-3.1-8B these are layers 18–21. Vectors are $a_{\ell,t}(x) \in \mathbb{R}^d$7-normalized. Steering is described as interacting naturally with standard sampling such as temperature and top-$a_{\ell,t}(x) \in \mathbb{R}^d$8/top-$a_{\ell,t}(x) \in \mathbb{R}^d$9; in the main experiments, a single temperature is set and other decoding defaults are used. The steering is tokenwise and constant across timesteps; no gating or context-dependent schedules are used in the core pipeline.

Fixed-shot prompting is integral to the reported pipeline. Every prompt embeds a small fixed-shot scaffold $t \in \{1,\dots,T(x)\}$0 of $t \in \{1,\dots,T(x)\}$1 exemplars selected under $t \in \{1,\dots,T(x)\}$2 via a DP histogram. These shots are used both in generating the synthetic negative set $t \in \{1,\dots,T(x)\}$3 for vector estimation and in conditioning downstream generation. The paper states that fixed-shots stabilize generation in base models and improve distributional alignment by anchoring the format and coarse attributes, which in turn yields a better synthetic negative set so that the vectors capture subtler differences between real and synthetic text.

The reported pseudocode consists of two phases. In vector extraction, paired representations are differenced, clipped, noised, and normalized to produce $t \in \{1,\dots,T(x)\}$4. In sampling, the model is initialized with a prompt containing $t \in \{1,\dots,T(x)\}$5; at each decoding step the hidden states are computed, each steered layer is updated by adding $t \in \{1,\dots,T(x)\}$6, logits are computed, and the next token is sampled. The generated text may optionally be filtered by rejection sampling using Qwen-3-4B-Instruct as an LLM-as-a-judge with threshold overall $t \in \{1,\dots,T(x)\}$7.

This design implies that the privacy-sensitive computation is concentrated in a one-time representation extraction stage, whereas the sampling loop remains operationally close to ordinary inference. That separation is central to the method’s stated compute and privacy advantages.

5. Efficiency and empirical evaluation

Estimating $t \in \{1,\dots,T(x)\}$8 requires $t \in \{1,\dots,T(x)\}$9 forward passes over private samples and $$h_{\ell}(x) = (1 / T(x)) \sum_{t=1}^{T(x)} a_{\ell,t}(x) \in \mathbb{R}^d.$$0 over synthetic candidates to compute $$h_{\ell}(x) = (1 / T(x)) \sum_{t=1}^{T(x)} a_{\ell,t}(x) \in \mathbb{R}^d.$$1, with complexity reported as $$h_{\ell}(x) = (1 / T(x)) \sum_{t=1}^{T(x)} a_{\ell,t}(x) \in \mathbb{R}^d.$$2 for mean-pooling hidden states across $$h_{\ell}(x) = (1 / T(x)) \sum_{t=1}^{T(x)} a_{\ell,t}(x) \in \mathbb{R}^d.$$3 layers (Banayeeanzade et al., 31 Jan 2026). The memory footprint is described as modest, involving storage of per-sample $$h_{\ell}(x) = (1 / T(x)) \sum_{t=1}^{T(x)} a_{\ell,t}(x) \in \mathbb{R}^d.$$4 and accumulation of means. The negative set $$h_{\ell}(x) = (1 / T(x)) \sum_{t=1}^{T(x)} a_{\ell,t}(x) \in \mathbb{R}^d.$$5 is generated once using fixed-shots. During generation, steering adds a single vector addition per steered layer per token and otherwise runs standard decoding. No per-token DP aggregation, batching, or teacher-student PATE steps are required.

Empirical efficiency is reported on a single A100 (80GB). EPSVec is said to generate full 2K samples efficiently, whereas baselines that require per-token aggregation, specifically PP and PP++, face scalability bottlenecks and even fail under tighter budgets; PP++ is reported to fail at $$h_{\ell}(x) = (1 / T(x)) \sum_{t=1}^{T(x)} a_{\ell,t}(x) \in \mathbb{R}^d.$$6. The paper further states that EPSVec achieves strong fidelity with the fewest private samples among compared baselines and remains viable for longer sequences under low $$h_{\ell}(x) = (1 / T(x)) \sum_{t=1}^{T(x)} a_{\ell,t}(x) \in \mathbb{R}^d.$$7, while per-token DP methods often exhaust budget or become prohibitively slow.

The evaluation uses four corpora: IMDb (25K, 2 classes), Yelp Polarity (560K, 2), BioRxiv abstracts (25K filtered to post-2023, 4 categories), and OpenReview ICLR reviews (11K, 2). The target is $$h_{\ell}(x) = (1 / T(x)) \sum_{t=1}^{T(x)} a_{\ell,t}(x) \in \mathbb{R}^d.$$8 synthetic samples per dataset, balanced across classes. Backbones are Llama-3.1-8B in pretrained and instruction-tuned forms. Distributional fidelity is measured via MAUVE with scaling factor 5 and 200 buckets, and downstream utility via BERT fine-tuning in the synthetic-train, real-test setting ("TSTR"). Additional analyses include text quality via LLM-as-a-judge, BERT TRTS, and in-context learning TRTS/TSTR. The paper explicitly notes that higher TRTS or text quality than real data can indicate oversimplified synthetic samples rather than higher fidelity, and identifies MAUVE as the primary fidelity metric.

Dataset	Reported EPSVec++ result	Additional reported utility
IMDb	MAUVE 72.3 at $$h_{\ell}(x) = (1 / T(x)) \sum_{t=1}^{T(x)} a_{\ell,t}(x) \in \mathbb{R}^d.$$9; 67.8 at $y$0	BERT TSTR $y$1–$y$2
Yelp	MAUVE 62.9 at $y$3; 67.3 at $y$4	BERT TSTR $y$5–$y$6
BioRxiv	MAUVE 62.2 at $y$7; 60.7 at $y$8	BERT TSTR $y$9
OpenReview	MAUVE 33.0 at $L$00 and $L$01	BERT TSTR $L$02–$L$03

Across datasets, EPSVec methods are reported to deliver substantial gains in fidelity, especially in smaller-data domains such as IMDb, BioRxiv, and OpenReview, at significantly lower compute than aggregation-based inference methods. The paper reports an average $L$04 MAUVE improvement across four datasets over the next-best method in low-data regimes.

The comparison to baselines is numerically explicit. On IMDb at $L$05, EPSVec++ MAUVE 72.3 exceeds PP++ 14.9, PP 3.1, InvisibleInk 0.4, and Aug-PE 0.5. On BioRxiv at $L$06, EPSVec++ 62.2 exceeds PP++ 1.9, PP 1.7, InvisibleInk 0.9, and Aug-PE 0.5. On OpenReview at $L$07, EPSVec++ 33.0 exceeds PP++ 5.5, PP 2.3, InvisibleInk 0.5, and Aug-PE 0.4.

The paper also reports that instruction-tuned models show lower lexical diversity, for example fewer unique opening 3-grams, and tend to produce overly fluent, homogenized text, whereas pretrained models with EPSVec produce more diverse, nuanced outputs closer to real distributions. MAUVE generally decreases as $L$08 decreases because of increased noise, but non-monotonic effects are also reported: moderate noise can increase diversity and slightly improve MAUVE for transitions from very large $L$09 to moderate $L$10. Compatibility tests on Olmo-3-1025-7B, Qwen-3-4B-Base, and Gemma-3-4B-PT report consistent gains over fixed-shots-only baselines, including average MAUVE improvement $L$11 and BERT accuracy $L$12 on Yelp at $L$13.

6. Hyperparameters, limitations, extensions, and reproducibility

The reported practical defaults for Llama-3.1-8B pretrained are injection layers $L$14, steering coefficients $L$15 per injected layer, clipping $L$16, and noise $L$17 calibrated by

$L$18

The main experiments use $L$19 private samples per class for vector construction, temperature $L$20, top-$L$21 defaults, fixed-shots with $L$22, $L$23, and Qwen-3-Embedding-8B for the embedding model (Banayeeanzade et al., 31 Jan 2026). Generation optionally uses rejection sampling via Qwen-3-4B-Instruct with threshold overall $L$24. For other models, the paper reports indicative settings: Olmo-3-1025-7B layers 15–16 with $L$25; Qwen-3-4B-Base layers 19–20 with $L$26; Gemma-3-4B-PT layers 17–18 with $L$27. These differences are attributed to internal normalization, and the paper states that DP guarantees are unaffected.

Ablation results are reported to show that fixed-shots improve fidelity substantially: on IMDb at $L$28, removing DP-histogram selection degrades EPSVec++ MAUVE from 67.8 to 42.9. Steering is also reported to increase fidelity consistently across zero-shot and fixed-shot baselines; on IMDb, MAUVE increases from zero-shot pretrained 19.2 to EPSVec++ without fixed shots 48.2 and to EPSVec++ with fixed shots 67.8. Later-layer injection is described as more robust, whereas too-large $L$29 over-steers and harms fidelity and quality. Moderate temperature is reported to increase diversity, while overly high temperature causes off-distribution content.

The paper identifies several limitations and risks. Extremely small $L$30 or very tight $L$31 can yield noisy, low-signal vectors that reduce steering effectiveness and fidelity. Domain shift can be problematic if the pretrained model cannot produce a reasonable candidate pool or fixed-shots; in such cases, the synthetic $L$32 may be too mismatched, and dataset vectors may correct large prompt mismatches rather than subtler gaps. Over-steering with large $L$33 can distort semantics or induce off-distribution modes, and earlier-layer injection is described as more sensitive and capable of degrading stability. The paper also cautions that some baselines show high TRTS or text quality because of oversimplification; EPSVec aims to match real distributions, but MAUVE can itself be sensitive to hyperparameters. Residual leakage risks are described as mitigated by DP guarantees, but practitioners are advised to adhere to correct accounting and conservative $L$34. Fixed-shot selection uses a small budget, yet selected exemplars may be close to private data in embedding space, so the paper recommends following organizational policies on release and use.

Several extensions are proposed. These include multi-vector mixtures with one vector per subtopic or label and conditional steering; layer ensembles that weight multiple layers and learn $L$35 via small public validation; low-rank projections or PCA to denoise vector estimates and reduce variance under strong privacy; adaptive $L$36 schedules that vary $L$37 over timesteps; and whitening or RDM-based normalization to reduce layer-wise scaling disparities across models. These are explicitly presented as extensions rather than components of the core pipeline.

For reproducibility, the paper provides code at https://github.com/noiseeboi/EPSVec. The reported backbones are Llama-3.1-8B in pretrained and instruction-tuned forms, with compatibility tests on Olmo-3-1025-7B, Qwen-3-4B-Base, and Gemma-3-4B-PT. The datasets are IMDb, Yelp Polarity, BioRxiv abstracts filtered to post-2023 categories—Neuroscience, Bioinformatics, Microbiology, and Cell Biology—and OpenReview ICLR reviews. The main tables use $L$38, fixed-shot $L$39, and $L$40 chosen per the Gaussian mechanism via Opacus. The theoretical guarantees are summarized as Gaussian mechanism privacy with layerwise sensitivity $L$41, composition to $L$42-DP across layers, DP-free generation by post-processing invariance, and subsampling amplification according to

$L$43

Taken together, EPSVec is characterized as a method that privatizes a dataset-level direction in representation space and then uses that direction for efficient synthetic text generation under differential privacy. The paper’s reported results position its main contribution in the combination of one-time DP vector release, fixed-shot scaffolding, and activation-space steering, with particular emphasis on low-data regimes, long-form generation, and the separation of privacy accounting from the act of sampling itself.