ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?

Abstract: Differentially private (DP) text synthesis promises to unlock sensitive corpora for model training, but it remains unclear whether DP synthetic data transmits genuinely new knowledge and capabilities present only in those corpora. This is because existing evaluations rely on tasks that are nearly solvable without training, so strong benchmark performance does not establish that DP synthesis can substitute original data access. Thus, we introduce ContinuousBench, a continuously and automatically-regenerated benchmark that measures capability gain from DP synthetic text. Each quarter, a new release pairs a never-before-seen training corpus with a derived QA set, constructed to be: (1) unsolvable sans-corpus; and (2) learnable under DP, as the tested knowledge is supported by hundreds of independent records. Researchers produce DP synthetic data from the training corpus and run our standardized training and evaluation harness on their synthetic data to measure gains. We instantiate two tracks: Geminon, a procedurally-generated dataset about fictional creatures; and News, a stream of newly crawled public news articles. Although standard benchmarks are nearly saturated, on ContinuousBench we find that non-private synthesis transfers substantial knowledge from the original corpus, while state-of-the-art DP synthesis methods generally fail to do so, even at $\varepsilon=100$.

• The paper introduces ContinuousBench as a benchmark to measure capability gains from DP-synthetic text using fresh, access-dependent tasks.
• It shows that state-of-the-art DP synthesis methods dramatically underperform compared to non-private synthesis in transmitting corpus-specific knowledge.
• The study reveals a decoupling between distributional similarity and factual QA accuracy, highlighting limitations of current DP-SGD approaches.

ContinuousBench: A Systematic Evaluation Framework for Differentially Private Synthetic Text

Motivation and Context

Differential privacy (DP) has emerged as the canonical framework for balancing knowledge extraction and privacy guarantees when training on sensitive corpora. While the vision of differentially private text synthesis is compelling—enabling training on sensitive or proprietary text while providing strict privacy guarantees—the operative question is whether DP synthetic data can convey genuine new capabilities that would otherwise be inaccessible, such as those uniquely encoded within private corpora. Previous literature has largely failed to address this question, due primarily to saturated or confounded benchmarks: evaluated tasks are either solvable by pretrained models (precluding measurement of corpus-specific knowledge transfer), driven by style rather than fact, or are confounded by teacher-student mismatches that do not isolate the corpus signal.

Problem Formulation and Benchmark Design

The authors present CONTINUOUSBENCH (2606.01849), a rigorously designed, continuously regenerated evaluation benchmark for DP-synthetic text. The methodology enforces three primary constraints:

1. Access-dependency/Freshness: Each release pairs a never-before-seen training corpus with a constructed QA set that cannot be solved without access to this release, removing contamination or pretraining confounds.
2. Grounded, Factual QA: Evaluation focuses on factual QA, querying for population-level knowledge (supported by hundreds of independent records), not singleton facts, thus measuring capacity for aggregate knowledge extraction under DP.
3. Frontier Constraint: The generator used to produce synthetic data and the downstream evaluator are matched in size and architecture, tightly controlling for distillation effects and ensuring capability gains are attributable to the data, not model scaling or teacher knowledge.

Two evaluation tracks are instantiated. GEMINON is a procedurally generated, fictional domain that allows for fine-grained control and unambiguous freshness. NEWS captures real-world properties, constructed from a rolling news crawl past the training cutoff of all relevant LMs. Both tracks facilitate targeted investigation of the factual knowledge transfer and learning dynamics under DP.

Experimental Results

Benchmark Validation

Base model checkpoints achieve near chance performance on the QA tasks without access to the corpus, direct evidence that the tasks are not contaminated and knowledge must be learned from the release. Training the same model on the real corpus yields substantial improvements (e.g., on GEMINON, accuracy jumps from 1.1% to 96.4% for a 4B model), validating the benchmark’s capacity to measure corpus-conditioned skill acquisition.

Capability Gap: DP-Synthetic versus Non-Private Synthesis

A large quantitative gap emerges between non-private ("Syn", $\epsilon = \infty$) and DP-synthetic (DP-Syn) data. For GEMINON, downstream QA accuracy with non-private synthetic corpora rises above 92%, while state-of-the-art DP-Syn ($\epsilon = 100$) yields only 13.7%. On NEWS, the analogous gap is 65.5% vs 20.5%. When tightening privacy to $\epsilon = 10$, the DP utility further collapses, with accuracy often close to the floor. These findings indicate that, even with high privacy budgets, current DP text synthesis algorithms fail to support substantive knowledge transfer.

Analysis of Distributional vs Factual Metrics

The authors show that metrics such as MAUVE, which measure the distributional similarity of generated corpora to the target, degrade mildly under DP constraints, while factual QA accuracy collapses. For instance, MAUVE on GEMINON drops by only ~0.10 (0.83 to 0.73) when moving from Syn to DP-Syn@10, but factual accuracy collapses almost completely. This reveals that distributional similarity is a weak proxy for factual knowledge retention under DP.

Failure of Private Evolution

A recent DP text synthesis approach—Private Evolution (PE)—which produces DP data using repeated paraphrasing and selection under differential privacy, is unable to transfer factual knowledge in the CONTINUOUSBENCH framework. Although PE achieves reasonable FID, downstream learning is at or below the "no training" baseline, indicating that PE can produce stylistically plausible, but factually vacuous, corpora when the target knowledge is not present in the generator.

Direct DP Fine-Tuning: Diagnostic Upper Bound

To isolate where the failure mode lies, the paper also studies direct DP-SGD fine-tuning on the original corpus (i.e., DP learning without synthetic data mediation). Even in this idealized setting, factual QA performance is far lower than in the non-private regime, indicating that DP-SGD itself struggles to encode fresh corpus-specific knowledge. Thus, the bottleneck is not just in the synthetic data generation or retraining pipeline, but is more fundamentally tied to the inability of DP-SGD to preserve rare facts even with substantial repetition.

Repetition and Learnability

The authors leverage the fact that each QA is tied to its support count (number of independent records containing the answer). As expected from DP theory, facts with higher support counts are modestly more learnable under DP, but the gap to the non-private regime remains substantial, especially in real-world noisy settings. Singleton facts are never reliably learned, consistent with the privacy guarantee.

Implications and Future Directions

This work demonstrates that, under realistic, contamination-free scenarios measuring factual transfer, current DP text synthesis pipelines are unable to deliver meaningful knowledge transfer, even with substantial privacy budgets. The results are robust against evaluation confounds that have plagued prior work, providing a clear empirical lower bound on the substitutability of DP-synthetic data for direct data access.

Practical implications: Synthetic text generated under DP with current algorithms cannot serve as a drop-in replacement for direct access to sensitive corpora when new factual capabilities are sought, even with significant engineering effort.

Theoretical implications: The central challenge is not just post-processing or sampling, but lies fundamentally in the gradient clipping and noise injection of DP-SGD, especially where information is encoded in rare or high-norm tokens. Mitigating this will require advances in both private optimization and more data-efficient, corpus-specific knowledge retention mechanisms under DP.

Benchmarking impact: CONTINUOUSBENCH sets a new standard for evaluation: it is access-dependent, continually regenerated, and removes confounders present in previous DP text evaluation pipelines. It will likely become a required yardstick for demonstrating progress in privacy-preserving text generation.

Research opportunities:

• Improving DP learning dynamics for factual knowledge (potentially through more careful reuse of population-level redundancy, or targeted mechanisms for high-support facts).
• Understanding and mitigating the representational bottleneck imposed by the per-example gradient clipping of DP-SGD in language modeling.
• Developing architectures or training paradigms that are explicitly designed for data-efficient factual acquisition under DP.
• Using CONTINUOUSBENCH as a driver for principled comparisons as new algorithms for DP-synthetic text are developed.

Conclusion

CONTINUOUSBENCH exposes a critical deficit in current DP text synthesis frameworks: despite looking plausible, DP-synthetic corpora fail to transfer the factual, corpus-specific skills that give value to private text in the first place. By constructing a continuously refreshed, contamination-resistant benchmark focused on population-level fact transfer, the authors pinpoint the real barrier to practical DP text synthesis. Closing the capability gap, as measured by CONTINUOUSBENCH, defines the principal open problem in the area. Progress on this benchmark will be an essential indicator of advancements in privacy-preserving language modeling.