Privacy-Preserving Data Drift Detection and Recovery for Large-Scale LLM Applications via Proxy Representations

Abstract: LLM applications deployed at scale face a fundamental challenge: privacy constraints prevent direct inspection of user interactions, making it difficult to obtain any representative evaluation dataset or to track the ongoing evolution of production traffic. We present ProxyDrift, a framework that (i) identifies and measures drift between production traffic and offline evaluation sets, and (ii) constructs and refreshes those evaluation sets accordingly; all without access to raw user data. Our approach operates entirely on non-PII proxy representations: structured, multi-dimensional descriptors derived from LLM-based classification of user interactions. We introduce (1) a chance-calibrated, redundancy-aware (RA) alignment score that aggregates per-dimension drift measurements via mutual information; (2) a conditional sampler that generates synthetic proxies respecting inter-dimensional dependencies; (3) a roundtrip consistency analysis that exposes generator/classifier disagreements and guides proxy taxonomy refinement; and (4) a feedback-linkage analysis that ties per-dimension and per-value proxy distributions to user satisfaction, surfacing actionable failure and success modes. Serving hundreds of millions of users, ProxyDrift enables continuous drift monitoring and targeted synthetic data generation without exposing sensitive user data. Experiments confirm strong roundtrip consistency, discriminator-level indistinguishability of synthetic queries from human queries, and tight end-to-end alignment (RA~0.9) with production.

• The paper introduces ProxyDrift, a privacy-preserving framework that measures LLM traffic drift and generates production-aligned evaluation data using 21-dimensional, non-PII proxy representations.
• Its permutation-calibrated Jensen–Shannon alignment score and redundancy-aware aggregation achieved 0.917 alignment for conditional synthetic data, compared with 0.284 for a legacy curated set, exposing inflated offline quality scores.
• The Chow–Liu sampler, statistical shrinkage, and roundtrip taxonomy refinement reduced reconstruction distance by 44% and failures from 6.9% to 0.4%, while proxy diagnostics improved Excel benchmark accuracy by up to 55.6 points.

Motivation and problem setting

LLM applications operating under strict privacy and compliance constraints cannot expose raw user queries, responses, or grounding documents for offline analysis. As a result, practitioners lack any verified view of production traffic, so offline evaluation datasets remain static snapshots whose distributional relevance to live traffic is unknown and degrades over time. This paper introduces ProxyDrift, a framework from Microsoft that addresses both detection of this misalignment and construction of production-aligned evaluation sets, operating entirely on non-PII proxy representations — structured multi-dimensional descriptors produced by an LLM classifier running inside the production compliance boundary, plus deterministically observable properties (length buckets, language, grounding types). The only data persisted to telemetry is the proxy object itself.

The framework provides four capabilities: drift measurement via a chance-calibrated alignment score; evaluation-set generation via a Chow–Liu conditional sampler with LLM hydration into natural-language queries; continuous monitoring through scheduled aggregation pipelines; and diagnostics via a roundtrip consistency loop (driving taxonomy refinement) and feedback-linkage analysis tying proxy distributions to user satisfaction.

Proxy representation

A proxy assigns values along $D = 21$ categorical dimensions defined by a JSON schema, spanning intent, formality, structure, tone, output type/format/length/creativity, domain, content specificity, observable length/language buckets, and grounding types. Dimensions are classified versus observable, nominal versus ordinal, and single- versus multi-valued; each label carries an integer confidence score in $\{0,\dots,5\}$ with 0 reserved for Unknown. A self-correction loop feeds schema violations back to the classifier for up to $K=10$ attempts, resolving malformed outputs without human intervention. Extensions handle multi-turn interactions at both interaction level and per-query-with-context level.

Drift measurement

Per-dimension divergence uses the normalized Jensen–Shannon distance over categorical distributions, chosen because it penalizes support mismatch heavily — a category present in one side and absent in the other incurs a large penalty. Raw distances are converted into a universally comparable score via permutation calibration: shuffling the observed distribution $T{=}50{,}000$ times yields a baseline expected distance under random label assignment, and alignment is

$$a(P,O) = \max\left(0,\;1 - \frac{d(P,O)}{\hat{\mathbb{E}}_\pi[d(P,\pi(O))]}\right).$$

A notable design detail: an earlier baseline drawn from a Dirichlet distribution was always dense and severely underestimated baseline distance for sparse production distributions, yielding misleadingly low scores; holding the reference fixed and shuffling only the observed side preserves real-world sparsity. Aggregate scoring across dimensions uses a redundancy-aware (RA) weighted average: dimensions are processed in decreasing order of importance weight times entropy, and each dimension's weight is discounted by its normalized mutual-information overlap with previously processed dimensions ($g_i = \max(0, 1-\lambda r_i)$, $\lambda{=}0.8$), preventing clusters of correlated dimensions from dominating the aggregate.

Synthetic dataset generation

An independence sampler calibrated to per-dimension marginals discards cross-dimensional correlations and produces implausible combinations (e.g., Legal Document output with Casual tone), which bias quality estimates. ProxyDrift instead models the joint with a Chow–Liu tree: maximum-weight spanning tree with plug-in mutual information as edge weights, built in $O(D^2\log D)$ time, rooted at the highest-degree hub node to limit error propagation through conditional steps.

Two finite-sample corrections are central. First, because plug-in MI is positively biased roughly as $(|\mathcal V_i|-1)(|\mathcal V_j|-1)/(2N_{ij})$, edge weights are multiplicatively dampened by a sigmoid factor keyed on sample size relative to table area (with $c{=}5$ following Cochran's rule), suppressing spurious edges among high-cardinality pairs without risking negative MI estimates. Second, each parent-value slice's empirical conditional is shrunk toward the marginal prior by coefficient $\{0,\dots,5\}$0, where $\{0,\dots,5\}$1 is a Pearson chi-squared p-value testing the slice against the prior (with a relaxed Cochran rule falling back to pure prior when cell counts fail); this prevents rare parent values from locking in spurious dependencies. The resulting proxies are hydrated into natural-language queries by an LLM generator that reverses the classification step.

Evaluation results

End-to-end alignment. Against a week of Excel document-generation traffic, both synthetic sets achieve "good" RA alignment — the conditional sampler reaches 0.917 versus 0.897 for the independent sampler — while the legacy hand-curated test set scores 0.284, in the bad band, with per-dimension alignments typically below 0.40. The consequence is concrete: LLM-judged quality scores drop by roughly 0.3 points on the aligned synthetic set (Accuracy 4.29→3.71, Reliability 3.85→3.44), indicating systematic inflation in conventional offline evaluation. The authors note the basic sampler scores high largely because the RA metric evaluates marginals it is directly calibrated to; the conditional sampler additionally preserves cross-dimension structure.

Roundtrip consistency. The generation–classification cycle serves as both an intrinsic fidelity measure and the optimization signal for taxonomy refinement. On the initial intuition-driven taxonomy, mean reconstruction distance was 0.282–0.291 with a 6.9% roundtrip failure rate; crucially, within-proxy variance was far smaller than across-proxy variance, showing the residual mismatch was systematic rather than noise. Per-dimension confusion matrices (built via proportional-redistribution mass accounting and summarized by size-adjusted diagonal concentration) drove iterative schema changes: retiring weak dimensions, merging/splitting values, introducing new dimensions, and rewriting prompts. Refinement cut mean distance to 0.158 (a 44% relative reduction) and failures to 0.4%; problematic dimensions improved sharply (query_engagement diagonal concentration 0.51→0.81, query_structure 0.65→0.81), while output_type held near-perfect (0.89→0.95). Multi-turn variants behave consistently (interaction-level $\{0,\dots,5\}$2, query-level 0.153), suggesting the representation extends to conversational settings without fidelity loss. Remaining failure modes are informative: rare emotions collapse into Neutral, and ordinal neighbor-smearing reflects genuine construct ambiguity rather than model failure.

Naturalness. An independent unguided LLM discriminator cannot reliably distinguish 100 synthetic from 100 human queries (54% accuracy, $\{0,\dots,5\}$3). A guided prompt raises accuracy to 65%, but the gain comes almost entirely from identifying human queries via surface markers (typos, code-switching); synthetic recall remains 39%, i.e., most synthetic queries still read as human.

Predictive signal and case study. Using binary thumbs-up/down as target, seven dimensions clear a stability bar (held-out CLL gain SNR ≥ 2): sentiment, intent type, emotion, domain, output creativity, formality, and word count. Per-value lift analysis shows dimension-level signal is diluted by dominant uninformative modes: Frustration-laced queries elicit thumbs-down 93% of the time ($\{0,\dots,5\}$4), while Summarization requests lift satisfaction to 51% against a 27% baseline ($\{0,\dots,5\}$5).

Applying the diagnostic to QA over Excel files (2,346 Copilot queries, 29.5% DSAT), four Excel-specific proxy dimensions (formula, filter/sort, chart, cell-styling requirements) all showed elevated DSAT rates. Augmenting the workbook encoding to preserve these semantics recovered 20–60% of held-out DSAT cases depending on category, and improved accuracy on sampled public spreadsheet benchmarks (RealHiTBench, SheetBench, MiMoTable) by +16.7 to +55.6 points on matched feature distributions.

Limitations and open questions

Several caveats bear on the results. Alignment experiments were run against the pre-refinement taxonomy and reported primarily for the Excel scenario, though the authors state other document types yield nearly identical results. The basic sampler's high RA score highlights a structural property of the metric: it evaluates marginals, not joint structure, so marginal-calibrated samplers look good regardless of dependency fidelity. The naturalness test uses a single discriminator model and a small balanced set; guided discrimination at 65% suggests surface tells do exist and could matter more with stronger or human discriminators. The discriminative-power analysis rests on binary thumbs signals from two specific scenarios, and the Excel refinement validation samples only five cases per proxy category, so post-refinement recovery percentages carry wide uncertainty. Open questions left by the paper include extending proxy coverage to grounding documents and generated artifacts, correlating proxies with runtime quality metrics rather than thumbs alone, and whether synthetic proxies can serve agent optimization beyond evaluation.

Conclusion

ProxyDrift demonstrates that structured, non-PII proxy representations suffice for meaningful drift monitoring and evaluation-set maintenance in privacy-constrained LLM deployments at scale. Its statistical core — permutation-calibrated JSD alignment with redundancy discounting, and a Chow–Liu sampler with sigmoid-dampened MI and chi-squared shrinkage toward priors — is deliberately conservative about committing to dependencies unsupported by finite telemetry. The deployment evidence (RA ≈ 0.9, 0.4% roundtrip failures after refinement, unguided discriminator indistinguishability, and measurable downstream engineering gains) supports the central claim that aggregate proxy statistics can substitute for raw-data access in both monitoring and data curation, provided the taxonomy itself is treated as an iteratively refined artifact rather than a fixed design.