Technical Threat Models of Privacy Leakage

• Technical threat models of privacy leakage are formal frameworks that define how attackers exploit vulnerabilities in ML systems using inference and reconstruction strategies.
• They employ rigorous metrics such as differential privacy, ROC-AUC, and empirical measures to quantify leakage, highlighting modality-specific risks across federated, tabular, and vision-language models.
• The topic surveys defense mechanisms like noise injection, post-processing, and secure aggregation to manage the trade-off between data utility and privacy.

Technical threat models of privacy leakage formalize the ways in which adversaries can infer, extract, or reconstruct sensitive information from machine learning systems and data ecosystems, particularly under collaborative, distributed, or black-box deployment scenarios. These models describe the knowledge, access, and capabilities of the attacker, quantify leakage under rigorous metrics such as differential privacy, and characterize the propagation of private information due to architectural, protocol, or regularization choices. Approaches span modalities (vision-language, tabular, sequential, federated) and encompass both classical and neuromorphic ML pipelines. This article delineates the principal threat models across contemporary research, provides mathematical formalizations, synthesizes empirical findings on privacy–utility trade-offs, and surveys defense mechanisms and future challenges.

1. Adversarial Capabilities and Modality-Specific Threat Models

Threat models precisely define attackers’ knowledge, query privilege, and ability to access or manipulate systems:

Black-Box Membership Inference (MM-VLMs):

For multi-modal vision–LLMs such as BLIP, PaliGemma 2, and ViT-GPT2, the attacker may only submit image queries and observe generated captions—without access to model internals, activations, or logits. Membership inference is formulated as a binary classifier $A(u,v)$ such that $A(u,v)=1$ if $(u,v)$ originated from the training set, exploiting semantic and lexical similarity scores (MPNet, ROUGE-2) to distinguish members from non-members (Amebley et al., 24 Nov 2025).

Gradient Leakage in Federated Learning:

Adversaries can be white-box, where a server observes raw parameter updates or gradients from clients prior to aggregation. Reconstruction attacks invert the gradient under the model’s architecture and loss function to recover the underlying data sample. These attacks transform the leakage estimation into an inverse-optimization problem (Fernandez-de-Retana et al., 27 Oct 2025).

Synthetic Data and Tabular Generative Models:

In the evaluation of privacy risks for synthetic data, the adversary can operate under no-box or no-box-calibrated regimes, where only synthetic records (and possibly a reference sample) are observable. Membership inference attacks produce scalar evidence scores, and aggregate maximum risks over diverse attack strategies to estimate worst-case leakage (Ward et al., 22 Sep 2025).

Network Traffic Synthesis and Trace-based Attacks:

TraceBleed implements a membership inference adversary at the traffic-source level, combining contrastive learning and behavioral fingerprinting across flows. The adversary’s advantage reflects the probability gain over random guessing, and empirical results show correlation between synthetic data volume and leakage (Jin et al., 15 Aug 2025).

Overlapping Grouped Federated Learning (DP-OGL):

The honest-but-curious adversary model is extended to support overlapping privacy groups. Leakage is quantified pairwise by tracking propagation delay (the number of group-hops needed for private updates to reach adversary), and degradation factors due to successive Gaussian noise injections (Kiani et al., 6 Mar 2025).

2. Formal Privacy Metrics and Leakage Quantification

Renyi Differential Privacy (RDP):

RDP is used to bound information leakage under sequential and parallel composition, permitting more granular analysis across networked nodes or overlapping groups. For decentralized gossip, RDP growth is rigorously shown to be $O(T)$ per node, a substantive advance over the $O(T^2)$ in previous pairwise network-DP analyses (Koskela et al., 26 May 2025).

ROC-AUC and Similarity Gap ($\Delta_\tau$):

Black-box MIA efficiency is measured by ROC-AUC, quantifying discriminability across all thresholds. The similarity gap $\Delta_\tau$ specifically measures the expectation difference between similarity scores for member and non-member pairs—smaller gaps indicate stronger privacy (Amebley et al., 24 Nov 2025).

Empirical Differential Privacy ($\epsilon_{\mathrm{emp}}$):

Synth-MIA operationalizes empirical risk as the supremum advantage achieved over all attacks and thresholds, reflecting the smallest privacy budget compatible with observed leakage. These metrics highlight discordance with non-adversarial measures (e.g., DCR-Proportion) (Ward et al., 22 Sep 2025).

Reconstruction and Membership Recall:

Federated LLM attacks quantify leakage by reconstruction rate (DRR), exposure metrics (e.g., Carlini–Papernot rank), and membership inference recall, all computed per model snapshot and informed by tampered layer changes (Rashid et al., 2023).

3. Leakage Propagation, Temporal Effects, and Noise Degradation

Propagation Delay in Overlapping Groups:

DP-OGL proves that privacy leakage between worker pairs depends not just on direct group overlap, but also on the minimal group-to-worker path length $\tilde\rho_{m',i}$ and the frequency $S$ of inter-group swaps. Leakage is thus temporally delayed as updates traverse intermediate groups (Kiani et al., 6 Mar 2025).

Information Degradation via Noisy Composition:

Repeated Gaussian noise addition at each group or synchronization acts as randomized post-processing, introducing a multiplicative degradation factor $\mu$, rigorously diminishing leakage with each hop. The composition yields closed-form bounds incorporating both delay and degradation, critical in string or ring topologies (Kiani et al., 6 Mar 2025).

Temporal Sequence Attacks:

In sequential data releases, an attacker using bi-directional HMMs and reinforcement learning can aggregate inferences across multiple timesteps, leveraging both past and future published regions. This undermines per-step privacy guarantees and fundamentally exposes a blind spot in time-agnostic DP (Cui et al., 28 Oct 2025).

4. Impact of Structural Inductive Biases and Regularization

Neuroscience-Inspired Topological Regularization:

By imposing a topographic penalty on internal representations ($R_\mathrm{topo}$), multi-modal networks reduce memory for specific training examples, narrowing the utility-leakage gap under membership inference attacks. Experimental findings show a drop of up to 30.5 absolute ROC-AUC points with negligible impact on semantic fidelity (Amebley et al., 24 Nov 2025).

Adversarial Training Amplifies Leakage:

Contrary to robustness expectations, adversarially trained deep nets in federated settings facilitate high-fidelity feature recovery and input reconstruction, due to decoupled batch gradients, dominating single-sample signals even at large batch sizes (Zhang et al., 2022). This presents a significant privacy amplification vector.

Machine-Unlearning as Poisoning:

Fine-tuning on intentionally “unlearned” challenge sets increases over-memorization during downstream adaptation, resulting in amplified membership inference and data extraction leakage. Empirically, poisoned pretrains lead to up to 12.3 percentage point increases in best accuracy and severe utility–privacy decoupling (Rashid et al., 30 Aug 2024).

5. Defense Mechanisms and Limitations

Noise Mechanisms and DP Variants:

Standard DP-SGD (per-sample clipping plus Gaussian noise) mitigates gradient leakage, achieving SSIM near zero for recovered images. In contrast, parameter-proportional DP mechanisms like PDP-SGD, despite formal $(\varepsilon,\delta)$-DP, are empirically ineffective against reconstruction attacks (Fernandez-de-Retana et al., 27 Oct 2025). Defense efficacy is tightly linked to both privacy budget and algorithmic structure.

Post-processing and SMT-Constrained Repair:

TracePatch demonstrates the feasibility of adversarial ML post-processing combined with SMT constraints to shield synthesized traces from MIA leakage, achieving random-guess privacy with minimal loss in distributional fidelity (Jin et al., 15 Aug 2025).

Group and Round-wise Aggregation:

Federated and overlapping group settings benefit from secure aggregation, temporal hiding of intermediate snapshots, and selective layer privacy amplification. However, increased defense stringency correlates with utility degradation, necessitating careful architectural and protocol design (Rashid et al., 2023, Kiani et al., 6 Mar 2025).

Quality–Privacy Trade-offs:

Empirical large-scale audits reveal a positive correlation between synthetic data fidelity and privacy leakage. Smaller training sets leak more, while non–adversarial similarity metrics underestimate true risk, mandating MIA-based audits for synthetic data releases (Ward et al., 22 Sep 2025).

6. Implications for Future Threat Modeling and Research

Technical threat models continue to evolve under increasing modality diversity, collaborative paradigms, and adversarial innovation. Key fronts include time-aware privacy accounting, structural regularization for privacy–utility balancing, systematic parameterization of leakage scenarios (KART), and empirical upper–bound estimation of risk. Persistent challenges such as sequential compositionality, correlated record leakage, cross-modal attacks, and formalization of actor-driven threat propagation in complex IoT systems demand integrative, cross-disciplinary approaches (Alalade et al., 24 Oct 2025, Nakamura et al., 2020).

Ongoing work seeks to extend Markovian or dynamic probabilistic modeling into PTMFs for IoT, automate empirical privacy risk scoring for real-world deployments, and establish robust privacy certificates for pretrained model sharing and synthetic data pipelines. As the breadth and depth of privacy attack vectors expand, rigorous, scenario-specific threat modeling and robust, context-adaptive defenses remain foundational to trustworthy machine learning systems.