- The paper demonstrates that shared representations in multitask learning are vulnerable to black-box task-inference attacks that expose training task inclusion.
- It rigorously formalizes a threat model and introduces efficient attack algorithms—coordinate-wise variance and pairwise inner product—to evaluate privacy risks.
- Empirical evaluations on vision and language benchmarks reveal high true positive rates for both strong and weak adversaries, highlighting practical privacy challenges in MTL.
Black-Box Privacy Attacks on Shared Representations in Multitask Learning
The paper systematically analyzes privacy risks in multitask learning (MTL) settings, focusing on the vulnerability of shared representations to black-box task-inference attacks. MTL frameworks, widely adopted for training models across tasks with limited individual data, typically rely on a shared representation for embedding data points into a common feature space. This study rigorously demonstrates that these shared components, though abstract, are susceptible to attacks that can reveal the inclusion of specific tasks in the training regime, even when adversaries are limited to black-box access and have only fresh samples from the target task.
The authors establish a novel threat model—task-inference—generalizing privacy attacks beyond sample-level membership to the granularity of entire tasks or distributions. Two adversary variants are articulated:
- Strong adversary: Accesses the actual training samples from the target task.
- Weak adversary: Accesses only fresh samples independently drawn from the target task's distribution.
A formal security game is defined, where the adversary, with black-box access to the shared encoder, attempts to infer whether a challenge task was present during training. Notably, the model does not require shadow or reference models for calibration, marking a significant reduction in practical attack prerequisites.
Theoretical Analysis
Through a mean estimation framework over mixtures of Gaussians, the paper quantifies the adversary's distinguishing power. Key findings include:
- The expected value of the attacker's test statistic is strictly larger when the task is present in training than when absent, for both adversary types.
- The gap between strong and weak adversaries depends on data dimensionality, number of tasks, and samples per task.
- Tracing entire tasks is substantially easier than tracing individual samples; adversarial power increases when attacks are performed at the group rather than individual level.
- The weak adversary's effectiveness is characterized by the diversity among task distributions (the between-task variance), whereas the strong adversary benefits additionally from having training data overlap.
These results rigorously connect task-inference to membership and property inference attacks, revealing that task-based attacks interpolate between various established privacy risks, depending on the definition of the task and data granularity.
Practical Black-Box Attacks
The paper introduces two efficient, purely black-box attack algorithms leveraging the tendency of MTL-shared encoders to produce codependent embeddings for samples from the same task:
- Coordinate-Wise Variance Attack: Computes the trace of the covariance matrix for embeddings of challenge samples; higher variance indicates inclusion.
- Pairwise Inner Product Attack: Computes (absolute) inner products or cosine similarities among embeddings; high mean similarity implies inclusion.
Both attacks operate without any reference OUT tasks or shadow models. Embedding whitening further improves attack signal, neutralizing dominant axes and isolating co-variance patterns tied to training inclusion.
Attack algorithms are succinctly summarized below:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
def variance_attack(encoder, challenge_set):
embeddings = [encoder(x) for x in challenge_set]
cov_matrix = np.cov(np.stack(embeddings).T)
return np.trace(cov_matrix) / len(embeddings[0]) # Test statistic: avg. coordinate-wise variance
def inner_product_attack(encoder, challenge_set):
embeddings = [encoder(x) for x in challenge_set]
# Optionally apply whitening transformation
n = len(embeddings)
similarities = []
for i in range(n):
for j in range(i+1, n):
sim = abs(np.dot(embeddings[i], embeddings[j]))
similarities.append(sim)
return np.mean(similarities) # Higher value suggests inclusion in training |
Empirical Evaluation
Experiments are conducted on both vision (CelebA, FEMNIST) and language (Stack Overflow) benchmarks, under two canonical MTL use cases:
- Personalization: Each task corresponds to a user, as in personalized photo tagging or handwritten character recognition.
- Multiple learning problems: Each task represents a distinct but related learning problem, e.g., identifying different facial attributes or text topics.
Key empirical findings:
- The strong adversary consistently achieves high true positive rates at low false positive rates (e.g., TPR > 80% for FPR < 5% in several settings).
- The weak adversary—using only fresh samples—still surpasses chance prediction, demonstrating that privacy vulnerabilities extend beyond training set memorization.
- Attack success rates are higher when tasks are tightly linked to specific distributions/labels; the gap between strong and weak adversaries shrinks as task inclusion becomes increasingly tied to unique data properties.
- Task-inference attack effectiveness correlates with generalization gaps; as models overfit further to training tasks, attack AUC rises.
Tables in the paper detail balanced accuracy as a function of decision threshold, showing nontrivial distinguishability even under practical black-box constraints.
Implications and Future Directions
The results have both theoretical and practical implications:
- Threat breadth: Task-level privacy leakage can subsume risks typically associated with membership, user, and property inference, depending on task definition, potentially affecting collaborative settings in federated learning and industry-scale personalization.
- Attack surface: Even the most minimal unit of sharing for cross-task generalization, the shared representation, is not exempt from inference attacks. Merely limiting access to task-specific output layers is insufficient for user privacy when black-box embedding queries are possible.
- Defenses: The findings motivate the need for user-level differential privacy or related group-wise privacy tools in MTL, as naive regularization or gradient clipping is insufficient.
- Audit tools: The proposed attacks offer practical auditing techniques for privacy evaluations in real-world MTL deployments, enabling practitioners to empirically assess the vulnerability of their models to task-inclusion leakage.
Prospective Research
Future AI privacy work may focus on:
- Designing shared representations that minimize codependency across tasks without substantially degrading utility.
- Developing robust, scalable user-level differentially private optimization techniques suitable for deep shared encoders.
- Extending the analysis to settings with dynamic task definitions or continual learning, where task-inclusion evolves over time.
- Exploring defenses leveraging adversarial training or randomized response in the embedding space.
In summary, the study provides a comprehensive theoretical and empirical treatment of group-level privacy risks in multitask learning and offers practical black-box attack methodologies for privacy auditing in contemporary ML systems.
