Privacy-Preserving Federated Embedding

Updated 27 November 2025

Privacy-preserving federated embedding learning is a set of techniques that enables collaborative model training on decentralized data while keeping sensitive information local.
These methods use a split architecture with shared and private parameters, integrating differential privacy and secure aggregation to mitigate inference and reconstruction risks.
Empirical benchmarks show that federated embedding approaches maintain competitive accuracy with centralized models while reducing communication overhead and enhancing data security.

Privacy-preserving federated embedding learning encompasses a family of methodologies that enable multiple data holders to collaboratively train embedding-based models—such as user representations, document or node embeddings, and class vectors—without sharing raw data or revealing sensitive information during representation formation or aggregation. These approaches rely on locally training embedding parameters, enforcing communication constraints, and introducing cryptographic or differential privacy mechanisms to mitigate inference and reconstruction risks. The field comprises foundational statistical frameworks, cryptographically secure protocols, and empirical advances enabling embedding learning in diverse domains including personalization, recommendation, knowledge graphs, medical imaging, and retrieval-augmented LLMs.

1. Architectural Foundations for Federated Embedding Learning

Most privacy-preserving federated embedding learning systems are built on variants of the federated averaging (FedAvg) protocol, utilizing a principled split between shared and private model parameters. A canonical architecture is detailed by Federated User Representation Learning (FURL), which divides model parameters into:

Federated parameters ( $\theta_F$ ): weights of shared layers (e.g., encoder, feature extractor) synchronized across participants by averaging.
Private parameters ( $\theta_{P,u}$ ): user- or client-specific embedding vectors, trained solely on local devices, never transmitted to the server or peers.

A personalized neural model computes predictions as $\hat y = f(x;\theta_F,\theta_{P,u})$ , with $\theta_F$ broadcast from the server and updated through aggregation of local deltas, while the local $\theta_{P,u}$ is optimized independently and retained device-side. This architecture yields provable equivalence in convergence rates and final model quality compared to centralized training with pooled embeddings, under the constraint that individual user losses depend strictly on their own private parameters; i.e., $\partial L_i / \partial \theta_{P,j} = 0$ for $j \neq i$ (Bui et al., 2019).

2. Privacy Mechanisms: Differential Privacy and Cryptographic Protocols

To enforce robust privacy against various threat models (curious servers, colluding clients, external adversaries), federated embedding learning employs multiple layers of privacy preservation.

Differential Privacy (DP)

Local DP: Embeddings are perturbed with calibrated Laplace or Gaussian noise before sharing, as in FedCL where $L_1$ -clipped user vectors $\mathbf{u}_i$ have iid Laplace noise added per coordinate. This guarantees that released noisy embeddings render each true vector indistinguishable within $\epsilon$ -LDP (Wu et al., 2022).
Global/Composition DP: Gradients and updates are clipped and noise is injected per round, tracked with advanced DP accountants (e.g., Moments, RDP, zCDP), yielding user-level privacy guarantees such as $(\epsilon, \delta)$ -DP (Xu et al., 2022, Salvo et al., 3 Jul 2025).

Cryptographic Aggregation

Secure Aggregation: Secure multi-party computation protocols (e.g., Secret Sharing, Homomorphic Encryption) hide individual updates or embedding vectors, so the server aggregates only masked or encrypted values (Solomon et al., 8 Mar 2024, Tang et al., 2022, Hangdong et al., 2023, Mao et al., 27 Apr 2025).
Homomorphic Encryption: Clients encrypt gradients or embedding statistics (e.g., CKKS FHE scheme), allowing the server to aggregate without decrypting (Mao et al., 27 Apr 2025).
Information-theoretic Guarantees: Protocols such as SecEA ensure that neither the server nor up to $T<N/2$ colluding clients learn anything about private entity sets or embeddings beyond what is implied by global aggregates, enabled by multi-secret sharing and PIR-style queries (Tang et al., 2022, Zhang et al., 2022, Peng et al., 2021).

3. Federated Embedding Learning Algorithms and Protocols

A variety of learning algorithms accommodate privacy constraints:

FedAvg-Style Schedules: Shared weights updated via federated averaging; private embeddings updated locally.
Contrastive and Retrieval Losses: InfoNCE or similar contrastive objectives are employed in RAG and recommendation systems, with semi-hard negative sampling facilitated by privacy-perturbed clustering of embeddings (Wu et al., 2022, Mao et al., 27 Apr 2025).
Autoencoder and Generative Modeling: DP Conditional VAEs are federated, with local decoders updated via DP-SGD and synthetic embeddings serving downstream tasks (Salvo et al., 3 Jul 2025).
Graph Embedding Protocols: In federated graph-based models, only mid-layer (contextual) embeddings are exchanged, not raw node features; in knowledge graph settings, relation-only aggregation and adversarial PATE-based alignment preserve privacy (Pan et al., 2022, Yu et al., 2022, Zhang et al., 2022, Peng et al., 2021).
Projection and Masking: Orthogonal random projections and mask-unmask operations prevent server-side recovery of true class or identity embeddings, shown to be mathematically equivalent to unprotected training in IPFed (Kaga et al., 7 May 2024).

Algorithmic pseudocode for these approaches follows the pattern: local DP/private computation $\to$ privacy-perturbed masking/encryption $\to$ secure or federated aggregation $\to$ update and repeat, with per-client privacy enforced throughout.

4. Communication Efficiency, Scalability, and Trade-offs

Federated embedding approaches markedly reduce communication and computational burden:

Communication Cost: Class-prototype sharing or mid-layer embedding aggregation can require only $\sim1$ –$5$ KB per round compared to $>100$ KB for full gradient or model transmission, as in FedPH, FURL, DP-CVAE, and Feras (Bui et al., 2019, Hangdong et al., 2023, Yu et al., 2022, Salvo et al., 3 Jul 2025).
Model Flexibility: Embedding-based data sharing models (e.g., DP-CVAE) enable diverse downstream tasks with a single compact representation, outperforming task-specific FL classifiers in adaptability (Salvo et al., 3 Jul 2025).
Privacy–Utility Trade-off: Tuning DP noise parameters $(\epsilon, \delta)$ and cryptographic aggregation thresholds ( $t$ in THE) impacts accuracy: utility drops range from $0$– $5\%$ for moderate privacy ( $\epsilon \sim 2$ –$5$), but strong privacy ( $\epsilon\leq1$ ) is obtainable with modest accuracy degradation (Xu et al., 2022, Hangdong et al., 2023).

5. Empirical Benchmarks Across Domains

Privacy-preserving federated embedding learning is validated across a variety of domains:

Personalized User Modeling: FURL achieves $8\%$ and $51\%$ model improvements with only $0\%$ and $4\%$ drop compared to centralized settings (Bui et al., 2019).
Face Recognition: Federated methods with and without secure aggregation deliver nearly identical performance to centralized models; GAN-generated impostor faces enable edge-based negative sampling (Solomon et al., 8 Mar 2024, Kaga et al., 7 May 2024).
Recommendation: FedCL with LDP clustering recovers semi-hard negatives and supports contrastive learning robust to DP noise (Wu et al., 2022).
Medical Imaging: DP-CVAE yields higher accuracy on synthetic embedding classification compared to FL and DP-CGAN, at $5\times$ lower parameter cost, even at $\epsilon<1$ (Salvo et al., 3 Jul 2025).
Knowledge Graphs: Relation-only aggregation (FedR), adversarial DP alignment (FKGE), and secure embedding aggregation (SecEA) achieve strong privacy and up to $99.9\%$ reduction in communication with negligible accuracy loss (Peng et al., 2021, Zhang et al., 2022, Tang et al., 2022).
Graph Neural Networks: Feras’ embedding-sharing architecture proves theoretically contractive and empirically robust to high private-node ratios (Yu et al., 2022).
Prompt Injection Detection & LLM Security: Embedding-based federated detection matches centralized accuracy while exposing no prompt data (Jayathilaka, 15 Nov 2025).
Retrieval-Augmented Generation: FedE4RAG’s homomorphic-encrypted federated retriever yields superior retrieval and generation accuracy on real-world financial corpora compared to centralized and vanilla FL baselines (Mao et al., 27 Apr 2025).

Tables in source publications report quantitative metrics such as accuracy, F1, hit rates, MRR, EER, privacy guarantees, and per-round communication size.

6. Limitations, Open Questions, and Future Directions

Active limitations include:

Dataset Size and Diversity: Some empirical studies (e.g., prompt injection detection) use limited datasets; scaling to larger, more diverse distributions is an open challenge (Jayathilaka, 15 Nov 2025).
Robustness to Advanced Threats: Poisoning attacks, adaptive adversaries, and model inversion risks (especially on released embeddings or logits) require additional defenses, including differential privacy and robust aggregation (Wu et al., 2022, Mao et al., 27 Apr 2025).
Formal Privacy Analysis: Many protocols rely on practical or cryptographic privacy, with limited $(\epsilon, \delta)$ DP accounting; further research is needed for rigorous guarantees under composition (Kaga et al., 7 May 2024, Peng et al., 2021).
Non-IID and Heterogeneity: Non-uniform and adversarial data splits, client drift, and label conflicts remain areas for continued algorithmic innovation, for example through client clustering, federated contrastive learning, and knowledge distillation (Silva et al., 2022, Mao et al., 27 Apr 2025).
Aggregation Protocols: Secure multi-party computation, homomorphic encryption, and PIR incur computational and communication overheads; optimizing trade-offs for large-scale deployment is an ongoing concern (Tang et al., 2022, Mao et al., 27 Apr 2025).

Continued work includes extension to stronger and more flexible privacy paradigms (e.g., integrating DP and cryptographic aggregation in combination), scaling to wider data and application domains (e.g., medical, financial, legal), developing fairness- and robustness-aware protocols, and formal leakage analyses—especially in information-theoretic and adaptive settings.

7. Representative Algorithms and Protocols

Algorithm	Privacy Mechanism	Communication Style
FURL	Local parameter split	Server aggregation of federated weights, embeddings local (Bui et al., 2019)
FedCL	$\epsilon$ -LDP, clustering	Noisy embeddings for negative sampling, secure agg for gradients (Wu et al., 2022)
FedPH	Gaussian DP + THE	Class prototypes only, THE aggregation (Hangdong et al., 2023)
IPFed	Orthonormal projection	Masked class embeddings, unmask at client (Kaga et al., 7 May 2024)
SecEA	Multi-secret sharing + PIR	Embedded secret sharing, PIR retrieval (Tang et al., 2022)
DP-CVAE	DP-SGD (Gaussian)	Decoders aggregated, local encoders fixed (Salvo et al., 3 Jul 2025)
FedR	SecAgg + PSU	Relation embeddings only, masking via PSU + SecAgg (Zhang et al., 2022)
FKGE	PATE-style DP	DP-aligned embeddings via GAN and voting (Peng et al., 2021)
Feras	Embedding sharing	Aggregation server for mid-layer embeddings (Yu et al., 2022)
FedE4RAG	Homomorphic encryption	Encrypted gradient aggregation, knowledge distillation (Mao et al., 27 Apr 2025)

These protocols differ in their technical strategies but all address critical aspects of privacy, accuracy, and efficiency required for practical federated embedding learning.

Privacy-preserving federated embedding learning is foundational for secure, collaborative, deployment-ready machine learning in modern data-restricted, regulated environments. Comprehensive empirical, algorithmic, and cryptographic frameworks now underpin robust solutions spanning personalization, recommendation, graph and knowledge representation, medical imaging, and advanced language modeling. Continued advances will further integrate formal privacy analyses, scalable communication, and adversarial resilience to meet emerging challenges in cross-institutional AI and secure representation learning.