Formal privacy guarantees for attention freezing

Establish whether freezing the attention parameters in GPT-2-class transformer models satisfies formal $(\varepsilon,\delta)$-differential privacy guarantees and determine how attention freezing composes with DP-SGD.

Background

AEGIS freezes all attention projection parameters to eliminate the attention-gradient subspace exploited by analytical gradient-inversion attacks. The paper provides an exact structural guarantee that the exported attention gradients are zero, but this is not a differential-privacy guarantee.

The conclusion explicitly identifies the relationship between attention freezing and formal differential privacy, including composition with DP-SGD, as unresolved. Such a result would connect AEGIS’s channel-specific masking guarantees to standard privacy accounting frameworks.

References

Open questions include whether attention freezing satisfies formal $(\varepsilon,\delta)$-DP guarantees, how it composes with DP-SGD, and how the single-step calibration extends to multi-step FedAvg.

— AEGIS: Attention-Embedding Gradient Isolation Shield - Triple-Channel Gradient Masking for Privacy-Preserving Federated LLM Fine-Tuning  (2608.19534 - Tao et al., 20 Aug 2026) in Section 6, Conclusion