Squeezing-Heads Distillation: Quantum & Neural Methods

• Squeezing-Heads Distillation is a unified protocol that enhances quantum optics and neural transformer architectures using specialized distillation techniques.
• In quantum optics, it employs non-Gaussian operations like photon subtraction and heralded Gaussification to purify and amplify squeezed states and multipartite entanglement.
• In machine learning, SHD applies convex combinations to compress teacher attention maps, facilitating efficient knowledge transfer without additional architecture overhead.

Squeezing-Heads Distillation (SHD) encompasses several protocols and variants for the distillation and purification of squeezed states of light, multipartite continuous-variable entangled states, and knowledge transfer in Transformer-based neural architectures. In quantum optics, SHD refers primarily to non-Gaussian resource distillation via selective photon subtraction, displacement operations, and heralded Gaussification to achieve strengthened squeezing and state purification, including multipartite settings. In machine learning, SHD designates a method for compressing, aligning, and transferring multi-head attention in neural transformers irrespective of head-count mismatch, thus enabling flexible, efficient knowledge distillation. Below, both domains are addressed according to the principal research findings.

1. SHD Protocols in Quantum Optics: Squeezed State Distillation and Purification

In quantum optics, SHD targets the enhancement and purification of single-mode and multipartite squeezed states, which are indispensable for quantum information and quantum sensing (Fiurášek et al., 1 Feb 2025).

1.1 Protocol Steps

• De-Gaussification via Modified Two-Photon Subtraction:

Begins with tapping a fraction of an input single-mode squeezed vacuum $\hat S(r_{\rm in})|0\rangle$ on a beam splitter (BS$_1$). The output is interfered with a weak coherent state on BS$_2$, and exactly one photon is subtracted from both output ports. This implements a non-Gaussian filter, $$\hat M \propto \hat D(-\delta) \hat a^2 \hat D(\delta) = \hat a^2 - \delta^2$$, with tunable displacement $\delta$:

$$\hat M \propto \hat D(-\delta)\hat a^2\hat D(\delta) = \hat a^2 - \delta^2$$

• Heralded Gaussification:

Two identical copies of the previously filtered state are interfered on a balanced beam splitter followed by projection onto vacuum:

$$\hat\rho' = \langle 0| U_{\rm BS} (\hat\rho_{\rm NG} \otimes \hat\rho_{\rm NG}) U_{\rm BS}^\dagger |0\rangle$$

Iterative repetitions further distill a squeezed Gaussian state.

• Alternative De-Gaussification by Fock-State Filtering:

The filter $\hat F_1 = \hat I - |1\rangle\langle 1|$ eliminates the single-photon component, possibly using photon catalysis or operator superpositions. Subsequent iterative Gaussification can yield pure squeezed states even from mixed initial conditions.

1.2 Covariance and Squeezing Formulae

After de-Gaussification, the squeezed and anti-squeezed quadrature variances become:

$$V_X = e^{2r_{\rm in}}\left[1 + 4\sinh^2 r_{\rm in} \frac{2\sinh^2 r_{\rm in}+\cosh r_{\rm in}\sinh r_{\rm in}-\delta^2}{2\sinh^4 r_{\rm in}+(\cosh r_{\rm in}\sinh r_{\rm in}-\delta^2)^2}\right]$$

$$V_Y = e^{-2r_{\rm in}}\left[1 + 4\sinh^2 r_{\rm in} \frac{2\sinh^2 r_{\rm in}-\cosh r_{\rm in}\sinh r_{\rm in}+\delta^2}{2\sinh^4 r_{\rm in}+(\cosh r_{\rm in}\sinh r_{\rm in}-\delta^2)^2}\right]$$

Optimizing the displacement yields

$_1$0

The final squeeze parameter $_1$1 after one Gaussification step is

$_1$2

1.3 Performance and Limitations

• Success probability:

Depends on the beam splitter transmittance $_1$3 and displacement $_1$4:

$_1$5

For typical parameters, $_1$6 lies in $_1$7 to $_1$8 range, as detailed in the data table.

• Loss and mixed states:

SHD using two-photon subtraction plus Gaussification cannot remediate pre-existing transmission loss, which strictly limits output fidelity.

• Regime of strong distillation:

Arbitrarily strong squeezing enhancement is theoretically possible as $_1$9 for small $_2$0, at the expense of vanishing success probability (Fiurášek et al., 1 Feb 2025).

2. Multipartite Continuous-Variable SHD: Local Squeezing with Single Photon Subtraction

Song Yang et al. extended SHD to multipartite entangled states, obviating the exponential decay in success probability for Opatrný-style photon subtraction (Yang et al., 2011).

2.1 Protocol Steps

• Single photon subtraction on one mode:

Instead of performing $_2$1 local photon subtractions for $_2$2-mode states, only a single mode is photon-subtracted, with all modes locally squeezed using symplectic transforms $_2$3.

• Measurement and heralding:

The heralded state post-measurement is a non-Gaussian mixture:

$_2$4

with mathematical details linked to the covariance matrices post beam-splitter and measurement.

• Success probability:

$_2$5

Crucially, $_2$6 stays constant ($_2$7) regardless of $_2$8.

2.2 Entanglement Enhancement

• Logarithmic negativity:

Quantifies entanglement gain:

$_2$9

For $$\hat M \propto \hat D(-\delta) \hat a^2 \hat D(\delta) = \hat a^2 - \delta^2$$0 modes, local squeezing optimized at $$\hat M \propto \hat D(-\delta) \hat a^2 \hat D(\delta) = \hat a^2 - \delta^2$$1 increases $$\hat M \propto \hat D(-\delta) \hat a^2 \hat D(\delta) = \hat a^2 - \delta^2$$2 over input entanglement.

2.3 N-Mode Transfer Theorem

A closed-form expression connects the Gaussian state’s phase-space representation to its Fock basis elements:

$$\hat M \propto \hat D(-\delta) \hat a^2 \hat D(\delta) = \hat a^2 - \delta^2$$3

where $$\hat M \propto \hat D(-\delta) \hat a^2 \hat D(\delta) = \hat a^2 - \delta^2$$4 is a Gaussian function of squeezing-dependent covariance matrices.

3. Comparative Analysis: SHD and Non-Gaussian Probabilistic Operations

Chandan Kumar’s work characterizes squeezing distillation using photon subtraction (PS), photon addition (PA), and photon catalysis (PC) (Kumar, 2023):

• Photon subtraction and catalysis:

Two-photon processes (PS, PC) can enhance squeezing; single-photon subtraction or addition never does.

• Operator description:

Each operation is realized via conditional Kraus maps after beam-splitter interaction, using heralded detection events.

• Squeezing parameter:

For two-photon subtraction ($$\hat M \propto \hat D(-\delta) \hat a^2 \hat D(\delta) = \hat a^2 - \delta^2$$5):

$$\hat M \propto \hat D(-\delta) \hat a^2 \hat D(\delta) = \hat a^2 - \delta^2$$6

For two-photon catalysis ($$\hat M \propto \hat D(-\delta) \hat a^2 \hat D(\delta) = \hat a^2 - \delta^2$$7), similar rational expressions with improved squeezing for small $$\hat M \propto \hat D(-\delta) \hat a^2 \hat D(\delta) = \hat a^2 - \delta^2$$8.

• Success probability:

For 2-PS and 2-PC, formulae are specified, with optimal values balancing improved squeezing and non-negligible heralding rates.

4. SHD in Neural Architectures: Multi-Head Attention Distillation

Squeezing-Heads Distillation also designates a knowledge distillation protocol for transformer-based neural networks that compresses multi-head attention (Bing et al., 11 Feb 2025).

4.1 Mathematical Formulation

• Head compression by convex combination:

Teacher heads $$\hat M \propto \hat D(-\delta) \hat a^2 \hat D(\delta) = \hat a^2 - \delta^2$$9 are combined into one compressed attention map $\delta$0 via linear interpolation with optimized $\delta$1:

$\delta$2

$\delta$3 is chosen to minimize Frobenius-norm distortion between value-propagation of the compressed and original head sets.

• Training loss:

KL-divergence is used between temperature-softened teacher and student attention maps:

$\delta$4

4.2 Computational Efficiency and Practical Advantages

• Complexity:

SHD’s convex combination is $\delta$5 per attention map, matching native self-attention costs.

• No extra parameters or architecture modifications:

The student model need not match the teacher’s head count nor include projection modules.

• Empirical performance:

SHD delivers consistent improvements across vision and language tasks, outperforming baseline distillation and feature-aligning methods, with demonstrable gains in FID, IS, ROUGE, and accuracy.

Domain	Key Protocol/Mechanism	Distillation/Compression Method
Quantum Optics	Squeezing enhancement, purification	Two-photon subtraction + Gaussification, Fock filters
Quantum Optics (Multipartite)	Entanglement gain, $\delta$6-mode stability	Local squeezing + single PS
Machine Learning	Attention map compression	Linear convex combinations, project-free, architecture-agnostic

5. Significance and Outlook

SHD unifies a class of resource distillation protocols in quantum optics and neural computation, addressing previously unsolved scalability, loss-resilience, and architectural-alignment barriers. In quantum optics, SHD protocols systematically enable squeezing and multipartite entanglement distillation with high heralding probabilities and explicit analytic connections to state transfer and purification (Fiurášek et al., 1 Feb 2025, Yang et al., 2011, Kumar, 2023). The transfer theorem augments the analytical tractability of non-Gaussian outputs for practical implementation. In neural architectures, SHD bridges head-count and attention map alignment without resource overhead or loss of fine-grained knowledge, confirmed by strong empirical results (Bing et al., 11 Feb 2025).