Quantile-Based Initialization in Federated Learning

• Quantile-based initialization is a technique that sets neural network biases using quantile computations to enforce sparse, controlled activations.
• The method leverages Gaussian approximations and empirical quantile estimates to ensure each neuron activates uniquely within a client batch, aiding data reconstruction.
• Extensions like PAIRS iteratively refine weight configurations for improved reconstruction recall, while countermeasures such as AGGP mitigate gradient-sparsity attacks.

Quantile-based initialization refers to techniques that set neural network bias parameters—most notably, in fully connected layers—via explicit quantile computations to control sparse activation probabilities. In the context of federated learning and private data reconstruction, quantile-based initialization is primarily used to engineer neural activations so that each neuron is likely to be activated by a unique input sample within a client batch. This bias configuration enables precise mapping between gradient rows and input samples, thereby facilitating near-perfect recovery of private data from model updates under certain attack models. The quantile-based bias initialization (QBI) method and its PAIRS extension set new empirical and theoretical reconstruction limits in the context of federated learning privacy attacks, and are counteracted by defensive frameworks such as AGGP (Nowak et al., 2024).

1. Mathematical Foundations of Quantile-Based Initialization

The core of quantile-based bias initialization (QBI) is the control of neuron activation probabilities via parameterized bias settings. For a fully connected layer $L$ with $N$ neurons (weight vectors $w_i \in \mathbb{R}^M$, biases $b_i$), the goal is to have each neuron activate on a prescribed fraction $p$ of samples in a client mini-batch $X = \{x_1, \dots, x_B\}$. The $p$-th empirical quantile $\hat{Q}_p(X)$ of a real-valued random variable $X$ is defined as

$$\hat{Q}_p(X) = \inf\{ t \in \mathbb{R} : \hat{F}_X(t) \geq p \}$$

where $N$0 is the empirical cumulative distribution function.

For gradient-based data leakage attacks, the critical requirement is for each neuron to activate on exactly one of $N$1 samples with high probability ($N$2). Under a Gaussian approximation, assuming weights and inputs are zero-mean and independent with $N$3, the pre-activation $N$4 is approximately $N$5. The bias $N$6 yielding activation probability $N$7 is

$N$8

where $N$9 is the standard normal CDF.

Alternatively, given calibration data $w_i \in \mathbb{R}^M$0, the bias can be empirically set as

$w_i \in \mathbb{R}^M$1

so that a fraction $w_i \in \mathbb{R}^M$2 of activations are positive.

As batch size $w_i \in \mathbb{R}^M$3 increases, the target $w_i \in \mathbb{R}^M$4 shrinks, leading to sparser activations and increased likelihood of unique neuron-sample associations.

2. Algorithmic Procedures for QBI

The QBI algorithm is executed server-side prior to each federated learning round. The procedure is as follows:

• For each neuron $w_i \in \mathbb{R}^M$5:
  • Initialize $w_i \in \mathbb{R}^M$6.
  • Set $w_i \in \mathbb{R}^M$7 for anticipated batch size $w_i \in \mathbb{R}^M$8.
  • If no calibration data, use theoretical bias $w_i \in \mathbb{R}^M$9.
  • If calibration data $b_i$0 of size $b_i$1 are available, set $b_i$2.

Computational cost is $b_i$3 for weights, and bias initialization is $b_i$4 (theoretical) or $b_i$5 (empirical). In practice, using the closed form formula incurs negligible overhead beyond standard random initialization.

3. Theoretical Limits of Gradient-Sparsity-Based Attacks

Closed-form expectations for extraction metrics under ideal QBI conditions have been derived. For batch size $b_i$6:

• Active neuron fraction: $b_i$7
• Extraction-precision: $b_i$8
• With $b_i$9 neurons, expected extraction-recall is

$p$0

As $p$1, $p$2, $p$3. These formulas set upper bounds for any gradient-sparsity attack under normality assumptions, quantifying how many samples can be isolated and perfectly reconstructed per batch.

4. PAIRS: Pattern–Aware Iterative Random Search Extension

Real data distributions often deviate from idealized i.i.d. Gaussian assumptions, resulting in suboptimal sparsity patterns for QBI bias settings. The PAIRS extension (Pattern–Aware Iterative Random Search) improves upon QBI by leveraging an auxiliary dataset to fine-tune weights. In this approach, for each group of $p$4 neurons, the goal is to configure weight rows so that every sample in a size-$p$5 auxiliary batch is uniquely isolated. The algorithm iteratively refines weight vectors using random restarts, freezing neurons that achieve the desired unique activation criterion. PAIRS increases perfect reconstruction recall $p$6 by 5–10 percentage points over QBI.

5. Empirical Performance on Benchmark Tasks

QBI and PAIRS have demonstrated substantial improvements in reconstruction rates over previous methods ("trap weights", Boenisch et al. 2023) on several standard datasets.

Dataset	(N, B)	Trap (%)	QBI (%)	PAIRS (%)
ImageNet	(200, 20)	35.5	82.5 ± 2.4	85.5 ± 1.3
ImageNet	(200, 50)	30.4	52.0 ± 1.5	56.0 ± 1.1
ImageNet	(200,100)	24.0	29.0 ± 0.9	34.6 ± 0.7
ImageNet	(200,200)	11.3	15.1 ± 0.6	19.5 ± 0.6
IMDB	20	100.0	100.0 ± 0.0	99.9 ± 0.2
IMDB	50	96.2	98.9 ± 0.5	98.6 ± 0.4
IMDB	100	65.4	90.5 ± 0.3	90.8 ± 0.8
IMDB	200	25.5	72.8 ± 0.6	73.3 ± 0.8

On ImageNet with $p$7, QBI achieves recall $p$8 of $p$9 vs. $X = \{x_1, \dots, x_B\}$0 for trap weights. On IMDB, QBI reaches $X = \{x_1, \dots, x_B\}$1 recall at $X = \{x_1, \dots, x_B\}$2, improving over $X = \{x_1, \dots, x_B\}$3 for prior methods.

6. Defensive Countermeasures: AGGP

Activation-based Greedy Gradient Pruning (AGGP) is a framework designed to defend against QBI-enabled gradient sparsity attacks. AGGP computes the number of samples $X = \{x_1, \dots, x_B\}$4 activating each neuron $X = \{x_1, \dots, x_B\}$5 per batch; neurons with $X = \{x_1, \dots, x_B\}$6 (threshold $X = \{x_1, \dots, x_B\}$7) are considered potential isolators, and their gradients are selectively pruned. The pruning probability is set as

$X = \{x_1, \dots, x_B\}$8

with $X = \{x_1, \dots, x_B\}$9, $p$0. Empirical evaluation demonstrates that AGGP reduces perfect reconstruction recall $p$1 to zero in all test conditions while maintaining typical validation accuracy within $p$2 percentage point, illustrating its efficacy as a defense mechanism.

7. Context and Broader Impact

Quantile-based initialization, as implemented in QBI and PAIRS, has established new performance limits for batch-wise reconstruction attacks in federated learning, raising both the offensive and defensive stakes in privacy-preserving machine learning. The explicit mathematical connection between quantile bias settings, activation sparsity, and recoverability provides a rigorous framework for future advances in both attack methods and countermeasures. The integration of empirical quantile computations, Gaussian approximations, and random search corrections reflects an overview of distributional modeling and combinatorial optimization tailored to privacy-sensitive settings (Nowak et al., 2024).