Fisher-Guided Token Selection (FGTS)

• FGTS is a mechanism that uses token-level Fisher information to quantify sensitivity, enabling dynamic and importance-aware token selection in federated learning.
• It computes an EMA-stabilized Fisher proxy to generate top-K token masks and adapt mixed-precision quantization, thus reducing uplink communication and energy usage.
• Empirical results show substantial uplink reduction (e.g., 46×) and improved time-to-accuracy, making FGTS ideal for resource-constrained edge deployments.

Fisher-Guided Token Selection (FGTS) is a principled mechanism for communication-efficient adaptation of LLMs in federated learning settings, specifically under resource constraints typical of edge deployments. FGTS employs a lightweight Fisher information proxy to estimate token-level sensitivity, enabling dynamic importance-aware selection and quantization of tokens during training and inference. Integrated as a drop-in primitive within parameter-efficient fine-tuning (PEFT) pipelines, such as LoRA, FGTS achieves substantial uplink reduction and energy efficiency, while preserving or improving model quality relative to baseline approaches (Li et al., 28 Apr 2026).

1. Fisher Proxy for Token Sensitivity

FGTS leverages ideas from information geometry, where the classical Fisher Information Matrix (FIM)

$$F(\theta) = \mathbb{E}_{x, y \sim D} \left[ \nabla_{\theta} \ell(x, y; \theta) \nabla_{\theta} \ell(x, y; \theta)^{\top} \right]$$

provides a measure of parameter sensitivity. Intractable for large LLMs, FGTS adopts the diagonal surrogate

$$\hat{F}(j) = \mathbb{E} \left[ \left( \frac{\partial \ell}{\partial \theta_j} \right)^2 \right]$$

and extends this concept to tokens. For a local minibatch step $s$ on client $k$, with $L$ input tokens $t_1, \ldots, t_L$ (embeddings $e_i \in \mathbb{R}^{d_e}$), FGTS defines the instantaneous per-token Fisher proxy as

$g_{k,s}(i) := \| \nabla_{e_i} \ell_{k,s} \|_2^2$

where $\ell_{k,s}$ is the sequence loss (e.g., cross-entropy). This metric directly quantifies the sensitivity of the loss to each input token, measuring how strongly perturbations in the embedding of token $i$ would affect the model's output loss. This Fisher proxy at the token level provides a data-driven importance score, rather than depending on heuristic criteria such as token frequency or attention.

2. On-Device Computation and Stabilization

On each client, the FGTS token sensitivity measure is periodically updated and stabilized for robust selection. At every local minibatch step:

• The forward pass computes $$\hat{F}(j) = \mathbb{E} \left[ \left( \frac{\partial \ell}{\partial \theta_j} \right)^2 \right]$$0.
• During the backward pass, token gradients $$\hat{F}(j) = \mathbb{E} \left[ \left( \frac{\partial \ell}{\partial \theta_j} \right)^2 \right]$$1—already present in the Transformer backpropagation—are recorded.
• The Fisher proxy $$\hat{F}(j) = \mathbb{E} \left[ \left( \frac{\partial \ell}{\partial \theta_j} \right)^2 \right]$$2 for each token $$\hat{F}(j) = \mathbb{E} \left[ \left( \frac{\partial \ell}{\partial \theta_j} \right)^2 \right]$$3 is computed with $$\hat{F}(j) = \mathbb{E} \left[ \left( \frac{\partial \ell}{\partial \theta_j} \right)^2 \right]$$4 overhead per token.
• To stabilize noisy per-minibatch estimates, an exponential moving average (EMA) is maintained:

$$\hat{F}(j) = \mathbb{E} \left[ \left( \frac{\partial \ell}{\partial \theta_j} \right)^2 \right]$$5

with typical decay parameter $$\hat{F}(j) = \mathbb{E} \left[ \left( \frac{\partial \ell}{\partial \theta_j} \right)^2 \right]$$6. Storage overhead remains minimal (one scalar per token), and compute cost is negligible relative to full backpropagation.

3. Importance-Aware Token Selection and Quantization

FGTS executes a two-stage importance-driven compression process:

3.1 Token Keep/Drop Criterion

At fixed intervals—every $$\hat{F}(j) = \mathbb{E} \left[ \left( \frac{\partial \ell}{\partial \theta_j} \right)^2 \right]$$7 steps (e.g., $$\hat{F}(j) = \mathbb{E} \left[ \left( \frac{\partial \ell}{\partial \theta_j} \right)^2 \right]$$8)—a binary token mask $$\hat{F}(j) = \mathbb{E} \left[ \left( \frac{\partial \ell}{\partial \theta_j} \right)^2 \right]$$9 is constructed by selecting the top-K tokens according to their stabilized Fisher scores:

$s$0

Here, $s$1 denotes the retained fraction. Only tokens with the highest empirical Fisher importance drive subsequent gradient-based adaptation.

3.2 Mixed-Precision Quantization

Following masked training, parameter-level Fisher importance is accumulated, enabling adaptive quantization:

• For each PEFT parameter coordinate $s$2 (e.g., LoRA update direction), compute Fisher-weighted signal:

$s$3

• Bit width $s$4 for each coordinate is assigned by thresholding $s$5 according to percentiles (with bit set $s$6):

$s$7

• Uniform quantization per group uses a per-group scaling factor:

$s$8

and quantized values are computed by clipping and rounding.

3.3 FGTS Client Update Algorithm

FGTS client-side token selection and quantization is summarized in the following key steps:

1. Maintain and update token-level EMA Fisher proxies.
2. Periodically generate token masks by top-K selection.
3. Perform masked local training using selected tokens.
4. Accumulate parameter-level Fisher proxies based on masked gradients.
5. Allocate bits for quantization based on parameter importance.
6. Pack and transmit sparse, mixed-precision updates using compact encodings, subject to uplink budget.

No modifications are required in the server aggregator, and the masking/quantization are performed entirely client-side (Li et al., 28 Apr 2026).

4. Integration with Federated PEFT (e.g., LoRA)

FGTS is architected as a model- and optimizer-agnostic module that fits into existing federated PEFT pipelines:

• Local adaptation loop: Token masks affect which token losses contribute to local parameter updates, focusing adaptation on the empirically most salient tokens per client.
• Sparse, mixed-precision message construction: At the conclusion of local adaptation, only coordinates with assigned bit width $s$9 are transmitted.
• Server aggregation: Standard FedAvg is applied to the dequantized, potentially sparse updates, requiring no changes in server-side code, secure aggregation, or DP infrastructure.
• Bandwidth heterogeneity: FGTS enables clients under differing uplink budgets to transmit messages with varying sparsity and granularity, minimizing straggler effects in realistic mobile environments.

5. Empirical Results in Non-IID Federated Adaptation

Experiments conducted on non-IID real-world FL benchmarks demonstrate the benefit of FGTS:

Task/Dataset	FL Setting	Key Result (vs. uncompressed FedAvg+LoRA)
Fed-Aya	Multilingual QA, α=0.1	46× uplink reduction, 52% faster time-to-accuracy
Fed-Med	Medical QA, α=0.1	Uplink & speed gains; downstream QA quality maintained
Fed-Code	Code generation, rare-tokens	Reliable rare-token signal preservation

Other quantitative outcomes:

• 6.8× round time speedup ($k$0 s from $k$1 s) on Jetson Nano with 4G LTE (20 Mbps).
• $k$2100 J energy/round versus $k$3600 J for uncompressed; transmit energy is dominant in the energy profile.
• Inference on Jetson devices accelerated by up to 1.55× via reuse of token mask for pruning.

Reliability indicators:

• Token recall: $k$4 for FGTS compared to $k$5 for attention-based heuristics.
• Downstream quality (e.g., ROUGE-L, METEOR) is on par or improved relative to baselines.

6. Extensions and Future Applications

FGTS enables several additional avenues:

• Standalone inference acceleration: The learned token saliency (EMA Fisher scores) enables token pruning or low-fidelity processing during inference on resource-limited edge devices.
• Quantizer generalization: Combination with non-uniform quantization (e.g., GPTQ, SmoothQuant) is possible, enabling finer control at the bit level subject to increased side-information.
• Asynchronous and partially synchronous FL: FGTS can extend to settings with client staleness and partial synchronization, where bit allocation must be adapted to staleness profiles.
• Secure/private aggregation: Fisher-guided masking enables minimization of metadata revealed in encrypted or shuffled updates, preserving semantic fidelity.

7. Conceptual Significance

FGTS reframes the Fisher information proxy as a token-level communication control primitive within distributed LLM adaptation. The mechanism dynamically allocates communication and computation resources to the most loss-sensitive tokens, tightly coupling information-theoretic importance estimation with efficiency constraints. FGTS thus enables practical and high-fidelity federated fine-tuning and inference acceleration on edge devices, with no required adjustment to server-side aggregation protocols (Li et al., 28 Apr 2026).