RidgeFT: Exact Federated Fine-Tuning

• RidgeFT is a framework for federated fine-tuning that leverages a frozen feature extractor and ridge regression head by maintaining additive sufficient statistics.
• It updates the model via fixed-size client messages (ΔS and ΔT) that ensure deterministic, order- and partition-invariant retraining, matching centralized solutions.
• The method is computationally efficient and validated on benchmarks, with near-machine precision and a Bayesian zero-KL certificate between federated and centralized learning.

RidgeFT, in the sense formalized by "Exact Federated Continual Unlearning for Ridge Heads on Frozen Foundation Models" (Quan et al., 13 Mar 2026), denotes ridge-based fine-tuning on frozen foundation-model features, together with an exact federated continual unlearning protocol for the resulting ridge head. The setting is a frozen, deterministic feature extractor $\phi$ paired with a small trainable linear head $W$, trained by ridge regression on private, user-generated data distributed across clients. In that regime, the global ridge optimum depends on the data only through two additive sufficient statistics, which makes it possible to support an arbitrary stream of add and delete requests via fixed-size client messages and to maintain, in exact arithmetic, a head that is pointwise identical to centralized retraining after every request (Quan et al., 13 Mar 2026).

1. Formal setting and ridge objective

The model assumed by RidgeFT is a frozen foundation model used as a deterministic feature extractor $\phi$, together with a ridge-regression head. If $n$ is the number of training samples, $d$ the feature dimension, and $k$ the output dimension, then the feature matrix is $X \in \mathbb{R}^{n \times d}$ with rows $x_i = \phi(\text{input}_i)$, the label matrix is $Y \in \mathbb{R}^{n \times k}$ with rows $y_i$, and the head is $W$0.

The optimization problem is the standard multi-output ridge objective

$W$1

with $W$2, called $W$3 in the paper. Its closed-form optimum is

$W$4

This formulation applies identically to multi-class classification, regression, and multi-task regression. For multi-class classification, $W$5 can be one-hot or soft, with $W$6 classes, and the ridge head trains $W$7 columns jointly, each as a regularized least-squares task sharing the feature Gram. For regression and multi-task regression, $W$8 is the number of targets and the same formulas apply. An intercept is handled by augmenting each feature $W$9 with a $\phi$0, increasing $\phi$1 by $\phi$2 (Quan et al., 13 Mar 2026).

2. Additive sufficient statistics and the source of exactness

The central structural fact behind RidgeFT is that ridge training depends only on two additive sufficient statistics:

$\phi$3

The optimum can therefore be written as

$\phi$4

Once $\phi$5 are known, raw samples are unnecessary for optimization. This is the basis of the unlearning protocol. For a single sample $\phi$6, the add/delete contributions are

$\phi$7

For a client batch $\phi$8, they become

$\phi$9

The server maintains a ledger

$n$0

Because $n$1 and $n$2 are sums, addition and subtraction commute and associate. The final $n$3 are therefore independent of the order of client reports, the partitioning of data across clients, and the interleaving of add/delete events, as long as the final retained multiset of samples is the same.

A key implication is that second-order information is indispensable. Two batches can share first moments $n$4 and $n$5 yet have different Grams $n$6; since $n$7 depends on $n$8, any protocol lacking $n$9 cannot be exact for all batches. This sharply distinguishes RidgeFT from approximate federated unlearning methods that target general deep networks but cannot reduce the problem to additive sufficient statistics (Quan et al., 13 Mar 2026).

3. Federated continual add/delete protocol

The operational protocol is client-side feature extraction plus server-side sufficient-statistic maintenance. Each client computes frozen features locally and, for every add or delete request, transmits fixed-size sufficient-statistic messages $d$0 together with minimal metadata such as request identifiers. Message size depends on $d$1 and $d$2, not on the number of local samples.

For large batches, clients may compute a thin QR factorization of a local feature matrix $d$3, with $d$4 orthonormal and $d$5 upper-triangular, and send $d$6 because $d$7; they still send $d$8. The server aggregates client messages, updates $d$9 and $k$0 by the ledger rules, and computes

$k$1

In exact arithmetic, this equals centralized retraining on the retained dataset at each time $k$2.

Two server-side realizations are described.

Variant	State and update rule	Notes
Variant A (Exact SPD solve)	Maintain $k$3, $k$4, form $k$5, compute Cholesky $k$6, solve $k$7	Numerically robust; recommended baseline
Variant B (Incremental inverse tracker via Sherman–Morrison–Woodbury)	Maintain $k$8 and current $k$9; apply low-rank adds/downdates using SMW	Efficient for rank-$X \in \mathbb{R}^{n \times d}$0 updates; requires feasibility checks and periodic reset

Variant A stores $X \in \mathbb{R}^{n \times d}$1, $X \in \mathbb{R}^{n \times d}$2, and optionally the Cholesky factor $X \in \mathbb{R}^{n \times d}$3. Cholesky can be recomputed each round, although rank-one or rank-$X \in \mathbb{R}^{n \times d}$4 updates and downdates are possible. Using $X \in \mathbb{R}^{n \times d}$5 ensures $X \in \mathbb{R}^{n \times d}$6, and fp64 server-side computation improves conditioning and reduces floating-point drift, especially when accumulating second-order statistics.

Variant B writes $X \in \mathbb{R}^{n \times d}$7 with $X \in \mathbb{R}^{n \times d}$8 and updates the inverse by Sherman–Morrison–Woodbury. For adds,

$X \in \mathbb{R}^{n \times d}$9

followed by an update of $x_i = \phi(\text{input}_i)$0 using the new inverse and $x_i = \phi(\text{input}_i)$1. For deletes,

$x_i = \phi(\text{input}_i)$2

which requires $x_i = \phi(\text{input}_i)$3, equivalently $x_i = \phi(\text{input}_i)$4. In finite precision, repeated downdates can accumulate drift; the prescribed safeguard is to reset by recomputing through Variant A whenever the downdate becomes ill-conditioned or failure is detected (Quan et al., 13 Mar 2026).

4. Deterministic guarantees and Bayesian interpretation

The paper’s core guarantee is deterministic retrain-equivalence. After any sequence of add/delete requests, the server’s head equals the centralized ridge solution on the resultant dataset:

$x_i = \phi(\text{input}_i)$5

where $x_i = \phi(\text{input}_i)$6 are the exact sufficient statistics of the retained multiset. The proof is straightforward in structure: ridge depends only on $x_i = \phi(\text{input}_i)$7, the ledger updates maintain $x_i = \phi(\text{input}_i)$8 exactly, Variant A computes the exact solution from those statistics, and Variant B maintains the exact inverse via SMW in exact arithmetic.

Order invariance and partition invariance follow from the same algebraic structure. For a fixed final retained dataset, $x_i = \phi(\text{input}_i)$9 is independent of the sequence of add/delete requests and independent of how data are partitioned across clients, because $Y \in \mathbb{R}^{n \times k}$0 depend only on the final multiset.

The paper also gives a Bayesian certificate of zero KL divergence. Ridge is interpreted as MAP estimation under a Gaussian prior and Gaussian likelihood:

• $Y \in \mathbb{R}^{n \times k}$1, equivalently $Y \in \mathbb{R}^{n \times k}$2;
• $Y \in \mathbb{R}^{n \times k}$3;
• $Y \in \mathbb{R}^{n \times k}$4.

The posterior is matrix-normal,

$Y \in \mathbb{R}^{n \times k}$5

with

$Y \in \mathbb{R}^{n \times k}$6

Because the protocol maintains $Y \in \mathbb{R}^{n \times k}$7 exactly, both the posterior mean and covariance match centralized retraining, so the two Gaussians are identical and the KL divergence is zero. This does not merely certify equal point estimates; it certifies equality of the full Gaussian posterior induced by the ridge model (Quan et al., 13 Mar 2026).

5. Computational profile and empirical validation

The server-side cost depends on the chosen variant. Variant A forms $Y \in \mathbb{R}^{n \times k}$8 and computes its Cholesky factorization once per request in $Y \in \mathbb{R}^{n \times k}$9 time, then solves $y_i$0 in $y_i$1 time. Variant B, for rank-$y_i$2 per-round updates, incurs $y_i$3 to update the inverse and $y_i$4 to update $y_i$5, with smaller constants when using reuse and block updates.

Communication is fixed-size per request. With direct statistics, each client sends $y_i$6 and $y_i$7, for message size $y_i$8 independent of the number of local samples. With QR-based compression, clients send $y_i$9 such that $W$00, plus $W$01, for message size $W$02. When $W$03, this substantially reduces communication. Each request completes in a single round with fixed-size messages and closed-form updates, avoiding multi-round FedAvg coordination and per-client local optimization.

Experiments were conducted on CIFAR-10, CIFAR-100, FeMNIST, and Sentiment140, under non-IID client partitions, using DINOv2-ViT B/14 for image tasks and RoBERTa (TweetEval) for Sentiment140, with $W$04 in both cases. Both Variant A and Variant B match centralized ridge retraining in accuracy across all four benchmarks and produce weights pointwise identical up to floating-point error. The reported relative Frobenius deviation

$W$05

is approximately $W$06 in fp64 on FeMNIST, with comparable behavior on the other datasets. Using fp32 for second-order accumulation yields approximately $W$07 to $W$08, which confirms the benefit of fp64 for $W$09 aggregation and solves.

In continual deletion experiments, 200 single-point deletions incur small, consistent per-request latency and are orders of magnitude faster than FedAvg retraining, while also being faster overall than Exact-Fun and FATS, which occasionally retrain. For repeated 20% chunk deletions, the variants remain near the centralized ridge baseline and outperform FedAvg-based baselines that require multiple rounds. In a continual add-back test, after deleting 200 single points and then re-adding them, the model returns to the original $W$10 with deviations approximately $W$11. Regularization through $W$12 ensures SPD and conditioning; larger $W$13 improves stability, and tuning $W$14 is straightforward because the solution is closed-form (Quan et al., 13 Mar 2026).

6. Assumptions, limitations, and relation to other uses of the term

RidgeFT’s exactness depends on a narrow but practically important regime. The feature extractor $W$15 must be frozen and deterministic, running in evaluation mode with no dropout. If $W$16 is stochastic, or if data are unavailable for recomputation, clients must cache feature vectors at add time to ensure exact deletes. The trainable head must be linear and trained by ridge regression; the guarantees hinge on the closed form and the sufficiency of $W$17. Exactness is therefore exactness in arithmetic structure, with fp64 yielding near-machine-precision equivalence in practice rather than symbolic exactness under arbitrary floating-point execution.

The privacy posture is limited but explicit. $W$18 and $W$19 are aggregate second-order statistics; they do not expose raw inputs, but they can leak distributional information, especially at large $W$20. The stated mitigation is to combine the protocol with secure aggregation and, where needed, differential privacy on statistics. Applicability is broad within the ridge-head regime: one-vs-rest or multinomial ridge with $W$21 classes, soft labels, multi-label heads, and regression or multi-task heads all fit directly. The main trade-off is scope: the backbone must remain frozen, and unlearning inside the backbone is orthogonal and remains challenging (Quan et al., 13 Mar 2026).

The name “RidgeFT” is not fully standardized across the literature. In lifelong machine-generated text attribution, "When New Generators Arrive: Lifelong Machine-Generated Text Attribution via Ridge Feature Transfer" uses RidgeFT for a replay-free closed-form update framework built around a frozen encoder, covariance calibration, fixed random features, and class-wise sufficient statistics (Sun et al., 4 Jun 2026). In heterogeneous federated learning, "Accelerating Heterogeneous Federated Learning with Closed-form Classifiers" describes Fed3R+FT, a ridge-regression-initialized fine-tuning pipeline on fixed pretrained features, which the detailed synthesis explicitly identifies as RidgeFT (Fanì et al., 2024). In few-shot class-incremental audio classification, the multi-level embedding extractor plus ridge regression classifier is described as, in essence, RidgeFT for audio (Si et al., 23 Jun 2025). This suggests that “RidgeFT” functions less as a single canonical acronym than as a family resemblance: frozen or stabilized feature extraction paired with analytic ridge updates. Within that family, the 2026 federated continual unlearning formulation is distinguished by exact retrain-equivalence, order and partition invariance, and a Bayesian zero-KL certificate (Quan et al., 13 Mar 2026).