---
title: 'RidgeFT: Exact Federated Fine-Tuning'
url: https://www.emergentmind.com/topics/ridgeft
type: topic
---

# RidgeFT: Exact Federated Fine-Tuning

RidgeFT, in the sense formalized by "Exact Federated Continual Unlearning for Ridge Heads on Frozen Foundation Models" [2603.12977], denotes ridge-based fine-tuning on frozen foundation-model features, together with an exact federated continual unlearning protocol for the resulting ridge head. The setting is a frozen, deterministic feature extractor $\phi$ paired with a small trainable linear head $W$, trained by ridge regression on private, user-generated data distributed across clients. In that regime, the global ridge optimum depends on the data only through two additive sufficient statistics, which makes it possible to support an arbitrary stream of add and delete requests via fixed-size client messages and to maintain, in exact arithmetic, a head that is pointwise identical to centralized retraining after every request [2603.12977].

## 1. Formal setting and ridge objective

The model assumed by RidgeFT is a frozen foundation model used as a deterministic feature extractor $\phi$, together with a ridge-regression head. If $n$ is the number of training samples, $d$ the feature dimension, and $k$ the output dimension, then the feature matrix is $X \in \mathbb{R}^{n \times d}$ with rows $x_i = \phi(\text{input}_i)$, the label matrix is $Y \in \mathbb{R}^{n \times k}$ with rows $y_i$, and the head is $W \in \mathbb{R}^{d \times k}$.

The optimization problem is the standard multi-output ridge objective
$$
J(W) = \|XW - Y\|_F^2 + \lambda \|W\|_F^2,
$$
with $\lambda > 0$, called $\gamma$ in the paper. Its closed-form optimum is
$$
W^* = (X^\top X + \lambda I_d)^{-1} X^\top Y.
$$

This formulation applies identically to multi-class classification, regression, and multi-task regression. For multi-class classification, $Y$ can be one-hot or soft, with $k$ classes, and the ridge head trains $c = k$ columns jointly, each as a regularized least-squares task sharing the feature Gram. For regression and multi-task regression, $k$ is the number of targets and the same formulas apply. An intercept is handled by augmenting each feature $x_i$ with a $1$, increasing $d$ by $1$ [2603.12977].

## 2. Additive sufficient statistics and the source of exactness

The central structural fact behind RidgeFT is that ridge training depends only on two additive sufficient statistics:
$$
S = X^\top X = \sum_i x_i x_i^\top \in \mathbb{R}^{d \times d}, \qquad
T = X^\top Y = \sum_i x_i y_i^\top \in \mathbb{R}^{d \times k}.
$$
The optimum can therefore be written as
$$
W^* = (S + \lambda I_d)^{-1} T.
$$

Once $(S,T)$ are known, raw samples are unnecessary for optimization. This is the basis of the unlearning protocol. For a single sample $(x,y)$, the add/delete contributions are
$$
\Delta S = xx^\top, \qquad \Delta T = xy^\top.
$$
For a client batch $\{x_i,y_i\}$, they become
$$
\Delta S = \sum_i x_i x_i^\top, \qquad \Delta T = \sum_i x_i y_i^\top.
$$

The server maintains a ledger
$$
S \leftarrow S + \Delta S_{\text{add}} - \Delta S_{\text{del}}, \qquad
T \leftarrow T + \Delta T_{\text{add}} - \Delta T_{\text{del}}.
$$
Because $S$ and $T$ are sums, addition and subtraction commute and associate. The final $(S,T)$ are therefore independent of the order of client reports, the partitioning of data across clients, and the interleaving of add/delete events, as long as the final retained multiset of samples is the same.

A key implication is that second-order information is indispensable. Two batches can share first moments $\sum_i x_i$ and $\sum_i y_i$ yet have different Grams $\sum_i x_i x_i^\top$; since $W^*$ depends on $S$, any protocol lacking $\Delta S$ cannot be exact for all batches. This sharply distinguishes RidgeFT from approximate federated unlearning methods that target general deep networks but cannot reduce the problem to additive sufficient statistics [2603.12977].

## 3. Federated continual add/delete protocol

The operational protocol is client-side feature extraction plus server-side sufficient-statistic maintenance. Each client computes frozen features locally and, for every add or delete request, transmits fixed-size sufficient-statistic messages $(\Delta S,\Delta T)$ together with minimal metadata such as request identifiers. Message size depends on $d$ and $k$, not on the number of local samples.

For large batches, clients may compute a thin QR factorization of a local feature matrix $F = QR$, with $Q$ orthonormal and $R$ upper-triangular, and send $R$ because $\Delta S = R^\top R$; they still send $\Delta T = F^\top Y$. The server aggregates client messages, updates $S$ and $T$ by the ledger rules, and computes
$$
W_t = (S_t + \lambda I_d)^{-1} T_t.
$$
In exact arithmetic, this equals centralized retraining on the retained dataset at each time $t$.

Two server-side realizations are described.

| Variant | State and update rule | Notes |
|---|---|---|
| **Variant A (Exact SPD solve)** | Maintain $S$, $T$, form $H = S + \lambda I_d$, compute Cholesky $H = LL^\top$, solve $LL^\top W = T$ | Numerically robust; recommended baseline |
| **Variant B (Incremental inverse tracker via Sherman–Morrison–Woodbury)** | Maintain $(S + \lambda I_d)^{-1}$ and current $W$; apply low-rank adds/downdates using SMW | Efficient for rank-$r$ updates; requires feasibility checks and periodic reset |

Variant A stores $S \in \mathbb{R}^{d \times d}$, $T \in \mathbb{R}^{d \times k}$, and optionally the Cholesky factor $L$. Cholesky can be recomputed each round, although rank-one or rank-$k$ updates and downdates are possible. Using $\lambda > 0$ ensures $H \succ 0$, and fp64 server-side computation improves conditioning and reduces floating-point drift, especially when accumulating second-order statistics.

Variant B writes $\Delta S = U^\top U$ with $U \in \mathbb{R}^{r \times d}$ and updates the inverse by Sherman–Morrison–Woodbury. For adds,
$$
T_{\text{inv}}^+ = T_{\text{inv}} - T_{\text{inv}} U^\top (I_r + U T_{\text{inv}} U^\top)^{-1} U T_{\text{inv}},
$$
followed by an update of $W$ using the new inverse and $\Delta T_{\text{add}}$. For deletes,
$$
T_{\text{inv}}^- = T_{\text{inv}} + T_{\text{inv}} U^\top (I_r - U T_{\text{inv}} U^\top)^{-1} U T_{\text{inv}},
$$
which requires $H^- \succ 0$, equivalently $I_r - U T_{\text{inv}} U^\top \succ 0$. In finite precision, repeated downdates can accumulate drift; the prescribed safeguard is to reset by recomputing through Variant A whenever the downdate becomes ill-conditioned or failure is detected [2603.12977].

## 4. Deterministic guarantees and Bayesian interpretation

The paper’s core guarantee is deterministic retrain-equivalence. After any sequence of add/delete requests, the server’s head equals the centralized ridge solution on the resultant dataset:
$$
W_t = (S_t + \lambda I_d)^{-1} T_t,
$$
where $(S_t,T_t)$ are the exact sufficient statistics of the retained multiset. The proof is straightforward in structure: ridge depends only on $(S,T)$, the ledger updates maintain $(S_t,T_t)$ exactly, Variant A computes the exact solution from those statistics, and Variant B maintains the exact inverse via SMW in exact arithmetic.

Order invariance and partition invariance follow from the same algebraic structure. For a fixed final retained dataset, $W_t$ is independent of the sequence of add/delete requests and independent of how data are partitioned across clients, because $(S_t,T_t)$ depend only on the final multiset.

The paper also gives a Bayesian certificate of zero KL divergence. Ridge is interpreted as MAP estimation under a Gaussian prior and Gaussian likelihood:
- $\operatorname{vec}(W) \sim \mathcal{N}(0,\tau^2 I_{dk})$, equivalently $W \sim \operatorname{MN}(0,\tau^2 I_d, I_k)$;
- $Y \mid X,W \sim \mathcal{N}(XW,\sigma^2 I_{nk})$;
- $\lambda = \sigma^2/\tau^2$.

The posterior is matrix-normal,
$$
W \mid (X,Y) \sim \operatorname{MN}(M,\Sigma,I_k),
$$
with
$$
M = (S + \lambda I_d)^{-1}T, \qquad
\Sigma = \sigma^2 (S + \lambda I_d)^{-1}.
$$
Because the protocol maintains $(S_t,T_t)$ exactly, both the posterior mean and covariance match centralized retraining, so the two Gaussians are identical and the KL divergence is zero. This does not merely certify equal point estimates; it certifies equality of the full Gaussian posterior induced by the ridge model [2603.12977].

## 5. Computational profile and empirical validation

The server-side cost depends on the chosen variant. Variant A forms $H = S + \lambda I_d$ and computes its Cholesky factorization once per request in $O(d^3)$ time, then solves $HW = T$ in $O(d^2k)$ time. Variant B, for rank-$r$ per-round updates, incurs $O(rd^2 + r^3)$ to update the inverse and $O(d^2k)$ to update $W$, with smaller constants when using reuse and block updates.

Communication is fixed-size per request. With direct statistics, each client sends $\Delta S \in \mathbb{R}^{d \times d}$ and $\Delta T \in \mathbb{R}^{d \times k}$, for message size $\Theta(d^2 + dk)$ independent of the number of local samples. With QR-based compression, clients send $R \in \mathbb{R}^{r \times d}$ such that $\Delta S = R^\top R$, plus $\Delta T \in \mathbb{R}^{d \times k}$, for message size $\Theta(rd + dk)$. When $r \ll d$, this substantially reduces communication. Each request completes in a single round with fixed-size messages and closed-form updates, avoiding multi-round FedAvg coordination and per-client local optimization.

Experiments were conducted on CIFAR-10, CIFAR-100, FeMNIST, and Sentiment140, under non-IID client partitions, using DINOv2-ViT B/14 for image tasks and RoBERTa (TweetEval) for Sentiment140, with $d = 768$ in both cases. Both Variant A and Variant B match centralized ridge retraining in accuracy across all four benchmarks and produce weights pointwise identical up to floating-point error. The reported relative Frobenius deviation
$$
\|W_{\text{fed}} - W_c\|_F / \|W_c\|_F
$$
is approximately $10^{-9}$ in fp64 on FeMNIST, with comparable behavior on the other datasets. Using fp32 for second-order accumulation yields approximately $10^{-3}$ to $10^{-4}$, which confirms the benefit of fp64 for $S,T$ aggregation and solves.

In continual deletion experiments, 200 single-point deletions incur small, consistent per-request latency and are orders of magnitude faster than FedAvg retraining, while also being faster overall than Exact-Fun and FATS, which occasionally retrain. For repeated 20% chunk deletions, the variants remain near the centralized ridge baseline and outperform FedAvg-based baselines that require multiple rounds. In a continual add-back test, after deleting 200 single points and then re-adding them, the model returns to the original $W$ with deviations approximately $10^{-11}$. Regularization through $\lambda$ ensures SPD and conditioning; larger $\lambda$ improves stability, and tuning $\lambda$ is straightforward because the solution is closed-form [2603.12977].

## 6. Assumptions, limitations, and relation to other uses of the term

RidgeFT’s exactness depends on a narrow but practically important regime. The feature extractor $\phi$ must be frozen and deterministic, running in evaluation mode with no dropout. If $\phi$ is stochastic, or if data are unavailable for recomputation, clients must cache feature vectors at add time to ensure exact deletes. The trainable head must be linear and trained by ridge regression; the guarantees hinge on the closed form and the sufficiency of $(S,T)$. Exactness is therefore exactness in arithmetic structure, with fp64 yielding near-machine-precision equivalence in practice rather than symbolic exactness under arbitrary floating-point execution.

The privacy posture is limited but explicit. $\Delta S$ and $\Delta T$ are aggregate second-order statistics; they do not expose raw inputs, but they can leak distributional information, especially at large $d$. The stated mitigation is to combine the protocol with secure aggregation and, where needed, differential privacy on statistics. Applicability is broad within the ridge-head regime: one-vs-rest or multinomial ridge with $k$ classes, soft labels, multi-label heads, and regression or multi-task heads all fit directly. The main trade-off is scope: the backbone must remain frozen, and unlearning inside the backbone is orthogonal and remains challenging [2603.12977].

The name “RidgeFT” is not fully standardized across the literature. In lifelong machine-generated text attribution, "When New Generators Arrive: Lifelong Machine-Generated Text Attribution via Ridge Feature Transfer" uses RidgeFT for a replay-free closed-form update framework built around a frozen encoder, covariance calibration, fixed random features, and class-wise sufficient statistics [2606.05626]. In heterogeneous federated learning, "Accelerating Heterogeneous Federated Learning with Closed-form Classifiers" describes Fed3R+FT, a ridge-regression-initialized fine-tuning pipeline on fixed pretrained features, which the detailed synthesis explicitly identifies as RidgeFT [2406.01116]. In few-shot class-incremental audio classification, the multi-level embedding extractor plus ridge regression classifier is described as, in essence, RidgeFT for audio [2506.18406]. This suggests that “RidgeFT” functions less as a single canonical acronym than as a family resemblance: frozen or stabilized feature extraction paired with analytic ridge updates. Within that family, the 2026 federated continual unlearning formulation is distinguished by exact retrain-equivalence, order and partition invariance, and a Bayesian zero-KL certificate [2603.12977].

Source: https://www.emergentmind.com/topics/ridgeft