---
title: 'FedHOO: Joint Resampling Tuning'
url: https://www.emergentmind.com/topics/fedhoo
type: topic
---

# FedHOO: Joint Resampling Tuning

FedHOO is an auxiliary hyperparameter-search component for jointly tuning client-wise resampling rates in federated learning under class imbalance. It is introduced alongside FedCGNM in "Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated Learning" [2607.01474]. In that framework, the vector of client rates $r=(r_1,\dots,r_K)\in\mathcal X=[r_{\min},r_{\max}]^K$ is treated as an arm in an X-armed bandit / hierarchical optimistic optimization problem, and the reward of an arm is defined by the validation performance of the aggregated federated model obtained when clients train with those rates. FedHOO is explicitly intended for small-client federations, where exhaustive joint search is combinatorial and expensive but still structurally exploitable through federated parallelism.

## 1. Placement within the FedCGNM framework

FedHOO is not the main optimizer of the overall method. Its role is to tune the resampling policy used together with FedCGNM, the paper’s client-side optimizer that groups classes, computes per-group momentum, normalizes each group momentum to unit norm, and sums them to form the update direction [2607.01474].

The paper formulates local training under resampling through
$$
f_k(x;r_k)=\mathbb E_{\xi\sim\mathcal D_k(r_k)}[\ell(x;\xi)],
$$
and the corresponding global objective through
$$
f(x;r)=\sum_{k=1}^K p_k f_k(x;r_k).
$$
Within this decomposition, FedCGNM optimizes the model parameter $x$ given a set of rates, whereas FedHOO searches over the rate vector $r$. The motivation is that resampling rates cannot be chosen independently across clients without ignoring cross-client interactions induced by federated aggregation. The best rate for one client may depend on the rates selected by others because the final global model depends on the aggregation of all client updates.

The paper presents this dependency as especially important under class imbalance, where the appropriate resampling policy depends on both the severity of skew and the local client label distributions. A plausible implication is that FedHOO addresses a coordination problem rather than merely a collection of client-local tuning problems.

## 2. Search space and hierarchical optimistic optimization

FedHOO models the continuous joint rate space with a tree over boxes in $\mathcal X$. Each node $\nu$ corresponds to a box
$$
I(\nu)=\prod_{k=1}^K [L_k(\nu),U_k(\nu)],
$$
with a midpoint representative, and the tree is organized as a $2^K$-ary tree because each client contributes two probe values per round [2607.01474].

The node statistics are HOO-like. Each node stores a running reward estimate $V(\nu)$, a visit count $N(\nu)$, and an optimistic score
$$
B(\nu)=V(\nu)+\tau\,\mathrm{diam}(\nu)^h+\sqrt{\frac{\alpha\ln(t+1)}{N(\nu)}}.
$$
Here $\tau,\alpha>0$ are exploration hyperparameters and
$$
\mathrm{diam}(\nu)=\max_k(U_k(\nu)-L_k(\nu))
$$
is the box width. The second and third terms provide optimism bonuses to broad nodes and rarely visited nodes.

At each communication round, the server selects the leaf with the highest $B(\nu)$. For the selected node, each client receives two probe rates within its current interval,
$$
r_k^L=\frac{3L_k+U_k}{4},\qquad r_k^U=\frac{L_k+3U_k}{4}.
$$
Each client then trains twice from the same global model, once with $r_k^L$ and once with $r_k^U$, and returns two local update deltas. The paper characterizes this as a federated hierarchical-optimistic-optimization procedure over a continuous XAB space, but with a specifically federated execution model in which all clients run in parallel.

## 3. Linear aggregation and combinatorial synthesis

The key algorithmic device in FedHOO is the use of linear federated aggregation to synthesize an exponential number of joint evaluations from only two local trainings per client [2607.01474]. For each binary vector $s\in\{0,1\}^K$, the server constructs the candidate update
$$
\Delta^{(s)}=\sum_{k=1}^K p_k\big[(1-s_k)\Delta_k^L+s_k\Delta_k^U\big],
$$
and then forms the corresponding candidate model
$$
x^{(s)}=x^{(t)}-\Delta^{(s)}.
$$

Because the server can mix the low and high updates independently across clients, the two local runs per client provide enough information to evaluate all $2^K$ corner combinations of the current search box. The paper emphasizes that this “mix and match” capability is what makes FedHOO dramatically faster than standard HOO in small federations. Rather than evaluating one configuration per round, the method obtains $2^K$ synthetic joint evaluations while incurring only linear local training and communication cost per client.

After validation of the synthesized candidates, the best-performing candidate becomes the next global model $x^{(t+1)}$. The selected node’s region is then refined by updating tree statistics and shrinking the search interval toward the chosen probe region. In the pseudocode, after evaluating all $2^K$ candidates, the server computes the average reward $\bar R$, updates the nodes on the path from the selected leaf to the root, increments their counts by $2^K$, updates reward estimates by a running average, and recomputes optimistic scores. This is the mechanism governing exploration and exploitation.

## 4. Privacy model, validation protocol, and practical operating regime

FedHOO does not require centralized validation data. After the server synthesizes the $2^K$ candidate models, clients evaluate them on their private validation splits and return only scalar validation metrics, such as validation loss or accuracy; the server aggregates these metrics to pick the best model [2607.01474].

The paper therefore presents FedHOO as privacy-preserving in the same sense as standard federated learning: raw data do not leave the clients. It also notes that, if a server-side holdout set exists, it can be used, but it is not required. Operationally, the method is a server-side hybrid coordination scheme: clients execute local training runs, while the server selects probe intervals, synthesizes candidate global models, performs validation, updates the search tree, and chooses the next interval to explore.

The implementation details reported in the experiments are specific. In the small-scale regime with $K=5$, the sampling rates are restricted to
$$
r_k\in[0.4,0.8],
$$
the tree is initialized on $[0.4,0.8]^K$, and the optimism constant is set to $\alpha=1.0$. The tree depth is capped at $5$. The authors run three warm-up rounds without expanding the tree to address cold start sensitivity, and then use FedHOO only in the early phase of training. Once the selected rates remain inside the same per-client intervals for five consecutive rounds, the rates are fixed and training continues with them. The stated purpose of this procedure is to find stable resampling rates quickly rather than to continue exploration indefinitely.

## 5. Theoretical motivation for stable rate schedules

The paper does not provide a regret analysis for FedHOO itself, but it gives a theoretical reason that motivates rapid stabilization of the rate schedule [2607.01474]. In the FedCGNM convergence theorem, the average gradient norm is bounded by
$$
\mathcal O(T^{-1/2})+\mathcal O(V_T\cdot T^{-1/2}),
$$
where
$$
V_T=\sum_{t=0}^{T-1}\sum_k p_k\,\mathbb E|r_k^{(t+1)}-r_k^{(t)}|
$$
is the cumulative variation of the rate schedule.

The details section also writes the variation term as
$$
V_T=\sum_t\sum_k p_k|r_k^{(t+1)}-r_k^{(t)}|,
$$
and states that convergence at the standard $\mathcal O(T^{-1/2})$ rate requires this variation to be controlled. The paper presents FedHOO as the practical mechanism for finding effective rates quickly and then stabilizing them so that $V_T/\sqrt{T}\to 0$.

This theoretical linkage is central to the method’s intended use. FedHOO is not designed as a persistent online tuner that explores throughout training; rather, it is designed to identify a satisfactory joint resampling configuration early, after which the rates are fixed. This suggests that the search procedure is justified less by asymptotic bandit optimality than by its compatibility with the convergence requirements of the underlying optimizer.

## 6. Empirical results, comparisons, and explicit limitations

Empirically, FedHOO is evaluated only as a rate-tuning strategy, both standalone with various optimizers and in combination with FedCGNM. On CIFAR-10-LT, CIFAR-100-LT, Adult Income, UNSW-NB15, and a proprietary chip-defect dataset, the paper reports that FedHOO consistently improves over using a uniform global sampling rate, with the best small-$K$ results obtained by FedCGNM + FedHOO [2607.01474].

| Setting | Reported comparison | Reported outcome |
|---|---|---|
| $K=5$ IID, CIFAR-10-LT20 | FedCGNM + uniform rate vs FedCGNM + FedHOO | $0.8568 \rightarrow 0.8628$ |
| $K=5$ IID, CIFAR-10-LT100 | FedCGNM + uniform rate vs FedCGNM + FedHOO | $0.7432 \rightarrow 0.7485$ |
| $K=5$ non-IID, CIFAR-10-LT20 | FedCGNM + uniform rate vs FedCGNM + FedHOO | $0.8316 \rightarrow 0.8381$ |
| $K=5$ non-IID, CIFAR-100-LT20 | FedCGNM + uniform rate vs FedCGNM + FedHOO | $0.4351 \rightarrow 0.4427$ |
| Chip-defect dataset | FedCGNM with fixed global rate vs FedCGNM + FedHOO | $86.88 \rightarrow 90.87$ |

For the chip-defect dataset, the paper isolates FedHOO’s contribution by reporting that FedCGNM rises from 86.88 with a fixed global rate to 90.87 with FedHOO, described as a 4.59% improvement. It also reports similar gains for FedAvg, Weighted CE, Ratio Loss, and FedCGN, which the authors use to support the claim that the benefit derives from joint rate search rather than from the optimizer choice alone.

The paper additionally compares FedHOO against two alternative local tuning strategies: a FAST-style client-local bandit method and a “locally optimal” per-client selection rule. In those ablations, FedHOO consistently outperforms both. The authors interpret this as evidence that independent per-client tuning misses cross-client interactions that matter under federated aggregation. This directly supports the characterization of FedHOO as a joint combinatorial exploration method rather than merely a better local rate selector.

The limitations and assumptions are stated explicitly. FedHOO is designed for small federations, and the paper repeatedly warns that the $2^K$ validation burden becomes prohibitive as $K$ grows. It therefore uses FedHOO when $K=5$ but falls back to a uniform global rate when $K$ is larger, such as $K=20$ or in large-client settings. The authors also recommend applying FedHOO at a cluster level as an alternative for larger systems. The method assumes that all clients participate when the $2^K$ joint synthesis is performed, because partial participation makes consistent corner synthesis more difficult. It is further used only in the early rounds in practice, and it assumes access to client-side validation splits, or an equivalent server-side validation set if available.

Source: https://www.emergentmind.com/topics/fedhoo