NewCluster: RND-based Federated Clustering

• NewCluster is a federation-discovery mechanism that uses RND novelty profiles to infer latent client groups based on clustered feature skew.
• It decouples similarity estimation from task model training by exchanging lightweight RND predictors, reducing evaluation overhead significantly.
• The method dynamically forms federations using adaptive compatibility thresholds, efficiently addressing non-IID data in federated learning.

Searching arXiv for the main paper and closely related foundational work on RND and clustered federated learning. Search query: "(Domini et al., 29 Jun 2026) Random Network Distillation clustered federated learning" NewCluster denotes the federation-discovery mechanism introduced for clustered federated learning in which client groups are inferred before federated optimization from Random Network Distillation (RND) novelty profiles rather than from repeated evaluation of the main task model (Domini et al., 29 Jun 2026). It is designed for structured non-independently and identically distributed data, especially settings in which clients form latent groups with aligned feature distributions inside each group and mismatched feature distributions across groups. By decoupling clustering from learning, NewCluster replaces per-round task-model-based cluster assignment with a lightweight auxiliary procedure based on exchanged RND predictors, allowing the collaboration structure and even the number of federations to emerge at runtime without raw-data exchange (Domini et al., 29 Jun 2026).

1. Structured heterogeneity and the clustering objective

The motivating setting is federated learning under non-IID client data. In that regime, averaging biased local updates can degrade both convergence and final accuracy, while a single global model may fail to represent the diversity of client distributions. Clustered federated learning addresses this by training specialized models for groups of similar clients, but prominent approaches such as IFCA couple cluster assignment with the main training loop, requiring repeated evaluation of full task models (Ghosh et al., 2020).

NewCluster formalizes the target regime as clustered feature skew. Let $\mathcal{D}=\{d_1,\ldots,d_N\}$ be a set of clients partitioned into latent clusters $C_1,\ldots,C_k$, and let client $d_i$ have local distribution $P_i(x,y)$. The defining assumption is that all clients share the same label marginal distribution, while the class-conditional feature distribution is identical within a cluster and different across clusters:

$P_i(y)=P_j(y), \qquad P_i(x\mid y)=P_j(x\mid y)$

for any $d_i,d_j \in C_q$, whereas for $d_i \in C_q$ and $d_j \in C_r$ with $q\neq r$,

$$P_i(y)=P_j(y), \qquad P_i(x\mid y)\neq P_j(x\mid y).$$

Given unknown latent clusters, the aim is to learn a mapping from devices to discovered federations,

$C_1,\ldots,C_k$0

that minimizes mismatch to the true partition:

$C_1,\ldots,C_k$1

where $C_1,\ldots,C_k$2 denotes permutations of cluster indices and $C_1,\ldots,C_k$3 is symmetric difference (Domini et al., 29 Jun 2026).

This formulation makes the role of NewCluster specific. It is not a generic clustering heuristic over model updates; it is a pre-training federation-discovery procedure targeted at feature-distribution similarity under a shared label space. A plausible implication is that the method is naturally aligned with clustered feature skew, but is not presented as a universal remedy for arbitrary forms of statistical heterogeneity.

2. Random Network Distillation as a client-similarity probe

The mechanism uses RND, adapted from Burda et al., as a novelty signal rather than as an exploration bonus (Burda et al., 2018). Two networks are defined over inputs $C_1,\ldots,C_k$4: a fixed random target network

$C_1,\ldots,C_k$5

and a predictor network

$C_1,\ldots,C_k$6

Client $C_1,\ldots,C_k$7 trains its predictor on local data by minimizing

$C_1,\ldots,C_k$8

After fitting, the per-sample novelty is the residual error

$C_1,\ldots,C_k$9

The core intuition is distributional: a predictor trained on one client’s data produces low residuals on another client’s data if the two feature distributions are similar, and higher residuals otherwise.

To compare clients without raw-data exchange, client $d_i$0’s trained predictor is evaluated by client $d_i$1 on $d_i$2’s data:

$d_i$3

Here $d_i$4 is self-novelty and $d_i$5 is cross-novelty. The vector

$d_i$6

is the client-level summary statistic, and the server stacks all rows into the novelty matrix $d_i$7 with $d_i$8 (Domini et al., 29 Jun 2026).

This construction is significant because it decouples similarity estimation from the downstream task model. The exchanged object is a compact auxiliary predictor, not a classifier or regressor specialized to the federated objective. The paper therefore characterizes the signal as task-agnostic.

3. Compatibility thresholds and emergent federations

NewCluster converts novelty scores into collaboration structure through a client-adaptive compatibility rule. Client $d_i$9 is considered compatible with client $P_i(x,y)$0 if

$P_i(x,y)$1

where $P_i(x,y)$2 is the standard deviation of $P_i(x,y)$3 and $P_i(x,y)$4 is a tolerance hyperparameter (Domini et al., 29 Jun 2026). The criterion is anchored to the self-baseline $P_i(x,y)$5 and scaled by the dispersion of cross-novelty values observed by client $P_i(x,y)$6.

Federations are then formed graphically. An undirected edge $P_i(x,y)$7 is inserted when both clients deem each other compatible under the threshold. The discovered federations are extracted as connected components or maximal compatibility groups in that graph. The number of federations is therefore not specified a priori; it emerges from the observed novelty structure.

Operationally, the device-side procedure is: train a local RND predictor on $P_i(x,y)$8, send the predictor to the server, receive all predictors, evaluate each received predictor on local data to compute $P_i(x,y)$9, and send the novelty row $P_i(y)=P_j(y), \qquad P_i(x\mid y)=P_j(x\mid y)$0. On the server, at round $P_i(y)=P_j(y), \qquad P_i(x\mid y)=P_j(x\mid y)$1, clustering is triggered if $P_i(y)=P_j(y), \qquad P_i(x\mid y)=P_j(x\mid y)$2 or $P_i(y)=P_j(y), \qquad P_i(x\mid y)=P_j(x\mid y)$3; predictors are collected and redistributed, novelty rows are aggregated into $P_i(y)=P_j(y), \qquad P_i(x\mid y)=P_j(x\mid y)$4, federations $P_i(y)=P_j(y), \qquad P_i(x\mid y)=P_j(x\mid y)$5 are extracted, and federated learning is run within $P_i(y)=P_j(y), \qquad P_i(x\mid y)=P_j(x\mid y)$6 (Domini et al., 29 Jun 2026).

A common misconception in clustered federated learning is that the number of clusters must be chosen in advance. NewCluster is explicitly constructed to avoid that requirement: the collaboration structure is discovered from mutual compatibility rather than imposed as a fixed-$P_i(y)=P_j(y), \qquad P_i(x\mid y)=P_j(x\mid y)$7 partition.

4. Integration with federated optimization

The clustering phase precedes, rather than interleaves with, task optimization. The workflow has three stages. First, a shared random target network $P_i(y)=P_j(y), \qquad P_i(x\mid y)=P_j(x\mid y)$8 is initialized, clients train local RND predictors, predictors are exchanged, novelty rows are computed, and federations $P_i(y)=P_j(y), \qquad P_i(x\mid y)=P_j(x\mid y)$9 are formed. Second, within each discovered federation $d_i,d_j \in C_q$0, a specialized model $d_i,d_j \in C_q$1 is trained by standard federated learning, for example FedAvg (McMahan et al., 2016). The per-federation objective is

$d_i,d_j \in C_q$2

Third, if distributions drift, clustering can be re-triggered every $d_i,d_j \in C_q$3 rounds so that clients may migrate to better-suited federations at runtime (Domini et al., 29 Jun 2026).

The principal systems consequence is decoupling. In coupled schemes, clients repeatedly score one or more full task models to determine cluster membership. In NewCluster, the auxiliary RND stage is invoked once initially or periodically, and the main task model is not used for similarity estimation. This suggests a cleaner separation between collaboration discovery and model fitting, with lower recurring overhead.

5. Experimental protocol and empirical behavior

The empirical study uses CIFAR-10 under clustered feature skew. The dataset is partitioned into $d_i,d_j \in C_q$4 groups with identical label distributions and group-specific feature skew induced by different Gaussian noise per group. There are $d_i,d_j \in C_q$5 devices, with $d_i,d_j \in C_q$6 devices per group, and the partition is constructed using ProFed (Domini et al., 29 Jun 2026).

Component	Configuration
Dataset	CIFAR-10, $d_i,d_j \in C_q$7 groups, clustered feature skew
Devices	$d_i,d_j \in C_q$8, 3 per group
RND predictor	Lightweight CNN with two convolutional layers followed by a flattening layer
Baseline	IFCA with a full task model ResNet-18
Repetitions and machine	10 random seeds; Intel i7-8700K, 64GB RAM, RTX 4070

The reported novelty structure is visually diagnostic. The heatmap of $d_i,d_j \in C_q$9 exhibits clear diagonal block patterns: within-group differences are near zero, indicating that predictors from the same group generalize to each other’s data, whereas cross-group differences are higher. This is the central empirical argument that RND residuals expose latent group structure before task-specific federated training begins (Domini et al., 29 Jun 2026).

The overhead comparison is equally central. In the conservative configuration in which clustering is repeated every round, the RND-based method shows about one order-of-magnitude lower overhead than IFCA, primarily because the auxiliary predictors are compact and cheaper to evaluate than full task models. The gap widens further when clustering is performed once or periodically rather than every round (Domini et al., 29 Jun 2026). The comparison is conceptually aligned with IFCA’s coupled design, which requires repeated full-model evaluation (Ghosh et al., 2020).

6. Privacy, scalability, and failure modes

The privacy model is limited but explicit. Clients exchange compact RND predictors and aggregated novelty rows; raw features and labels are never transmitted. The novelty matrix $d_i \in C_q$0 functions as a task-agnostic similarity proxy while preserving data locality. At the same time, the paper does not provide formal privacy or leakage bounds, and it recommends treating predictors and error profiles as potential side channels and applying standard federated privacy tools such as secure aggregation if required (Domini et al., 29 Jun 2026).

The main scalability bottleneck is all-to-all predictor evaluation. In the centralized form, each client evaluates every received predictor on local data, so score computation scales quadratically in the number of clients. The paper therefore identifies several mitigations: neighborhood-only exchange over $d_i \in C_q$1, which reduces per-device evaluation from $d_i \in C_q$2 to $d_i \in C_q$3; gossip or leader-election schemes replacing the central server; periodic re-clustering with large $d_i \in C_q$4; and sparsification strategies such as approximate KNN or retaining only top-$d_i \in C_q$5 compatible peers (Domini et al., 29 Jun 2026).

Limitations are also tied to the thresholding mechanism. The compatibility rule depends on $d_i \in C_q$6 and on the variability captured by $d_i \in C_q$7. Too tight a threshold can fragment clusters; too loose a threshold can merge distinct groups. Clients with very small local datasets may train noisier predictors and therefore produce less reliable novelty scores. The adaptive threshold relative to $d_i \in C_q$8 and $d_i \in C_q$9 is already meant to calibrate per-client scale, but threshold tuning remains operationally important.

7. Position within clustered federated learning

NewCluster occupies a specific niche within clustered federated learning. It is not a new per-round optimization rule, nor a new task model, nor a privacy guarantee. Its contribution is a lightweight federation-discovery layer that infers collaboration structure from auxiliary novelty estimation. In contrast to coupled approaches such as IFCA (Ghosh et al., 2020), it does not require repeated scoring of the main model for cluster assignment. In contrast to standard FedAvg (McMahan et al., 2016), it does not assume that a single shared model is adequate for all clients.

Its most distinctive property is that federations emerge from local novelty estimates at runtime, making the method suitable for autonomous large-scale distributed systems in which neither the number of clusters nor the collaboration structure can be specified a priori (Domini et al., 29 Jun 2026). The paper’s results support that claim under clustered feature skew: RND residuals recover the latent block structure, and the decoupled clustering stage imposes substantially lower overhead than the coupled baseline.

A plausible broader implication is that NewCluster reframes clustered federated learning as a two-layer problem. The first layer discovers who should collaborate, using lightweight distributional probes. The second layer performs the actual task optimization within those discovered federations. Under the assumptions of shared label marginals and clustered feature skew, that separation is the defining idea of the method.