Panel Kmeans Clustering

• Panel Kmeans algorithm is a least-squares clustering method for panel data that uses full time-series profiles of transformed loss differentials rather than collapsing data into averages.
• It employs a Lloyd-type iteration that repeatedly assigns units to clusters based on minimizing accumulated Euclidean distances over time, ensuring valid post-clustering inference.
• Key considerations include preprocessing, scaling, and selecting the number of clusters to maintain robust predictive-ability testing and accurate measurement of group heterogeneity.

The Panel Kmeans algorithm denotes a class of K-means or least-squares clustering procedures adapted to panel data, in which cross-sectional units are assigned to latent groups using repeated observations over time. In the formulation developed for testing clustered equal predictive ability (C-EPA), the algorithm clusters units on the full $T\times P$ profiles of transformed loss differentials $Z_{it}$, so that time-series variation enters the clustering criterion directly rather than being collapsed into time averages; the estimated partition then becomes part of the selection event for valid post-clustering inference (Akgun et al., 19 Jul 2025).

1. Formal setup and clustered objects

In the C-EPA framework, the panel consists of $N$ units, $i=1,\ldots,N$, observed over $T$ periods, $t=1,\ldots,T$. Two agents or models, $a=1,2$, produce forecasts of $Y_{it}$ with loss $L(\cdot,\cdot)$, and the loss differential is

$$\Delta L_{it}=L(\widehat Y_{1,it},Y_{it})-L(\widehat Y_{2,it},Y_{it}).$$

Given a $Z_{it}$0 testing function $Z_{it}$1, such as $Z_{it}$2 for unconditional EPA or $Z_{it}$3 for conditional EPA, the transformed panel array is

$Z_{it}$4

The units $Z_{it}$5 are clustered into $Z_{it}$6 groups, and each cluster $Z_{it}$7 has a $Z_{it}$8-dimensional centroid $Z_{it}$9 equal to the time-and-unit average of $N$0 within that cluster (Akgun et al., 19 Jul 2025).

The asymptotic structure is built from

$N$1

where $N$2 is weakly stationary over $N$3, satisfies standard $N$4- or $N$5-mixing and moment conditions, and has long-run variance

$N$6

that is positive definite. Cross-sectional dependence is allowed and can be strong, of factor-type, or weak, of spatial-type; a scaling parameter $N$7 governs CLT normalization in the presence of cross-sectional dependence. Cluster sizes are asymptotically non-negligible, $N$8, and under alternatives the clusters are separated in the sense that $N$9. No independence across units is required, and time-series dependence is generic.

This setup makes clear that the clustered objects are not the raw outcomes $i=1,\ldots,N$0 themselves, but the transformed loss-differential profiles $i=1,\ldots,N$1. A recurring misconception is to view Panel Kmeans as ordinary K-means applied to unit-level averages. In the C-EPA construction, the operative data are full trajectories indexed by $i=1,\ldots,N$2, and the inferential target is clusterwise predictive ability rather than generic geometric partitioning.

2. Objective function, metric, and Lloyd-type iteration

Using the full $i=1,\ldots,N$3 profiles $i=1,\ldots,N$4, the Panel Kmeans estimator solves a least-squares clustering problem with Euclidean distance accumulated over time. Its centroid update is

$i=1,\ldots,N$5

and the assignment step for each unit $i=1,\ldots,N$6 is

$i=1,\ldots,N$7

The distance is the standard squared Euclidean norm in $i=1,\ldots,N$8, with equal weights across $i=1,\ldots,N$9 (Akgun et al., 19 Jul 2025).

The algorithm is a Panel Lloyd iteration. Initialization randomly assigns each unit to one of $T$0 clusters, not via distance-based Kmeans++, and computes initial centroids as time-and-unit averages in each cluster. The procedure then alternates between reassignment and centroid recomputation until assignments stabilize, changes fall below a threshold, or a maximum iteration cap is reached. Per iteration, the assignment step has complexity $T$1, the updates have complexity $T$2, and with $T$3 iterations and $T$4 random starts the total complexity is approximately $T$5.

Its distinguishing feature relative to classical K-means is explicit use of time-series variation. Classical K-means on time averages would cluster $T$6 by minimizing $T$7. Panel Kmeans instead minimizes $T$8 with $T$9 equal to the grand average over time and units in cluster $t=1,\ldots,T$0. If $t=1,\ldots,T$1, the method reduces to standard K-means on the $t=1,\ldots,T$2-dimensional cross section $t=1,\ldots,T$3. If time is collapsed ex ante by using $t=1,\ldots,T$4, the objective becomes standard K-means on $t=1,\ldots,T$5, but that ignores the time variation the proposed algorithm exploits.

3. Preprocessing and selection of the number of clusters

Implementation begins with construction of $t=1,\ldots,T$6, which requires choosing the testing function $t=1,\ldots,T$7 and computing $t=1,\ldots,T$8 from forecast errors and the chosen loss, such as square or absolute loss. When $t=1,\ldots,T$9, scaling each coordinate of $a=1,2$0, for example by the overall standard deviation, is recommended to avoid dominance by high-variance components. Demeaning over time is not required because the centroid already averages across time and units. If strong drifts or seasonality are present, pre-detrending may help interpretability, but it is not part of the estimator’s theory. Missing data require equal-length series or consistent handling through imputation, use of common sample periods, or averaging over observed $a=1,2$1 with rescaling for comparability. The core algorithm uses squared Euclidean loss; robust alternatives such as $a=1,2$2 or Huber loss are not analyzed, and adopting them would change the selection-region geometry and require re-deriving selective inference (Akgun et al., 19 Jul 2025).

For unknown $a=1,2$3, the proposed information criterion is

$a=1,2$4

where

$a=1,2$5

The tuning constant satisfies $a=1,2$6, and the simulations use $a=1,2$7. A key theoretical point is that conditioning on the final clustering $a=1,2$8 implicitly conditions on $a=1,2$9, so selecting $Y_{it}$0 by this criterion before testing does not require further conditioning for validity within the selective conditional inference framework. A consistent cross-validation option also exists through time-split training and validation, although it can be computationally heavier; in the empirical section, cross-validation often selected $Y_{it}$1.

These choices affect both clustering quality and inference. The paper’s guidance is therefore not merely computational. It is part of the inferential design, because preprocessing, dimension scaling, and cluster-count selection determine the selection event on which valid testing is conditioned.

4. Selective conditional inference after clustering

The C-EPA null is

$Y_{it}$2

After transformation by the testing function, this becomes

$Y_{it}$3

The hypothesis decomposes into homogeneity across clusters, $Y_{it}$4 for all $Y_{it}$5, and overall EPA,

$Y_{it}$6

The overall EPA parameter does not depend on the partition (Akgun et al., 19 Jul 2025).

For pairwise homogeneity, the method uses the square root of a Wald statistic,

$Y_{it}$7

where

$Y_{it}$8

Under known clusters and regularity conditions, $Y_{it}$9. After clustering, however, valid critical values require conditioning on the estimated clusters. Conditional on the entire sequence of cluster assignments produced by the Panel Kmeans iterations and on nuisance directions, $L(\cdot,\cdot)$0 is asymptotically $L(\cdot,\cdot)$1 truncated to a data-dependent set. The perturbation path $L(\cdot,\cdot)$2 varies the component of the data along the estimated direction of cluster separation while holding the orthogonal component fixed, and the truncation set

$L(\cdot,\cdot)$3

is determined by the assignment inequalities. In the classical point-clustering case this structure is polyhedral; with Panel Kmeans, because assignment uses sums of squared distances over time, the inequalities are quadratic in $L(\cdot,\cdot)$4, so $L(\cdot,\cdot)$5 is a finite union of intervals.

Selective $L(\cdot,\cdot)$6-values are obtained from the survival function of a truncated $L(\cdot,\cdot)$7 distribution. Homogeneity across $L(\cdot,\cdot)$8 clusters is then tested by merging the $L(\cdot,\cdot)$9 pairwise $$\Delta L_{it}=L(\widehat Y_{1,it},Y_{it})-L(\widehat Y_{2,it},Y_{it}).$$0-values with the M-family generalized mean

$$\Delta L_{it}=L(\widehat Y_{1,it},Y_{it})-L(\widehat Y_{2,it},Y_{it}).$$1

which is valid under arbitrary dependence. Overall EPA is tested with

$$\Delta L_{it}=L(\widehat Y_{1,it},Y_{it})-L(\widehat Y_{2,it},Y_{it}).$$2

using an orthonormal-series long-run variance estimator robust to time dependence and strong cross-sectional dependence, with fixed-$$\Delta L_{it}=L(\widehat Y_{1,it},Y_{it})-L(\widehat Y_{2,it},Y_{it}).$$3 reference distribution $$\Delta L_{it}=L(\widehat Y_{1,it},Y_{it})-L(\widehat Y_{2,it},Y_{it}).$$4. The final C-EPA test merges the homogeneity and O-EPA $$\Delta L_{it}=L(\widehat Y_{1,it},Y_{it})-L(\widehat Y_{2,it},Y_{it}).$$5-values through

$$\Delta L_{it}=L(\widehat Y_{1,it},Y_{it})-L(\widehat Y_{2,it},Y_{it}).$$6

The inferential consequence is central. Naive post-clustering Wald tests severely over-reject, with size approximately $$\Delta L_{it}=L(\widehat Y_{1,it},Y_{it})-L(\widehat Y_{2,it},Y_{it}).$$7 in all Monte Carlo designs. By contrast, the selective-inference C-EPA procedure controls size accurately for $$\Delta L_{it}=L(\widehat Y_{1,it},Y_{it})-L(\widehat Y_{2,it},Y_{it}).$$8 and $$\Delta L_{it}=L(\widehat Y_{1,it},Y_{it})-L(\widehat Y_{2,it},Y_{it}).$$9, with rejection rates near the nominal $Z_{it}$00. Split-sample tests generally control size, but they can over-reject in the presence of structural breaks because training and testing partitions fail to account for time variation in cluster centers.

5. Related panel K-means formulations

The term “Panel Kmeans” is not restricted to predictive-ability testing. In the supplied literature it also denotes least-squares clustering procedures for grouped intercepts, grouped slopes, latent-group IV models, and panel models with small groups (Dzemski et al., 2020, Choi et al., 2024, Nishi, 21 Aug 2025).

Setting	Core criterion	Distinctive issue
Predictive-ability panels	Cluster on full $Z_{it}$01 profiles $Z_{it}$02	Post-clustering inference requires selective conditioning
Grouped-intercept panels	Minimize $Z_{it}$03	Root-$Z_{it}$04 estimation can hold with some asymptotic misclassification
Endogenous-regressor panels	Second-stage least squares on predicted regressors $Z_{it}$05	Direct K-means on the GMM objective does not yield unique estimates
Small-group panels	Least-squares clustering with or without grouped fixed effects	Standard information criteria can miss asymptotically negligible groups

In grouped-intercept long panels, the estimator alternates between within-cluster averaging and reassignment by unit-level residual sums of squares. Under long-panel asymptotics, sub-exponential tails, group separation, and a rate condition controlling the share and variance of units outside the “easy-to-classify” set, the group-specific means can be estimated at the parametric root-$Z_{it}$06 rate even if some units are asymptotically misclassified and error variances diverge with $Z_{it}$07. This result makes precise that uniform membership recovery is not always necessary for first-order asymptotic normality of the group parameters.

In latent-group IV models, the literature identifies a different failure mode. Direct application of K-means to pooled or groupwise GMM criteria is not well defined in the just-identified case and can fail to identify the true grouping in overidentified settings. The proposed remedy is a two-stage structure: first regress endogenous regressors on instruments, using either a homogeneous first stage, a grouped first stage (TGFE), or a unit-specific first stage (UGFE); then apply K-means or grouped fixed-effects least squares to the regression of $Z_{it}$08 on predicted regressors $Z_{it}$09. This establishes a panel K-means route for endogenous-regressor settings without relying on naive K-means-GMM minimization.

In panels with small groups, the principal issue is not endogeneity or post-selection inference but group-size asymptotics. When true group sizes satisfy $Z_{it}$10 with some $Z_{it}$11, longer $Z_{it}$12 is required as the smallest groups become smaller. Earlier information criteria can then be inconsistent or perform poorly: Bai–Ng penalties may underestimate $Z_{it}$13 by missing small groups, whereas BIC-type penalties may overfit in finite samples. Modified information criteria, MIC1 and MIC2, are proposed precisely for this regime.

6. Empirical use, implementation, and interpretation

A minimal implementation in the C-EPA setting takes as inputs the $Z_{it}$14-dimensional arrays $Z_{it}$15, a chosen $Z_{it}$16 or a grid for $Z_{it}$17, a number of random restarts $Z_{it}$18, and a maximum iteration count $Z_{it}$19. One run consists of random assignment of units to clusters, centroid computation by time-and-unit averaging, repeated reassignment by minimizing $Z_{it}$20, and retention of the solution that minimizes the objective over $Z_{it}$21 random starts. The paper reports values such as $Z_{it}$22–$Z_{it}$23 in simulations, $Z_{it}$24 in the application, and $Z_{it}$25. For selective inference, long-run variance estimation uses an orthonormal-series estimator,

$Z_{it}$26

with practical choice

$Z_{it}$27

Available software includes the R packages clusteredEPA, for EPA testing in panels with latent clusters including selective-inference and split-sample procedures, and PanelKmeansInference, for post-clustering inference for coefficient homogeneity across Panel Kmeans clusters; replication materials are provided at https://github.com/akoguzhan/ (Akgun et al., 19 Jul 2025).

The empirical application concerns exchange-rate forecasting. The data comprise $Z_{it}$28 bilateral exchange rates and approximately $Z_{it}$29 forecast periods with recursive window $Z_{it}$30. Models compared to AR(1) include AR(p)-BIC, Elastic Net, XGBoost in a data-poor specification, and SVM and Random Forest in data-rich specifications. Loss-differential summaries show substantial heterogeneity, and XGBoost often exhibits the largest average gain, corresponding to a negative mean loss differential. In formal testing, overall EPA frequently rejects, homogeneity often fails, especially when conditioning on lagged targets or macro regimes, cross-validation frequently selects $Z_{it}$31, and the selective-inference C-EPA procedure detects heterogeneous gains robustly.

Several interpretive cautions follow directly from the literature. First, Panel Kmeans is not a synonym for clustering unit averages; in its predictive-ability form it uses entire time-series profiles. Second, naive post-clustering Wald inference is invalid because cluster membership is estimated from the same data. Third, robust alternatives to squared Euclidean loss are not covered by the selective-inference theory. A plausible implication is that extensions to robust losses, irregular missingness, or alternative assignment metrics are not merely computational variants; they alter the selection event and therefore the reference distribution used for valid post-clustering testing.