How Much Do RF Drone Benchmarks Overstate? A Controlled Study and Theory of Data Leakage in UAV Signal Identification

Abstract: Radio-frequency (RF) sensing is a central modality for counter-unmanned-aerial-system (counter-UAS) defence because it exploits the control, telemetry, and video links between a drone and its operator. Reported accuracies for RF-based drone detection and identification are often very high, but many are obtained using cross-validation that splits a small number of continuous recordings into short segments. This can place near-duplicate slices of the same recording in both training and test partitions, creating data leakage. We study this leakage problem through theory and measurement. We formalise the optimism of segment-level cross-validation and show, using Cover's function-counting theorem, that a classifier can exactly memorise the recording-to-label map when the number of independent recordings, R, is small relative to the feature dimension, d. In particular, this can occur when 2R is less than or approximately equal to d. Under these conditions, naive accuracy approaches 1, and the inflation gap approaches 1 - ACC*, where ACC* is the Bayes accuracy. The inflation eases only once R grows beyond this separability threshold. A controlled synthetic experiment with 10 seeds confirms the predicted curves: naive balanced accuracy rises from the Bayes level toward 1.0 as recording-specific nuisance variation grows, while honest recording-grouped evaluation declines to chance, with a gap reaching about 0.5. On the public DroneRF dataset, pooled leave-one-recording-out cross-validation shows drone type identification, AR versus Bebop, collapsing from a naive macro-F1 of 0.74 to 0.46, the two-class chance level. A leakage-pathway ablation attributes essentially all of the inflation to segment-level leakage.

• The paper introduces a generative model revealing that segment-level cross-validation inflates accuracy by exploiting recording-specific nuisance features.
• It validates the theoretical predictions with synthetic experiments showing that as nuisance strength increases, naive accuracy erroneously approaches 1.0 compared to chance-level grouped estimates.
• The study establishes that limited independent recordings relative to feature dimensionality are the primary cause of overstated performance in RF drone identification benchmarks.

Data Leakage in RF Drone Identification Benchmarks: A Theoretical and Empirical Analysis

Introduction

RF-based drone detection leverages the persistent and structured emissions of UAV control, telemetry, and video links for counter-UAS operations. Despite widespread adoption of segment-based cross-validation protocols in academic and applied research, reported model accuracies are persistently and unjustifiably high. This phenomenon is systematically dissected through the lens of estimator optimism resulting from data leakage, specifically when segments from the same continuous recording contaminate both training and testing partitions. The manuscript provides both a rigorous quantitative framework and empirical validation, demonstrating that benchmark inflation is fundamentally determined by the number of independent recordings relative to feature dimensionality.

Formal Model of Data Leakage and Theoretical Analysis

The core theoretical contribution is a generative model in which each RF segment $\mathbf{x}$ is composed of a class-specific discriminative mean, a recording-specific nuisance vector projected onto nondiscriminative axes, and isotropic Gaussian noise. Critically, segment-level cross-validation allows classifiers to exploit the recording-specific nuisance, effectively mapping segments to recordings and thus to their associated labels, irrespective of any generalizable discriminative content.

Using Cover's function-counting theorem, the probability of linear separability of the $2R$ recording centroids in $\mathbb{R}^{d-1}$ is derived as $C(2R,d-1)$. The optimism (i.e., inflation) of naive cross-validation is therefore substantial in the regime $2R \lesssim d$, where perfect or near-perfect separation is almost always achievable via nuisance memorization. The predicted inflation approaches $1-\mathrm{ACC}^{\star}$, with $\mathrm{ACC}^{\star}$ representing the Bayes-optimal accuracy on previously unseen recordings.

Controlled Experimental Validation

Synthetic experiments instantiate the theoretical model with $d=20$ features and $S=50$ segments per recording, directly validating the predicted trends:

• Inflation versus nuisance strength ($\lambda$): As the nuisance amplitude increases, naive cross-validation accuracy monotonically progresses to $2R$0, diverging from the Bayes-level grouped estimate which converges to chance. The measured inflation gap empirically reaches the predicted $2R$1.

  Figure 1: As nuisance $2R$2 increases, naive cross-validation accuracy approaches $2R$3, while honest grouped evaluation accuracy drops to chance, quantifying the leakage-induced optimism.

• Inflation versus number of recordings ($2R$4): At low $2R$5, where $2R$6, naive accuracy remains maximal, declining gradually as $2R$7 increases past the separability threshold. In contrast, grouped accuracy remains near chance but demonstrates diminishing variance with increasing $2R$8.

  Figure 2: Naive inflation dominates for small numbers of recordings, corresponding to the regime where recording centroids are linearly separable; variance of the honest estimate declines as $2R$9 increases.

Empirical naive accuracy tracks the theoretical separability probability $\mathbb{R}^{d-1}$0, with nonlinear models such as Random Forests maintaining inflated accuracy even beyond the linear threshold, confirming that the derived bounds are conservative for flexible classifiers.

Figure 3: The empirical naive accuracy overlays the theoretical linear separability curve, highlighting that nonlinear models sustain leakage-induced optimism further than Cover’s linear threshold predicts.

Real-World Case Study: DroneRF Dataset

The DroneRF benchmark, often cited in the RF-based drone detection literature, is shown to be acutely affected due to its extremely limited count of independent recordings per drone type (four per AR and Bebop class after proper grouping, one per background). Pooled leave-one-recording-out cross-validation reduces AR vs. Bebop two-class macro-$\mathbb{R}^{d-1}$1 from $\mathbb{R}^{d-1}$2 under naive segmentation to $\mathbb{R}^{d-1}$3, which is indistinguishable from chance.

Ablation studies demonstrate that almost all of the inflation is attributable to segment-level leakage, with minimal impact from finer-grained grouping (e.g., separating band-halves). This finding decisively attributes performance inflation to the leakage pathway and verifies that practical deployments relying on naive evaluation are at risk of complete performance collapse in operational environments.

Recommendations for Evaluation and Implications

The analysis implies that cross-validation must always be conducted with recording-group granularity to avoid estimator optimism. Benchmarks must report the number of independent captures per class and demonstrate that $\mathbb{R}^{d-1}$4 if confident claims about classifier performance are to be substantiated. Simultaneous multi-channel/band data from a single flight must be merged into a single group to nullify any residual within-event confounding.

The practical implication is immediate: deployment decisions or procurement based on highly optimistic headline numbers are fundamentally unsound unless proper evaluation discipline is enforced. These conclusions extend beyond RF drone identification to any ML setup where repeated measurements from a fixed subject, session, or environment are at risk of contaminating partition boundaries.

Limitations and Future Directions

While the controlled experiments are rigorously parameterized, real-world RF signals are governed by further complexities, including nonlinear and correlated nuisance effects. The analysis is restricted by the minimal number of independent recordings in standard public datasets, resulting in inherently high-variance grouped estimates. Future work should target richer benchmarks with larger, more diverse independent captures and consider further theoretical generalizations to nonlinear or feature-learned representations beyond interpretable spectral descriptors. Expanding benchmark protocols and dataset availability, as well as developing variance reduction techniques and uncertainty quantification for grouped estimates, will be important for maturing this evaluation paradigm.

Conclusion

A comprehensive theoretical and empirical treatment is provided for data leakage and estimator optimism in RF-based drone identification. The central finding is that standard segment-based evaluation produces systematically overstated accuracies due to recording memorization, and that the magnitude of this overstatement is quantitatively controlled by the ratio of recordings to feature dimensionality. Honest, grouped evaluation protocols and sufficient dataset structuring are both necessary and, in current benchmarks, largely absent, casting doubt on a number of prior claims in the field. All code, synthetic data generation pipelines, and analysis scripts are released to enable immediate and reproducible adoption of these best practices.