WiDistill: Wireless Distillation Methods

• WiDistill is a framework for compressing large wireless datasets and foundation models using distillation techniques, including trajectory matching and adaptive knowledge transfer.
• The methodology enables up to 10× faster training, retains over 90% accuracy with minimal samples per class, and reduces storage requirements by up to 90%.
• WiDistill supports cross-architecture generalization, allowing distilled data and compressed models to perform robustly across different neural network architectures in wireless applications.

WiDistill encompasses a family of methods targeting model or dataset compression via distillation, with a primary focus on applications to wireless domains such as Wi-Fi Channel State Information (CSI) based human activity recognition and wireless channel prediction. WiDistill methodologies address critical challenges in storage, computational cost, and training speed when handling large-scale wireless datasets or deploying foundation models on edge hardware. Distillation is operationalized through techniques such as trajectory matching, multi-component adaptive knowledge transfer, and knowledge selection, all engineered to maximize information retention and generalization capability under severe resource constraints (Wang et al., 2024, Zhang et al., 6 Nov 2025).

1. Motivation and Problem Setting

The rapid expansion of Wi-Fi CSI-based human activity recognition and wireless channel modeling has led to the proliferation of extremely large datasets—e.g., XRF55 exceeding 65 GB and Widar3.0 at 5 GB. Traditional learning-based methods both require and produce large-scale models and data, resulting in prohibitive storage and training time for practical, low-latency applications, especially on resource-constrained edge devices. Standard dataset compression methods—such as coresets using K-means, K-center, or herding—yield only limited compression ratios and often display deficient generalization across different model architectures. Likewise, compressing wireless foundation models (FMs) purely by pruning or quantization is often insufficient, particularly if special hardware support is unavailable or generalization is degraded. WiDistill methods seek to produce data "distilled" to minimal size or models "fused" with maximal knowledge retention, supporting about 10× faster training and inference while preserving performance within 1–2% of original baselines (Wang et al., 2024, Zhang et al., 6 Nov 2025).

2. Dataset Distillation via Trajectory Matching

The core of WiDistill's dataset distillation (Wang et al., 2024) employs a trajectory-matching approach that operates as follows:

• Let $D_{\rm real}$ denote the full dataset and $D_{\rm syn}$ the compact synthetic dataset ($M \ll N$).
• An expert teacher model $f(\cdot;\theta_E)$ is trained on $D_{\rm real}$, recording the full parameter trajectory $\Theta_E = \{\theta_E(t)\}_{t=0}^T$.
• For student initialization, at a randomly sampled epoch $t_0$, set $\theta_S(t_0) = \theta_E(t_0)$ and perform $K$ SGD steps on $D_{\rm syn}$ with a trainable learning rate $\alpha$:

$$\theta_S(t_0+n+1) = \theta_S(t_0+n) - \alpha \nabla_\theta \ell(f(\tilde x;\theta_S(t_0+n)), \tilde y)$$

• The distillation loss matches the student's terminal trajectory to the expert's via a normalized squared-$\ell_2$ objective:

$$L_{\rm match} = \frac{ \| \theta_S(t_0+K) - \theta_E(t_0+J) \|_2^2 }{ \| \theta_E(t_0) - \theta_E(t_0+J)\|_2^2 }$$

Trajectory matching is performed over $D_{\rm syn}$, optimizing both synthetic samples and learning rate.

• This process produces a distilled set that, when used to train networks from scratch, preserves both convergence dynamics and generalization properties.

When evaluated on Widar3.0, XRF55, and MM-Fi, WiDistill achieves:

• Up to 10× reduction in training time and storage.
• $>90\%$ of original accuracy at 100 samples per class (spc).
• Robust transfer when distilled on one model (ResNet-18), then re-trained/tested with alternative architectures (MLP, CNN), with cross-network generalization drop limited to ~3–4% (Wang et al., 2024).

3. Multi-Component Adaptive Model Distillation (MCAKD) for Wireless FMs

For foundational models such as WiFo (Transformer-based wireless channel predictors), WiDistill leverages MCAKD to compress a 21.6M-parameter teacher to a 5.5M-parameter student ("Tiny-WiFo") (Zhang et al., 6 Nov 2025). MCAKD is characterized by:

• Three principal matching objectives:
  • Attention matching: Cosine similarity over student and teacher attention maps, computed across encoder and decoder attention blocks.
  • Embedding matching: Cross-attention-based knowledge selection (CA-KS) extracts teacher features most aligned with the student.
  • Hidden-state matching: Cosine-based loss on encoder and decoder hidden states (teacher vs. student).
• Aggregate loss:

$$\mathcal{L}_{\rm MCAKD} = \mathcal{L}_{\rm attn} + \mathcal{L}_{\rm embed} + \mathcal{L}_{\rm hs}$$

• CA-KS module computes multi-head attention across student and teacher embedding dimensions, selecting the top-$D_s$ teacher channels for transfer based on attention-weighted relevance.
• Autonomous Learning–Passive Learning (AL-PL): Training alternates epochs:
  • Autonomous (self-supervised, no teacher forward pass): minimize masked-reconstruction MSE.
  • Passive (distillation): minimize MSE plus $\lambda \mathcal{L}_{\rm MCAKD}$, with $\lambda=0.1$.
  • This reduces compute by avoiding redundant teacher passes and mitigates teacher over-reliance.
• Final architecture (Tiny-WiFo): 6 encoder/4 decoder layers, 8 heads, $d=256$, 5.5M parameters.
• Tiny-WiFo achieves real-time inference (1.6 ms) on Jetson AGX Orin (after quantization), $>98\%$ of WiFo's accuracy, and strong zero-shot generalization (Zhang et al., 6 Nov 2025).

4. Empirical Benchmarks and Results

The following table summarizes WiDistill's empirical performance on dataset distillation tasks (Wang et al., 2024):

Dataset	spc	WiDistill	KMeans	KCenter	Herding	Random	Full Acc.
Widar3.0	10	0.3907	0.3898	0.2648	0.2653	0.3155	0.9271
	50	0.6423	0.4768	0.3595	0.3799	0.4820	--
	100	0.6865	0.5501	0.4361	0.4432	0.5407	--
XRF55	10	0.2348	0.2197	0.1927	0.1718	0.1941	0.8905
	50	0.5736	0.6395	0.5181	0.3204	0.5006	--
	100	0.7124	0.7603	0.6520	0.4103	0.6807	--
MM-Fi	10	0.1841	0.1682	0.0507	0.0473	0.1970	0.3402
	50	0.1779	0.2485	0.1783	0.0634	0.1815	--
	100	0.2809	0.2706	0.2485	0.1012	0.2782	--

WiDistill outperforms or matches all baseline coreset selection methods at low spc, with competitive results at 100 spc (1–2% of original data). Distilled datasets decrease wall-clock training time by $\sim7$–$10\times$ and shrink disk requirements by up to 90% (Wang et al., 2024, Zhang et al., 6 Nov 2025).

For model distillation, Tiny-WiFo retains $\geq98\%$ of teacher's NMSE with $75\%$ fewer parameters and achieves 1.6 ms inference latency, supporting real-time deployment even in dynamic channels.

5. Cross-Model and Cross-Architecture Generalization

A salient feature is cross-architecture transferability: synthetic data distilled on one architecture (e.g., ResNet-18) can train a different architecture (MLP, CNN, alternative ResNet) without significant accuracy loss:

• Example (Widar3.0, spc=50):
  • Distilled on ResNet→Test MLP: 0.5434
  • Distilled on ResNet→Test CNN: 0.5356
  • Distilled on ResNet→Test ResNet: 0.5693

This demonstrates that WiDistill does not overfit to architecture-specific signals, instead condensing data in a generally usable form (Wang et al., 2024).

For MCAKD-compressed models, Tiny-WiFo generalizes robustly to channel conditions unseen during training, retaining prediction quality on new datasets (e.g., D17/D18 within 0.21 dB NMSE of WiFo) (Zhang et al., 6 Nov 2025).

6. Implementation and Practical Considerations

WiDistill prescribes:

• Removing outliers and per-channel normalization of CSI data; no computer-vision augmentations required.
• Typical hyperparameters: distillation steps $K=10$, teacher learning rate $\eta=0.01$, optimizer is SGD+momentum (0.9), distilled data fine-tuning runs 5–10 epochs with low learning rate ($1\times10^{-3}$).
• Distillation can initialize $D_{\rm syn}$ by randomly sampling or using noise, with iterative updates via standard backpropagation.

For MCAKD/Tiny-WiFo:

• Learning rate: $0.1$ for synthetic data.
• Adaptive scheduling of passive/active distillation epochs to balance computational load and knowledge transfer from the teacher.
• CA-KS for embedding transfer is mandatory when $D_t > D_s$.

7. Limitations and Perspectives

Although WiDistill demonstrates high compression ratios and cross-architecture generalization, its effectiveness may decrease with extremely low spc budgets (e.g., below 10 per class in highly imbalanced or noisy regimes). MCAKD requires teacher model accessibility and increases distillation phase complexity due to multiple objective terms and cross-attention search procedures.

Nevertheless, WiDistill and its model-level extensions represent effective solutions for deploying data-driven wireless systems and channel-prediction models in bandwidth-, compute-, or real-time-constrained environments (Wang et al., 2024, Zhang et al., 6 Nov 2025).