EKDC-Net: Compact Ensemble Distillation

• EKDC-Net is a convolutional neural network that employs multiple parallel student branches to distill and aggregate knowledge from an ensemble of heterogeneous teacher networks.
• Its architecture features explicit logit-level ensembling and a multi-term loss function that combines hard-label classification with knowledge distillation, promoting robust learning.
• Evaluated on CIFAR datasets, EKDC-Net balances high generalization and improved top-1 accuracy with efficient inference, making it a practical approach to model compression.

EKDC-Net (Ensemble Knowledge Distillation Compact Network) is a convolutional neural network framework for learning compact student models with improved classification accuracy and generalization, designed to transfer ensemble knowledge from multiple teacher networks. It achieves the dual goal of leveraging ensemble teacher supervision while maintaining high efficiency through parallel student branches. EKDC-Net is characterized by a student architecture comprising multiple parallel branches, a precise ensembling strategy, and a multi-term training objective that combines classification and knowledge distillation losses, supporting efficient inference and robust learning in settings such as image classification (Asif et al., 2019).

1. Student Network Architecture

The EKDC-Net student (“CompNet”) is constructed as a sum-ensemble of $N_s$ parallel branches. Each branch processes the input $x \in \mathbb{R}^{H \times W \times 3}$ independently using a fixed sequence of convolutional and residual layers. For CIFAR experiments ($H=W=32$), each branch computes as follows:

• Initial convolutional layer: $3 \times 3$ kernel, $C$ output channels (typ. $C=16$), BatchNorm, ReLU.
• Three residual blocks: Each block contains two $3\times3$ convolutions (with BN and ReLU), a skip (identity) connection, and a $2\times2$ pooling with stride $2$. The typical feature map sizes across the spatially downsampled sequence are $32\times32\times16$, $x \in \mathbb{R}^{H \times W \times 3}$0, $x \in \mathbb{R}^{H \times W \times 3}$1, $x \in \mathbb{R}^{H \times W \times 3}$2.
• Global Average Pooling (GAP): Compresses spatial dimensions, yielding a $x \in \mathbb{R}^{H \times W \times 3}$3 feature vector.
• Fully connected (FC) output: $x \in \mathbb{R}^{H \times W \times 3}$4 for $x \in \mathbb{R}^{H \times W \times 3}$5 classes.

All $x \in \mathbb{R}^{H \times W \times 3}$6 branch logits are summed to obtain the student ensemble logit vector:

$x \in \mathbb{R}^{H \times W \times 3}$7

A softmax operation on $x \in \mathbb{R}^{H \times W \times 3}$8 yields the final class probabilities. No additional weighting per branch is used, ensuring each branch’s equal contribution in both training and inference. The architecture omits dropout and applies BatchNorm after every convolution.

2. Ensembling and Branch Coupling Strategy

EKDC-Net’s design uses explicit ensembling at the logit level by summing the outputs from all student branches (Eq. 1). This is performed for every forward pass, both during training and inference, with the final prediction computed as:

$x \in \mathbb{R}^{H \times W \times 3}$9

All branches are architecturally identical and independently process the same input. The branch outputs are not directly coupled via explicit loss terms; rather, coupling is induced implicitly through the knowledge distillation losses that involve both individual branches and their sum, promoting both diverse (per-branch) and collaborative (ensemble) learning.

3. Training Objective and Loss Terms

EKDC-Net utilizes a multi-term loss integrating hard-label supervision and knowledge distillation from an ensemble of $H=W=32$0 teacher networks. Each teacher $H=W=32$1 yields its own output $H=W=32$2; these logits are summed to form the teacher ensemble logit $H=W=32$3. The overall training loss is:

$H=W=32$4

where $H=W=32$5 is the ground-truth label, and $H=W=32$6, $H=W=32$7, $H=W=32$8 balance the loss terms (in practice: $H=W=32$9, $3 \times 3$0, $3 \times 3$1). The terms are defined as:

• Classification loss:

$3 \times 3$2

applied to both $3 \times 3$3 (student) and $3 \times 3$4 (teacher ensemble) versus $3 \times 3$5.

• Knowledge Distillation loss:

Using a temperature $3 \times 3$6 (typically $3 \times 3$7) to soften logit distributions,

$3 \times 3$8

where $3 \times 3$9, $C$0, $C$1, $C$2, $C$3 denotes Kullback–Leibler divergence, and $C$4 is the squared Euclidean ($C$5) distance.

This construction jointly penalizes distinctions between the student ensemble and the teacher ensemble at the distributional and logit level (ensemble-to-ensemble), and between matching branches (branch-to-branch), the latter being performed between corresponding student-teacher pairs.

The ensemble-coupling effect emerges from the $C$6 and $C$7 terms—these encourage the aggregate of all student branches to reproduce the behavior of the teacher ensemble, thereby reducing variance among student branch outputs and promoting collaborative learning.

4. Teacher Ensemble and Heterogeneity

The teacher component consists of an ensemble of $C$8 independently trained networks, each possibly varying in model depth or channel width (the paper cites e.g., ResNet14 through ResNet110 as valid choices). During student training, each teacher $C$9 produces its own branch output $C=16$0 in an identical forward pattern and all are summed to $C=16$1.

This use of a heterogeneous teacher ensemble enhances the diversity of the knowledge transferred to the student branches. Each branch in the student network learns from the corresponding teacher branch via knowledge distillation, promoting heterogeneity in learned representations. The effect is to enable the compact student model to exploit an efficient approximation to ensemble learning, yielding increased generalization and accuracy, especially in low-data regimes.

5. Hyperparameterization and Training Protocol

Key design and training parameters are as follows:

Parameter	Typical Values	Notes
$C=16$2 (student branches)	1–7	Accuracy increases with $C=16$3; more FLOPs
Channel width $C=16$4	16 (on CIFAR)	Fixed across all branches
Per-branch depth	1 initial conv, 3 residual blocks	Each residual block ≈2 convs + skip + pool
$C=16$5 (teacher ensemble)	Up to 7	Heterogeneous ResNet depths possible
Temperature $C=16$6	10	For softened softmax in distillation
Loss weights $C=16$7	0.5, 0.5, 0.6	From hyperparameter search
Weight decay	$C=16$8	On all weights
Initialization	Xavier/Gaussian (σ=0.01), bias=0	No dropout
Optimizer	Adam, LR = 0.01, decay at 50%/75% (+0.1×), 500 epochs

Batch Normalization is applied in every convolution layer, consistently across all branches and teachers. The optimizer is Adam with scheduled learning rate drops; no learning rate warmup is indicated. The regularization regime omits dropout entirely, relying on weight decay and architecture-intrinsic regularization.

6. Experimental Context and Significance

EKDC-Net was evaluated in image classification settings on CIFAR-10 and CIFAR-100. It demonstrated improved top-1 accuracy and generalization performance compared to other state-of-the-art knowledge distillation frameworks, particularly when the available labeled training data is limited. Notable gains were observed in mean accuracy on test datasets. The design allows practitioners to balance inference cost against accuracy by adjusting the number of student branches $C=16$9. The integration of teacher ensemble knowledge into a compact student with parallel branches is central to the observed improvements (Asif et al., 2019).

A plausible implication is that EKDC-Net represents an effective approach for practitioners who require efficient inference with an ensemble-like performance, making strategic use of distillation from heterogeneous and potentially larger teacher ensembles without incurring their full computational cost at deployment. This architecture is modular and parameter-efficient, with design choices that are directly supported by empirical results on benchmark datasets.