---
title: Split Early-Exit Neural Networks
url: https://www.emergentmind.com/topics/split-early-exit-neural-network
type: topic
---

# Split Early-Exit Neural Networks

A split early-exit neural network is a deep model that combines split computing and early exiting: an initial “head” of the network is executed on a resource-constrained device, while the remaining “tail” can be executed on an edge or cloud server, and intermediate exit mechanisms decide whether inference should terminate locally or continue remotely. In the survey literature, split computing is formulated as a partition of a model into a head and a tail, while early exiting augments internal layers with auxiliary predictors so that “easy” inputs can stop before the full depth is evaluated [2103.04505, 2004.12814]. More recent systems make the combined decision explicit: SplitEE chooses a splitting layer and then decides whether a sample exits locally or is offloaded [2309.09195], while the abstract of I-SplitEE describes an online unsupervised method that determines the splitting layer and whether to infer on the edge device or offload to the cloud, considering accuracy, computational efficiency, and communication costs [2401.10541].

## 1. Concept and scope

In split computing, a model is partitioned at some layer $\ell$, with a head $\mathcal{M}_H$ executed on the device and a tail $\mathcal{M}_T$ executed on the server:
$$
\mathbf{z}_{\ell} = \mathcal{M}_H(\mathbf{x}), \qquad \hat{\mathbf{y}} = \mathcal{M}_T(\mathbf{z}_{\ell}).
$$
In early exiting, intermediate stages produce predictions $\mathbf{y}_i = \mathcal{B}_i(\mathbf{z}_i)$, and inference can stop when an exit is sufficiently confident [2103.04505]. A split early-exit architecture therefore adds a second layer of adaptivity: it does not merely decide how deep to compute, but also where the remaining computation should take place.

This combination addresses a three-way systems trade-off. Full local inference avoids communication but can exceed device budgets. Full offloading preserves model depth but can be dominated by transmission delay and bandwidth volatility. Early exits reduce average computation but, by themselves, do not determine whether difficult samples should be continued locally or remotely. Split early-exit systems treat these choices jointly, so that local exits can suppress unnecessary transmission while remote completion remains available for hard inputs [2103.04505, 2309.09195].

A common umbrella term in the literature is “multi-output network,” which denotes architectures that attach auxiliary predictors to intermediate representations. “Early-exit network” refers to the runtime use of those outputs for adaptive stopping, while “split computation” refers to deployment across device, edge, fog, or cloud tiers [2004.12814]. Split early-exit neural networks lie at the intersection of these categories.

## 2. Architectural organization and exit criteria

The standard architectural pattern consists of a device-side prefix, one or more device-side exits, and a server-side suffix. In SplitEE, each incoming sample is processed up to a learned splitting layer $i_t$, and the confidence at that layer,
$$
C_i(x) = \max_{c \in \mathcal{C}} \hat{P}_i(c),
$$
is compared to a threshold $\alpha$. If $C_i(x) \ge \alpha$, the sample exits locally; otherwise it is offloaded and processed to the final layer $L$ [2309.09195].

Other systems refine this decision logic. In the device-edge co-inference framework with an Exit Predictor, the device first computes predictor scores $s_n$ for the on-device exits. The $n$-th exit is evaluated only if $s_n \ge \gamma_n$, and early termination occurs only if the exit confidence
$$
c_n = {\rm max}({\rm softmax}(\bm{z}_n))
$$
also satisfies $c_n \ge \lambda_n$. This changes execution from “evaluate every exit until confident” to “evaluate only exits predicted to be useful,” thereby reducing on-device overhead from the exit heads themselves [2206.07269].

In heterogeneous split learning, Hetero-SplitEE attaches a lightweight client output layer at each client’s cut layer $l_i$. For inference, the client computes probabilities
$$
p_i^{(c)} = \text{softmax}(\hat{y}_i^{(c)}),
$$
measures confidence through entropy,
$$
H_i^{(c)} = -\sum_j p_{i,j}^{(c)} \log p_{i,j}^{(c)}, \qquad C_i^{(c)} = -H_i^{(c)},
$$
and exits locally if $C_i^{(c)} > \tau$; otherwise it transmits the intermediate feature $h_i$ to the server [2512.09313]. This formulation is notable because the cut layer is not globally fixed: different clients choose different cut layers according to computational capacity.

More general early-exit CNN research clarifies the structure of exit blocks themselves. EENets use a separate confidence branch and softmax branch at each exit:
$$
\hat{y}_n = \mathrm{softmax} (\mathbf{w}_1^{T} z(\mathbf{x})), \qquad
h_n = \sigma (\mathbf{w}_2^{T} z(\mathbf{x})),
$$
with inference terminating when $h_i \ge T$, where the paper uses $T=0.5$ [2409.05336]. Although EENets are not, by themselves, split-computing systems, they provide a canonical exit-head design that later split early-exit systems either reuse or replace.

## 3. Optimization, calibration, and training

The training problem is not uniform across the literature. One family of methods treats early exits as jointly supervised auxiliary outputs. The survey on multi-output networks describes the standard joint objective as
$$
f^* = \arg\min \left\{ L + \sum_{i=1}^{L-1}\alpha_i L_i \right\},
$$
where each exit contributes its own loss term [2004.12814]. EENets add an explicit computation-aware component,
$$
\mathcal{L}=\mathcal{L}_\mathrm{MC}+\lambda \mathcal{L}_\mathrm{Cost},
$$
with a soft expected-cost recursion
$$
C_i = h_i c_i + (1-h_i) C_{i+1},
$$
so that accuracy and FLOP cost are optimized together [2409.05336].

SplitEE instead formulates split-layer selection as an online bandit problem. Let $\gamma_i$ be the computation cost up to layer $i$, $o$ the offloading cost, and $\mu$ the cost-conversion factor. The reward for splitting at layer $i$ is
$$
r(i) = \begin{cases}
C_i - \mu \gamma_i, & \text{if } C_i \ge \alpha \text{ or } i=L, \\
C_L - \mu(\gamma_i + o), & \text{otherwise}.
\end{cases}
$$
The optimal split layer is
$$
i^* = \arg\max_{i \in [L]} \mathbb{E}[r(i)],
$$
and SplitEE uses an Upper Confidence Bound rule
$$
i_t = \arg\max_{i \in [L]} \left( Q(i) + \beta \sqrt{\frac{\ln t}{N(i)}} \right)
$$
to learn that policy online without labels [2309.09195]. This is a distinct shift from supervised confidence-threshold tuning to streaming, unsupervised adaptation.

A different line of work argues that gating and intermediate predictors should not be trained separately. JEI-DNN states that decoupled gating creates a train-test mismatch, because intermediate modules are trained on the full dataset but, at inference time, only see the subset routed to them. It therefore jointly learns exit probabilities and intermediate inference modules under an expected cost-aware objective
$$
L \approx \frac{1}{N}\sum_{i=1}^N \sum_{l=1}^L \Big(L^{CE}(y_i,\hat p_\theta^l(x_i))+\lambda IC_{\text{norm}^l}\Big)P(G=l\mid x_i)
$$
[2310.09163]. This position contrasts with modular approaches but does not eliminate them: in pruning experiments on Shallow-Deep Networks, joint pruning generally performs better overall, while ordered pruning of the backbone and then the exit heads becomes similar at high accuracy rates, implying that pruning and early exit can sometimes be separated without loss of optimality in that regime [2207.03644].

Compression-aware design extends the same question to model topology. The predefined sparsity approach for split computing and early exit fixes a sparse connectivity pattern before training rather than pruning after training, with junction edge counts
$$
|W_i| = N_{i-1}\cdot d^{out_i} = N_i \cdot d^{in_i}.
$$
This reduces storage and computation throughout training and inference while preserving the head–tail–exit deployment logic [2407.11763].

## 4. Distributed, edge-cloud, and heterogeneous execution

Split early-exit networks are used in several deployment regimes beyond simple one-device/one-server inference. In hierarchical training, early exits are used not for adaptive inference but to separate the backward pass across edge and cloud. The edge computes a local loss from its early exit, updates its own parameters without gradients from the cloud, and transmits only an intermediate feature map. The cloud computes its own forward and backward passes independently. This removes backward-pass communication and enables overlap between edge backward computation and cloud computation [2303.02384].

Heterogeneous IoT training pushes the idea further. In Hetero-SplitEE, each client $i$ chooses its own cut layer $l_i$, computes
$$
h_i = f_i^{(c)}(x), \qquad \hat{y}_i^{(c)} = f_i^{(o)}(h_i),
$$
and trains a local early-exit branch while the server trains the deeper continuation. Two cooperative strategies are proposed. Sequential training uses one shared server-side model and processes client updates sequentially. Averaging assigns each client its own server-side model and performs cross-layer aggregation
$$
\bar{\theta}_l^{(t)} = \frac{1}{|\mathcal{C}_l|} \sum_{i \in \mathcal{C}_l} \theta_{l,i}^{(t)},
$$
for layers present in multiple server-side subnetworks [2512.09313]. The defining feature is that heterogeneous cut layers are treated as first-class design variables rather than as noise around a fixed global split.

Distributed collaborative inference also appears outside client-server training. DistrEE assumes an edge cluster $D=\{d_1,d_2,\ldots,d_N\}$ with student models derived from a teacher network, and uses a feature-difference exit criterion
$$
diff(F_j,F_1)=\frac{1}{sim_{cos}(F_j,F_1)}=\frac{\|F_j\|\cdot\|F_1\|}{F_j\cdot F_1}.
$$
Inference exits at branch $j$ when $diff_{cos}(F_j,F_1) > t_j$ [2502.15735]. This is notable because it does not rely on class-probability confidence; instead, it uses the maturity of an internal representation relative to the shallowest exit.

The same architectural pattern has also been instantiated in network security. In a P4-SDN collaborative DDoS detector, a quantized CNN and early-exit classifier run in the P4 data plane, while a GRU backend in the SDN controller handles uncertain cases. The switch exits locally if the CNN output lies outside the uncertainty interval defined by precomputed logit thresholds; otherwise it forwards a compact sequence of feature vectors to the controller [2509.12291]. Here the split early-exit design is realized as a confidence-gated switch/controller cascade rather than as a mobile/cloud application.

## 5. Empirical behavior and application domains

Reported results span image classification, NLP, DDoS detection, split learning, and training-time acceleration. The literature does not present a single benchmark protocol, but it consistently treats split early-exit design as an accuracy–computation–communication trade-off rather than as a pure accuracy-maximization problem.

| System | Domain/setup | Reported outcome |
|---|---|---|
| I-SplitEE [2401.10541] | Caltech-256 and Cifar-10 with varied distortions | minimum 55% cost reduction with at most 5% performance degradation |
| SplitEE [2309.09195] | Five NLP datasets with ElasticBERT | significant cost reduction ($>50\%$) with slight accuracy drop ($<2\%$) |
| Exit Predictor [2206.07269] | CIFAR100, VGG16-BN, Raspberry Pi 3, varying bandwidth | meets the 30 ms latency target across bandwidths from 0.1 to 100 Mbit/s |
| Predefined sparsity with SC and EE [2407.11763] | Sparse MLP split/exit models on MNIST | more than \(4\times\) reduction in storage and computational complexity without compromising performance |
| Hetero-SplitEE [2512.09313] | CIFAR-100, homogeneous Layer-3 setting | Sequential server accuracy 39.37% versus 25.81% for distributed training |
| Hierarchical training with early exits [2303.02384] | VGG-16 and ResNet-18, low bit rate channel | runtime reduced by 29% and 61% on CIFAR-10, and by 25% and 81% on Tiny ImageNet |

These results show that the empirical advantage can arise from different sources. In SplitEE and I-SplitEE, the benefit is mainly online selection of the split and exit/offload policy under shifting conditions [2309.09195, 2401.10541]. In the Exit Predictor and predefined sparsity work, the benefit comes from suppressing local exit overhead or reducing the size of the head model itself [2206.07269, 2407.11763]. In Hetero-SplitEE and hierarchical training, the gain is as much about training or system organization as about inference latency [2512.09313, 2303.02384].

The application range is correspondingly broad. The survey literature emphasizes computer vision, edge inference, and 5G/fog environments [2103.04505, 2004.12814]. Subsequent work extends the paradigm to NLP with ElasticBERT [2309.09195], to heterogeneous IoT training with ResNet-18 [2512.09313], and to real-time DDoS mitigation in programmable networks [2509.12291]. This suggests that split early-exit logic is better understood as a systems pattern than as a modality-specific architecture.

## 6. Design tensions, misconceptions, and open problems

A common misconception is that early exiting is equivalent to thresholding the maximum softmax probability. That description matches some systems, but the literature is materially broader. EENets learn an explicit confidence branch rather than deriving confidence heuristically from class scores [2409.05336]. Hetero-SplitEE uses entropy-based confidence [2512.09313]. DistrEE uses feature difference instead of output confidence [2502.15735]. EERO formulates the choice of exit head as classification with reject option and calibrates the classification probabilities across heads under a fixed budget by exponential weights, rather than by ad hoc threshold choice [2402.03779].

A second misconception is that the split point is an architecture constant. The split-computing survey notes that, without bottleneck injection, the best split in many tested models was often at the input or output, meaning that naive partitioning can collapse to full local or full edge inference [2103.04505]. SplitEE therefore learns the split point online [2309.09195], while Hetero-SplitEE allows distinct cut layers across clients [2512.09313]. This suggests that “the” optimal split is usually conditional on workload, bandwidth, device heterogeneity, and data distribution.

The literature also disagrees, in a constructive way, on how tightly the components should be optimized together. JEI-DNN argues that decoupled training of gates and intermediate modules creates train-test mismatch [2310.09163]. By contrast, pruning experiments show that separated pruning of the backbone and exits can be close to joint pruning in the high-accuracy regime [2207.03644]. EERO adds a third perspective by treating head selection as a constrained statistical allocation problem with explicit risk and budget terms [2402.03779]. The unresolved issue is not whether joint optimization can help, but when its additional complexity is justified.

Open problems identified in the survey literature include more realistic evaluation beyond small datasets, better bottleneck design and placement, automatic threshold selection, expansion to broader sensing domains, and stronger information-theoretic foundations for the accuracy–compression trade-off [2103.04505]. More recent work adds online unsupervised adaptation under deployment shift: the abstract of I-SplitEE explicitly targets sequential data without ground truths and environmental distortions such as time of day, lighting, and weather [2401.10541]. A plausible implication is that future split early-exit systems will be judged less by static average FLOPs and more by how stably they adapt to changing resource and data conditions.

Source: https://www.emergentmind.com/topics/split-early-exit-neural-network