Recursive Early-Exit Framework

• Recursive Early-Exit is a neural network inference framework that enables dynamic, goal-oriented decision-making via recursive multi-exit architectures.
• It integrates reinforcement learning-based policies to adaptively decide early exits, offloading, or continuation to optimize accuracy, latency, and wireless resource usage.
• Empirical results on CIFAR-10 with ResNet20 show 30% FLOP reduction or 1–2% accuracy gains, demonstrating its effectiveness for edge inference under time constraints.

Recursive early-exit is a framework for neural network inference designed to enable dynamic, goal-oriented decision-making in edge-device and server-collaborative scenarios. The approach combines a novel recursive mechanism for multi-exit neural architectures with reinforcement learning-based policies to adaptively determine when to halt computation, where to partition workloads, and whether to offload embeddings for remote inference. This framework jointly optimizes inference accuracy, computational latency, and wireless resource consumption, and is particularly suited to edge inference settings with fluctuating resource availability and strict real-time constraints (Pomponi et al., 2024).

1. Recursive Multi-Exit Neural Network Structure

The foundational component is a base model consisting of $b$ sequential blocks (convolutional or residual), noted as $\ell_1, \ldots, \ell_b$, followed by a terminal classifier $c_b$. In conventional form, inference is computed as:

$$f_b(x) = c_b \circ \ell_b \circ \ell_{b-1} \circ \cdots \circ \ell_1(x)$$

To enable early inference, a selected subset of indices $\mathcal{I} \subset \{1, \ldots, b-1\}$ is chosen for auxiliary "early-exit" branches. Each branch at index $i \in \mathcal{I}$ attaches an exit-specific classifier $c_i$, itself decomposed into a feature-processing head $e_i(x)$ (e.g., convolution + pooling) and a linear projection $p_i(\cdot)$ to output class scores.

Recursive prediction is implemented as follows. Let $f_i(x) \in [0,1]^{|\mathcal{C}|}$ denote the class probability vector at exit $\ell_1, \ldots, \ell_b$0. The combination mechanism is:

• For $\ell_1, \ldots, \ell_b$1,

$\ell_1, \ldots, \ell_b$2

• For $\ell_1, \ldots, \ell_b$3,

$\ell_1, \ldots, \ell_b$4

• For $\ell_1, \ldots, \ell_b$5,

$\ell_1, \ldots, \ell_b$6

Here, $\ell_1, \ldots, \ell_b$7 is the "moving mass" term that reallocates class probabilities:

$\ell_1, \ldots, \ell_b$8

with $\ell_1, \ldots, \ell_b$9 defined as linear plus sigmoid heads per exit. This recursive update allows intermediate exits to refine and compensate for class assignments based on new representations, yielding a path-dependent and efficiently updatable probability vector at each exit.

2. Confidence Evolution and Early-Exit Criterion

At each exit $c_b$0, the model outputs class probabilities $c_b$1. The instantaneous confidence is defined as

$c_b$2

The halting criterion employs a margin-based rule: denoting $c_b$3 and $c_b$4 as indices of the largest and second-largest entries of $c_b$5, the system halts at $c_b$6 if

$c_b$7

where $c_b$8 is a user-specified safety margin. This ensures halting only when the model is sufficiently more confident in the top prediction compared to subsequent candidates.

Alternative schemes used in classical early-exit models include simple thresholds $c_b$9 and difference-based strategies $$f_b(x) = c_b \circ \ell_b \circ \ell_{b-1} \circ \cdots \circ \ell_1(x)$$0 in combination with $$f_b(x) = c_b \circ \ell_b \circ \ell_{b-1} \circ \cdots \circ \ell_1(x)$$1. The margin-based halting rule, however, generalizes to the multiclass scenario and directly encodes the desired separation between leading predictions.

3. Reinforcement Learning-Based Online Exit Policy

The joint optimization of exit decisions, computation partitioning, and offloading is formulated as a Markov Decision Process (MDP), with each inference request constituting an episode of up to $$f_b(x) = c_b \circ \ell_b \circ \ell_{b-1} \circ \cdots \circ \ell_1(x)$$2 decision steps (number of exits). The state at step $$f_b(x) = c_b \circ \ell_b \circ \ell_{b-1} \circ \cdots \circ \ell_1(x)$$3 is

$$f_b(x) = c_b \circ \ell_b \circ \ell_{b-1} \circ \cdots \circ \ell_1(x)$$4

where $$f_b(x) = c_b \circ \ell_b \circ \ell_{b-1} \circ \cdots \circ \ell_1(x)$$5 indexes the current exit, and $$f_b(x) = c_b \circ \ell_b \circ \ell_{b-1} \circ \cdots \circ \ell_1(x)$$6 is the instantaneous wireless Modulation & Coding Scheme selected per the observed channel SNR.

At each state, agent actions $$f_b(x) = c_b \circ \ell_b \circ \ell_{b-1} \circ \cdots \circ \ell_1(x)$$7:

• $$f_b(x) = c_b \circ \ell_b \circ \ell_{b-1} \circ \cdots \circ \ell_1(x)$$8 (exit now): output prediction at exit $$f_b(x) = c_b \circ \ell_b \circ \ell_{b-1} \circ \cdots \circ \ell_1(x)$$9 locally,
• $\mathcal{I} \subset \{1, \ldots, b-1\}$0 (continue): compute $\mathcal{I} \subset \{1, \ldots, b-1\}$1 and evaluate next exit,
• $\mathcal{I} \subset \{1, \ldots, b-1\}$2 (offload): transmit current embedding to server and terminate.

Transitions for $\mathcal{I} \subset \{1, \ldots, b-1\}$3 yield a new state with incremented $\mathcal{I} \subset \{1, \ldots, b-1\}$4 and a fresh channel sample; $\mathcal{I} \subset \{1, \ldots, b-1\}$5 are terminal.

Terminal rewards aggregate three key performance indicators:

• Computation saving: $\mathcal{I} \subset \{1, \ldots, b-1\}$6
• Communication saving: $\mathcal{I} \subset \{1, \ldots, b-1\}$7 (embedding size in bits at $\mathcal{I} \subset \{1, \ldots, b-1\}$8; set to 1 if $\mathcal{I} \subset \{1, \ldots, b-1\}$9)
• Goal-effectiveness: If total delay $i \in \mathcal{I}$0 and margin $i \in \mathcal{I}$1, then indicator $i \in \mathcal{I}$2.

$i \in \mathcal{I}$3

Tabular $i \in \mathcal{I}$4-learning is used to optimize policy $i \in \mathcal{I}$5, enabling adaptive selection of exit, continuation, or offloading actions under varying device, channel, and task constraints.

4. Computation Partitioning and Offloading Strategy

Computation splitting is operationalized by model partitioning: after evaluating up to exit $i \in \mathcal{I}$6, the device has performed $i \in \mathcal{I}$7 FLOPs and holds the corresponding $i \in \mathcal{I}$8-bit embedding. At this point, it can either:

• Exit locally, transmitting zero data and finalizing inference, or
• Offload the $i \in \mathcal{I}$9-bit embedding to the server, which then computes the remaining $c_i$0 FLOPs.

The detailed delay components are:

• Local compute delay: $c_i$1
• Transmission delay: $c_i$2, with $c_i$3, $c_i$4 MHz, $c_i$5 as instantaneous rate
• Remote compute delay: $c_i$6
• Total delay: $c_i$7

The wireless environment is modeled with path-loss plus Rayleigh fading. At each step, a fresh SNR determines the maximum achievable MCS and data rate, directly affecting $c_i$8 and thus $c_i$9.

5. Empirical Evaluation and Resource-Utility Trade-offs

The recursive early-exit method was benchmarked using CIFAR-10 and a ResNet20 backbone with nine early exits. Performance was compared against two canonical baselines:

• Highest-probability halting (softmax threshold $e_i(x)$0 on top-class probability)
• Patience-based halting (exit after $e_i(x)$1 consecutive exits agree on predicted class)

Results indicate that the recursive-mass approach dominated both baselines, delivering the same accuracy with roughly 30% fewer FLOPs, or yielding 1–2% higher accuracy for a fixed FLOPs budget.

In an edge inference scenario with randomized device locations (10–100m from access point), 20 MHz @3.5 GHz, $e_i(x)$2 W, device/server compute times of 50/10 ms, and delay constraint $e_i(x)$3 ms, margin thresholds $e_i(x)$4 and communication weight $e_i(x)$5 were varied. Trade-off curves illustrate that as $e_i(x)$6 (the emphasis on communication savings) increases, the RL policy shifts from early offloading (maximal goal-effectiveness, low resource savings) to deeper local exits (higher computation saving, lower communication saving), eventually trading off goal-effectiveness if pushed to extremes.

Exit index distributions as a function of resource requirements showed that under strict communication-saving targets, the agent favored deep local exits; under abundant resources, early offloading was preferred for optimal accuracy. The learned policy effectively adapts exit, split, and offload behavior in real time, balancing accuracy, latency, and resource consumption.

6. Key Components and Hyperparameter Considerations

The recursive early-exit framework encompasses:

• A recursive, multi-exit network structure updating class probabilities per layer via moving mass
• A margin-based early-exit rule generalizing conventional confidence-based halting
• Integration of early-exit, computation partitioning, and offloading as a joint MDP with RL optimization
• Practical wireless channel and latency models for realistic deployment settings
• Tabular Q-learning policy for real-time adaptive exit, split, and offload decision-making

Hyperparameters including safety margins ($e_i(x)$7), reward coefficients ($e_i(x)$8), maximum delay ($e_i(x)$9), margin threshold ($p_i(\cdot)$0), available MCS values, and device/server processing rates are all explicitly defined and tunable. These enable reproducibility and facilitate adaptation to alternative architectures, datasets, and wireless environments (Pomponi et al., 2024).