Skip2-LoRA: Fast Fine-Tuning for Edge Devices

• Skip2-LoRA is a lightweight fine-tuning method that uses parallel low-rank adapter insertions and forward-pass caching to speed up training on resource-constrained devices.
• It reduces fine-tuning time by up to 90% while keeping accuracy within 1–2% of full fine-tuning, as demonstrated on hardware like the Raspberry Pi Zero 2 W.
• The method focuses training on low-rank adapter matrices, significantly lowering computational and memory demands for deploying modest DNNs on the edge.

Skip2-LoRA is a lightweight fine-tuning method for deep neural networks (DNNs), optimized for deployment on low-cost edge devices where compute and memory resources are severely constrained. It introduces a modified low-rank adaptation (LoRA) architecture that enables efficient backpropagation and forward-pass computation by combining parallel adapter insertions with a forward-pass caching strategy. The method achieves up to a 90% reduction in fine-tuning time compared to standard LoRA implementations with comparable trainable parameter counts, all while maintaining accuracy within 1–2% of full fine-tuning approaches. Skip2-LoRA has been empirically validated on Raspberry Pi Zero 2 W single-board computers and tested on small-to-moderate DNNs with practical datasets such as fan vibration classification and human activity recognition (Matsutani et al., 2024).

1. Background and Motivation

The original LoRA (Low-Rank Adaptation) framework reduces the number of trainable parameters by introducing a low-rank update to fixed pre-trained weights. For a layer with pre-trained weight $W$, LoRA sets $W' = W + BA$, where $B \in \mathbb{R}^{N \times r}$, $A \in \mathbb{R}^{r \times M}$, and $r \ll \min(N,M)$. This approach allows fine-tuning by training only the small matrices $A$ and $B$ while keeping $W$ fixed. Despite reduced parameter count, standard LoRA (LoRA-All) still requires a full forward and backward pass on every layer that hosts a LoRA adapter, often exceeding the compute/memory budget of microcontrollers and low-cost edge hardware.

Skip2-LoRA addresses these constraints by introducing two orthogonal improvements:

• Skip-LoRA architecture: All low-rank adapters are attached in parallel off the last (output) layer, so only the final layer’s output path remains active during adaptation.
• Skip-Cache: Activations of all frozen network layers are cached after their first computation, dramatically reducing unnecessary recomputation across epochs for the same samples.

This composite approach is specifically tailored for scenarios with limited hardware capabilities, delivering significant speedup without sacrificing expressive power or adaptation capacity (Matsutani et al., 2024).

2. Methodological Framework

2.1 Skip-LoRA Network Architecture

Consider an $n$-layer fully connected DNN, with input to layer $k$ denoted $W' = W + BA$0 and output $W' = W + BA$1:

$W' = W + BA$2

where $W' = W + BA$3 denotes nonlinearity, typically batch-norm and ReLU.

Skip-LoRA modifies adaptation by keeping all original $W' = W + BA$4 frozen while attaching trainable low-rank pairs $W' = W + BA$5 for $W' = W + BA$6 in parallel to the last layer. The final layer output is corrected as:

$W' = W + BA$7

As training propagates backward, only the $W' = W + BA$8, $W' = W + BA$9 matrices are updated, yielding a significant reduction in computational and memory complexity relative to LoRA-All, which introduces LoRA adapters in every layer.

2.2 Forward-Pass Caching (Skip-Cache)

Notations:

• $B \in \mathbb{R}^{N \times r}$0 — set of all fine-tuning samples, $B \in \mathbb{R}^{N \times r}$1.
• $B \in \mathbb{R}^{N \times r}$2 — number of training epochs ($B \in \mathbb{R}^{N \times r}$3 typical).
• $B \in \mathbb{R}^{N \times r}$4 — cached output $B \in \mathbb{R}^{N \times r}$5 of layer $B \in \mathbb{R}^{N \times r}$6 for sample $B \in \mathbb{R}^{N \times r}$7.

Algorithmic steps:

1. For each minibatch and each layer $B \in \mathbb{R}^{N \times r}$8, if $B \in \mathbb{R}^{N \times r}$9 exists, reuse it; otherwise, compute and store it.
2. Since $A \in \mathbb{R}^{r \times M}$0 are frozen for $A \in \mathbb{R}^{r \times M}$1, caches remain valid and forward computation is bypassed after the initial epoch for each sample.
3. For the final layer and the low-rank adapter corrections, computation is always performed, as these parameters are trained.

The forward cost per sample thus asymptotically drops by approximately $A \in \mathbb{R}^{r \times M}$2, yielding nearly zero cost after epoch one.

2.3 Training Pseudocode

$r \ll \min(N,M)$6

3. Implementation and Hardware Considerations

Skip2-LoRA was implemented and evaluated on the Raspberry Pi Zero 2 W, featuring a 1 GHz ARM Cortex-A53 processor and 512 MiB RAM (approximately $15 USD). The software stack was restricted to plain C with only the standard math library (libm), compiled using gcc 8.3.0 with –O3 and NEON SIMD activation.

For the Damage1 dataset ($A \in \mathbb{R}^{r \times M}$3 samples, each $A \in \mathbb{R}^{r \times M}$4 up to 96 float32 values), the full cache consumed approximately 358 KiB, which was less than the input data size; sample-to-cache lookup was an $A \in \mathbb{R}^{r \times M}$5 array-index by sample ID.

Performance metrics per batch (batch size $A \in \mathbb{R}^{r \times M}$6) demonstrated:

Operation	Skip2-LoRA	LoRA-All
Forward pass (ms)	0.3	2.8
Backward (ms)	0.13	1.1
Weight update (ms)	0.01	—

Peak power measured at ≈1.45 W with thermal values remaining below 44.5 °C during a 2.8 s end-to-end run.

4. Empirical Evaluation

4.1 Datasets and Model Structures

• Damage1 / Damage2: 3-class fan vibration, 256–96–96–3 FC network, $A \in \mathbb{R}^{r \times M}$7, rank $A \in \mathbb{R}^{r \times M}$8.
• HAR: 6-class human activity, 561–96–96–6 FC network, $A \in \mathbb{R}^{r \times M}$9, rank $r \ll \min(N,M)$0.

4.2 Accuracy

Skip2-LoRA achieves accuracy within 1–2% of LoRA-All and FT-All, and matches or slightly outperforms LoRA-Last/FT-Last. For example, on Damage1: FT-All 98.7%, LoRA-All 98.3%, Skip2-LoRA 96.2%. On HAR, Skip2-LoRA is comparable to TinyTL on a ProxylessNAS backbone.

4.3 Speed and Total Training Time

Method	Fan (ms)	HAR (ms)
LoRA-All	4.11	7.46
Skip-LoRA	2.95	6.33
Skip2-LoRA	0.45	0.60

Skip2-LoRA delivers a 90%+ speedup over LoRA-All (equal trainable parameters). Complete training times were sub-second for all considered tasks: Damage1 ($r \ll \min(N,M)$11.06 s, 100 epochs), Damage2 ($r \ll \min(N,M)$20.64 s, 60 epochs), HAR ($r \ll \min(N,M)$32.8 s, 200 epochs).

5. Comparative Analysis and Limitations

Trade-offs

• Cache size versus speed: Full per-sample, per-layer caching achieves maximum speed but requires approximately $r \ll \min(N,M)$4 bytes. If memory is limited, a smaller key-value cache may be adopted, trading cache hit rate and speed.
• Invariance of frozen layers: The caching mechanism only applies when layers upstream of the adapters remain unchanged. Any method that updates biases or weights before the last layer (e.g., FT-All, FT-Bias, LoRA-All) invalidates the cache after each batch.

Scalability and Generalization

• For very large $r \ll \min(N,M)$5 (e.g., streaming/online learning), the cache size can become prohibitive. LRU or approximate hash-based caches are potential mitigation strategies.
• For highly non-stationary data or when each sample occurs only once, the benefit of caching diminishes.

Extension Possibilities

Potential avenues include adaptive rank selection per adapter/layer, extending the framework to convolutional layers (channel-wise feature map caching), lower-precision cache representations, and approximate hashing for scalable key-value caches.

6. Significance and Prospects

Skip2-LoRA provides a practical solution for low-cost on-device fine-tuning, occupying a “sweet spot” between expressive adaptation and computational feasibility. By combining multi-layer LoRA’s richness with the backward simplicity of last-layer fine-tuning and epoch-level forward-pass caching, it enables end-to-end adaptation of modest DNNs on hardware as constrained as the $15 Raspberry Pi Zero 2 W using only several hundred kilobytes of additional memory and with total per-run times on the order of seconds. It thus substantially expands the envelope of feasible DNN personalization and adaptation on embedded and edge devices (Matsutani et al., 2024).