SlideFormer: Single-GPU LLM Tuning

• SlideFormer is a system that fine-tunes large language models on single GPUs using a sliding window approach over model layers.
• It employs a layer-sliding pipeline, heterogeneous memory management, and optimized Triton kernels, achieving throughput gains up to 6.27×.
• The design reduces GPU memory usage by approximately 50% and allows fine-tuning of models exceeding 123B parameters on consumer-grade hardware.

SlideFormer is a system for efficient fine-tuning of LLMs on single-GPU platforms, utilizing a co-design of asynchronous scheduling, heterogeneous memory management, and custom compute kernels to significantly reduce memory usage and increase throughput. By treating the GPU as a “sliding window” over model layers, SlideFormer supports full-parameter fine-tuning of models exceeding 123B parameters on consumer hardware such as the RTX 4090, enabling up to 8× larger batch sizes and 6× larger model footprints compared to previous single-GPU solutions. Its key contributions include a layer-sliding pipeline, highly efficient multi-tier memory management, optimized Triton kernels, and a streamlined I/O strategy, collectively yielding 1.40×–6.27× throughput gains and substantial reductions in peak memory usage versus established frameworks (Yang et al., 17 Mar 2026).

1. Architectural Principles and Layer-Sliding Pipeline

The architectural foundation of SlideFormer is the interpretation of the GPU as a “sliding window” over the LLM’s layers. Rather than loading the entirety of model parameters and gradients into GPU VRAM, SlideFormer maintains a fixed queue of GPU cache slots, each sized to fit the parameters and gradients for a single layer in FP16/BF16 precision.

During forward and backward passes, only the parameters for the current layer reside in GPU VRAM. After computing the backward pass for layer $i$, its gradients are offloaded to the CPU (device-to-host, d2h), and the corresponding GPU cache slot is reused for layer $i-1$. This mechanism enables a substantial reduction in memory footprint.

A lightweight asynchronous engine orchestrates the process across three CUDA streams:

• C0: Forward and backward computation.
• C1: CPU-to-GPU (h2d) parameter prefetch.
• C2: GPU-to-CPU (d2h) gradient offload.

Two CPU threads complement the pipeline:

• Transfer thread: Manages h2d/d2h on pinned host buffers.
• Optimizer thread: Performs Layer-Adam updates on host-resident parameter masters.

Pipeline overlap is formalized by the following conditions:

• Forward step achieves lossless overlap if $T_\mathrm{fwd} \ge T_\mathrm{param\_h2d}$.
• Backward step achieves lossless overlap if $$T_\mathrm{bwd} \ge T_\mathrm{grad\_d2h} + T_\mathrm{update}$$.

The pseudocode governing this scheduling logic is:

$T_\mathrm{fwd} \ge T_\mathrm{param\_h2d}$5

2. Heterogeneous Memory Management

SlideFormer advances the memory management paradigm for single-GPU LLM fine-tuning by refining the classical model:

$$\text{Mem}_\text{req} \approx 2N \text{ (params)} + 2N \text{ (grads)} + 12N \text{ (Adam states + FP32 masters)} + O(nhsb) \text{ (activations)}$$

By contrast, baseline approaches (e.g., ZeRO-Offload) require peak GPU memory $M_\mathrm{base} = 16N + M_\mathrm{act}$. SlideFormer, by only caching parameters and gradients for individual layers, reduces peak usage to:

$$M_\mathrm{GPU} = \max_\ell(2N_\ell) + M_{\mathrm{act,window}}$$

where $N_\ell=N/n$ for a model with $n$ layers, and $M_{\mathrm{act,window}} = O(h \cdot s \cdot b)$ corresponds to the activation budget required for one layer.

CPU memory usage is minimized by flattening FP32 masters (using $i-1$0 bytes pinned) and sharing BF16/FP16 buffers for gradients and type conversion, which drops the per-layer gradient footprint to $i-1$1 (reused across layers).

Relative to ZeRO-Offload, which consumes $i-1$2 GPU memory, SlideFormer achieves a typical peak GPU usage of $i-1$3, representing more than 50% reduction in practical deployments (Yang et al., 17 Mar 2026).

3. Optimized Triton Kernels and Model-Specific Primitives

The computational efficiency of SlideFormer is substantially boosted by custom Triton kernels for transformer primitives and the output layer.

• Fused Linear + Cross-Entropy (LCE) Kernel: This kernel process avoids explicit instantiation of the $i-1$4 logits tensor (where $i-1$5 is batch size, $i-1$6 is sequence length, $i-1$7 is vocabulary size), instead operating on vocabulary chunks of size $i-1$8. For each chunk $i-1$9:
  • Compute $T_\mathrm{fwd} \ge T_\mathrm{param\_h2d}$0
  • Compute and accumulate partial losses and gradients
  • Aggregate results in global gradient buffers

This approach achieves $T_\mathrm{fwd} \ge T_\mathrm{param\_h2d}$1 reduction in final layer VRAM usage.

• Other Kernels: The system also provides highly optimized versions of FlashAttention (for self-attention computation), Fused RMSNorm, SwiGLU, and Rotary Positional Encoding (RoPE). Register-blocking and tile-level scheduling further minimize memory allocation peaks during kernel execution.

4. Multi-Tier I/O and Offload Strategy

SlideFormer employs a three-tier memory structure:

• GPU VRAM: For current layer parameters, gradients, and sliding activations.
• CPU DRAM: Storage for parameter masters (FP32), shared gradients (BF16/FP16, pinned).
• NVMe SSD (optional): Offload storage for activations and optimizer states, integrated via GPUDirect Storage. Transfers from NVMe to GPU bypass the CPU.

Configurable offload fractions (50% or 100%) are supported for optimizer states, and activation offload is available for extreme-scale model training. The data movement protocol ensures each object transitions only between adjacent tiers, eliminating multi-hop transfer paths.

Param offloading to NVMe for model parameters is not implemented, as it induces severe throughput degradation.

5. Quantitative Benchmarks

SlideFormer achieves state-of-the-art throughput and efficiency in representative fine-tuning workloads. For Llama-3.1-8B on an RTX 4090 (batch size 32):

Framework	Throughput (tok/s)	GPU VRAM (GB)	CPU DRAM (GB)
ZeRO-Offload	24,000	30.2	190
ColossalAI Gemini	18,000	28.6	160
LoHan	12,500	25.8	145
SlideFormer	45,500	14.1	90

Throughput gains range from 1.40× to 6.27× over baselines for models in the 3B–123B parameter range and across batch sizes. SlideFormer enables the fine-tuning of 72B+ models on a single 32GB GPU, with $T_\mathrm{fwd} \ge T_\mathrm{param\_h2d}$250% GPU memory reduction and $T_\mathrm{fwd} \ge T_\mathrm{param\_h2d}$340% lower CPU memory usage compared to leading alternatives. On Qwen2.5 (3B–72B) variants, SlideFormer sustains $T_\mathrm{fwd} \ge T_\mathrm{param\_h2d}$495% of device peak TFLOPS. Increasing batch sizes does not substantially raise GPU memory usage, allowing for up to 8× batch size growth relative to other systems (Yang et al., 17 Mar 2026).

6. Limitations and Prospects

Parameter offloading to NVMe is intentionally precluded due to drastic throughput degradation when model parameters traverse NVMe↔GPU channels. I/O bandwidth becomes a limiting factor under full activation offloading. Future work may address this bottleneck via compression or fine-grained pipeline scheduling.

SlideFormer’s design is currently scoped to single-GPU environments. Prospective directions include extension of the layer-sliding protocol to multi-GPU systems, adaptive adjustment of the window size (number of concurrent layers in GPU cache), and integration with emergent NVMe-based memory pooling methods.

The holistic co-design introduced by SlideFormer demonstrates that full-parameter fine-tuning of models beyond 123B parameters is now computationally feasible on consumer-grade single-GPU systems, with leading performance across a range of metrics (Yang et al., 17 Mar 2026).