SlimPack: Adaptive Packaging in Deep Learning

• SlimPack is a concept for compact, adaptive packaging that decomposes model states or computational work into fine-grained slices to improve efficiency.
• It employs slice-level decomposition and asymmetric partitioning to balance forward and backward computation in variable-length LLM training.
• Empirical evaluations demonstrate up to 2.8× throughput improvement and reduced straggler effects, highlighting its impact on optimizing training in heterogeneous workloads.

SlimPack is a label used in several arXiv works for compact, adaptive packaging of model state, intermediate representations, or computational work. Its most specific use is the framework described in "SlimPack: Fine-Grained Asymmetric Packing for Balanced and Efficient Variable-Length LLM Training," which targets variable-length LLM training by decomposing samples into slices and assembling balanced scheduling units for asymmetric forward and backward execution (Liu et al., 30 Sep 2025). Related usages include a communication-efficient “SlimPack” strategy for synchronous data parallelism (Sun et al., 2017), a slimmable packing design for split deep neural networks in bandwidth-constrained IoT systems (Assine et al., 2023), and a deployment artifact for width-adaptive ConvNeXt inference (Haberer et al., 21 May 2026). This suggests that the term functions less as a single standardized algorithm than as a recurring design motif: replacing coarse, fixed transfers or workloads with smaller adaptive units.

1. Nomenclature and scope

In the cited literature, SlimPack does not denote a single canonical method. Instead, it appears in multiple subfields with a shared emphasis on compactness, selective transmission, or elastic configuration.

Work	Domain	Meaning of “SlimPack”
"Slim-DP: A Light Communication Data Parallelism for DNN" (Sun et al., 2017)	Distributed DNN training	Pack and transmit only a subset of parameter indices and values
"Slimmable Encoders for Flexible Split DNNs in Bandwidth and Resource Constrained IoT Systems" (Assine et al., 2023)	Split computing	Slimmable ensemble encoder with runtime-adaptable width and bitrate
"SlimPack: Fine-Grained Asymmetric Packing for Balanced and Efficient Variable-Length LLM Training" (Liu et al., 30 Sep 2025)	Variable-length LLM training	Slice-level decomposition and Asymmetric Partitioning into MicroPacks
"Slimmable ConvNeXt: Width-Adaptive Inference for Efficient Multi-Device Deployment" (Haberer et al., 21 May 2026)	Width-adaptive vision inference	Package one set of full-width weights with predefined width configurations

Among these, the 2025 LLM framework is the only work whose title is precisely "SlimPack" (Liu et al., 30 Sep 2025). The earlier and later uses are structurally related but domain-specific. A plausible implication is that the term has evolved into a shorthand for slim packaging of either communication payloads, encoded features, scheduling units, or nested subnetworks.

2. Variable-length LLM training and the origin of SlimPack

The 2025 SlimPack framework addresses the training inefficiencies created by extreme variance in context lengths during LLM pretraining. Real datasets such as CommonCrawl, GitHub, and Wikipedia are described as having pronounced long-tailed sequence-length distributions: empirically, ≈80% of samples may be shorter than 4K tokens, while ≈1% of ultra-long samples (>128K) can contribute nearly half of the total compute because attention FLOPs scale quadratically in length (Liu et al., 30 Sep 2025).

This heterogeneity produces stragglers. In hybrid parallel settings combining data parallelism and pipeline parallelism, the per-iteration time is dictated by the slowest micro-batch, and a delay in one data-parallel rank propagates across pipeline stages as “cascading imbalance bubbles.” SlimPack treats this as a systems problem rather than solely a packing problem: conventional packing reduces padding, but “equal token count” packing does not equalize compute under quadratic attention cost, and fixed microbatching with best-fit packing can appear forward-balanced while remaining backward-imbalanced (Liu et al., 30 Sep 2025).

A central motivation is the asymmetry between forward and backward costs. With memory-efficient attention, attention backward recomputes discarded intermediates and incurs ≈2.5× the forward cost; GEMM backward is ≈2× forward because gradients must be computed with respect to activations and weights (Liu et al., 30 Sep 2025). This invalidates symmetric scheduling assumptions. A micro-batch that is acceptable in the forward pass can become a backward straggler, especially when it contains ultra-long sequences. One common misconception is that packing quality can be judged by token counts alone; SlimPack explicitly rejects that premise.

3. Slice-level decomposition and Asymmetric Partitioning

SlimPack’s core mechanism is slice-level decomposition. Samples are not treated as indivisible units. Each sample is decomposed into fine-grained slices, where a slice is a contiguous span of tokens from the sample, and slices preserve original order (Liu et al., 30 Sep 2025). The stated purpose is to transform large, volatile workloads into a stream of smaller, manageable units.

Correctness is preserved through causal attention via KV caches. In the forward pass of slice $j$, queries attend to all previous tokens using per-layer KV cache from the previous slices; in the backward pass, gradient dependencies are respected by FILO ordering within a sample’s slices (Liu et al., 30 Sep 2025). This makes it possible to serialize ultra-long samples without breaking causal semantics.

Slices are assembled into minimal schedulable containers called MicroPacks. Under a uniform FLOPs budget, the framework distinguishes three MicroPack types: Slim, consisting of consecutive slices from a single long sample; Mix, consisting of leftover slices of a slimmed sample co-packed with complete short samples or slices from other samples; and Pack, consisting entirely of complete short samples (Liu et al., 30 Sep 2025). A single sample may span multiple MicroPacks, which converts high-variance workloads into multiple low-variance units.

The second core mechanism is Asymmetric Partitioning. SlimPack produces distinct MicroPack configurations for forward and backward, denoted $\mathcal{P}_f$ and $\mathcal{P}_b$ (Liu et al., 30 Sep 2025). Forward slices are grouped to equalize forward FLOPs across MicroPacks and pipeline stages. Backward slices are re-grouped to match backward FLOPs targets, counteracting the ≈2.5× attention and ≈2× GEMM multipliers. The framework therefore permits $\mathcal{P}_f \neq \mathcal{P}_b$ while strictly preserving per-sample slice dependencies. If backward MicroPacks fall short because of dependency ordering, SlimPack injects an extra forward MicroPack earlier to ensure backward balance and continuous pipeline flow.

The global scheduling objective is makespan minimization under memory and communication constraints:

$$\min_{\mathcal{P}_f,\mathcal{P}_b,m,\{\ell_{x,j}\}} \max_{i \in \text{devices}} \Big( T_i^{(f)}(\mathcal{P}_f) + T_i^{(b)}(\mathcal{P}_b) + T_i^{(\mathrm{comm})} \Big).$$

The formulation is constrained by slice contiguity and order, forward FIFO MicroPack ordering across pipeline stages, backward FILO per sample’s slices, per-device memory limits, communication scheduling, and forward and backward FLOPs targets for each MicroPack (Liu et al., 30 Sep 2025).

4. Solver, simulator, and systems integration

SlimPack uses a two-phase solver. In Phase 1, the global batch is assigned to data-parallel ranks to equalize forward FLOPs per rank, with samples sorted by forward FLOPs and greedily distributed (Liu et al., 30 Sep 2025). In Phase 2, each data-parallel rank chooses the number of MicroPacks $m$ and constructs them with fixed per-pack budgets,

$$\tau_{\mathrm{MicroPack}} = \frac{\sum_{x\in \mathrm{dp}_i} f^{\mathrm{fwd}}(x)}{m}, \quad m = i\cdot \mathrm{PP}, \; i\in \mathbb{Z}_{>0}.$$

Long samples are slim-sliced across multiple MicroPacks; short samples are packed until the budget is met; a MILP refines slice boundaries and enforces memory and ordering constraints (Liu et al., 30 Sep 2025).

For rare ultra-long outliers, SlimPack introduces DP-Merge. A subset of data-parallel ranks is temporarily merged into a “super-DP,” and context parallelism with size $g$ is applied only for that outlier’s slices. The effective per-rank compute is

$f_{\mathrm{eff}}(x^\star) = \frac{f(x^\star)}{g}.$

Model and optimizer states remain local; only the outlier’s attention uses context-parallel collectives such as Ring or Ulysses for its MicroPacks (Liu et al., 30 Sep 2025). This is a selective use of context parallelism rather than a global topology change.

Schedule selection is mediated by a high-fidelity DAG-based simulator. Its inputs are a candidate schedule, calibrated runtime models for GEMM, attention, and norms, communication models, and memory profiles. Its outputs are critical-path makespan $T_{\text{total}}$, per-stage utilization and bubble analysis, and peak memory per device (Liu et al., 30 Sep 2025). Compute, memory, and communication are all explicitly modeled. For example, data-parallel all-reduce time for gradients of size $\mathcal{P}_f$0 bytes is represented as

$\mathcal{P}_f$1

pipeline stage-to-stage point-to-point transfer as

$\mathcal{P}_f$2

and context-parallel communication for DP-Merge as

$\mathcal{P}_f$3

The stated effect of slicing is to increase message count while keeping total bytes constant; under steady-state 1F1B, the latency is largely overlapped (Liu et al., 30 Sep 2025).

Implementation is described as being atop Megatron-LM with a PyTorch backend, with the solver running on CPUs in C++ with a thread pool and overlapped with data loading and prefetching so that GPUs never stall (Liu et al., 30 Sep 2025). Data parallelism keeps gradient all-reduce unchanged; tensor parallelism and sequence parallelism are compatible; pipeline parallelism uses MicroPacks as the smallest scheduling units; optimizer states remain local and stationary; activation checkpointing is supported and can be adaptively used per stage to meet memory constraints.

5. Empirical behavior, operating regimes, and limitations

The reported evaluation uses nodes with 2× Intel Xeon Platinum CPUs, ≈1 TB RAM, and 8× NVIDIA Hopper 80GB GPUs per node, with NVLink 400 GB/s per GPU and a 400 Gbps NIC per GPU (Liu et al., 30 Sep 2025). Models are LLaMA-style dense 7B, 13B, 70B (GQA), and 150B (GQA), vocabulary size 32K, trained on CommonCrawl, GitHub, and Wikipedia. The baseline is Megatron-LM with best-fit sample packing; both baseline and SlimPack use 1F1B and similar activation recompute and offload features. Reported metrics include tokens/sec per GPU, utilization and bubble analyses, and simulator-versus-real memory peak comparisons (Liu et al., 30 Sep 2025).

The headline result is up to a $\mathcal{P}_f$4 throughput improvement over the Megatron-LM baseline, with gains generally increasing with sequence length. For LLaMA-150B, the reported gains are 1.15× at 64K, 1.54× at 128K, and 1.68× at 256K context (Liu et al., 30 Sep 2025). Violin plots are said to show forward and backward computation times per device rank tightly clustered under SlimPack versus broad tails under sample-level packing, indicating suppression of stragglers and bubbles. The DAG-based memory simulation matches measured peak GPU memory with MAPE ≈ 1.6% (Liu et al., 30 Sep 2025).

The reported benefits are largest for long-tailed datasets and long context lengths, especially in hybrid DP+PP systems. Deeper pipelines benefit more from MicroPack granularity. Practical guidance recommends choosing MicroPack counts $\mathcal{P}_f$5 as multiples of PP and sweeping a small log-spaced set such as $\mathcal{P}_f$6 to balance warmup bubble reduction against message overhead (Liu et al., 30 Sep 2025). For Hopper-class GPUs, finer slicing is described as acceptable; on higher-latency interconnects, excessively small slices are discouraged because they can enter latency-dominated regimes.

The limitations are equally explicit. Very small sequences, such as ≤1–2K tokens, see limited benefit, and slicing or scheduling overheads may slightly offset gains (Liu et al., 30 Sep 2025). Extreme communication constraints can make the additional point-to-point messages more visible. If the environment mandates context parallelism for most samples, gains diminish because SlimPack’s communication advantage depends on using context parallelism only for rare outliers. The paper also notes that applicability to strongly heterogeneous model families such as MoE with dynamic routing would require extending cost models and dependency tracking (Liu et al., 30 Sep 2025).

6. Related uses of the SlimPack label in other subfields

The 2017 work "Slim-DP: A Light Communication Data Parallelism for DNN" can be understood as a “SlimPack” strategy for synchronous parameter-server training (Sun et al., 2017). Instead of pushing and pulling the full model state every synchronization, each worker communicates only a set $\mathcal{P}_f$7, where $\mathcal{P}_f$8 is a server-selected core of significant parameters and $\mathcal{P}_f$9 is a worker-specific random explorer subset. Parameter significance is measured as

$\mathcal{P}_b$0

and the server selects the top-$\mathcal{P}_b$1 fraction as the core. The resulting per-synchronization communication volume is $\mathcal{P}_b$2 rather than $\mathcal{P}_b$3 per direction in dense Plump-DP. On ImageNet with GoogLeNet and VGG-16, the method is reported to save ≈55% communication time for GoogLeNet and ≈70% for VGG-16 relative to Plump-DP, while preserving or slightly improving top-5 accuracy (Sun et al., 2017). In this usage, SlimPack refers to sparse, index-aware packaging of updates and parameters.

In split computing, "Slimmable Encoders for Flexible Split DNNs in Bandwidth and Resource Constrained IoT Systems" describes a “SlimPack”-style slimmable packing design for transmitting intermediate features from a device to an edge server (Assine et al., 2023). The setting is object detection on COCO2017 using a teacher detector EfficientDet-D2 split after the second bottleneck. The on-device encoder is a slimmable ensemble of $\mathcal{P}_b$4 identical, very small CNNs; runtime configuration is given by ensemble size $\mathcal{P}_b$5 and bit-depth $\mathcal{P}_b$6, for 16 configurations. Aggregation is progressive,

$\mathcal{P}_b$7

and the transmitted tensor size is controlled mainly by $\mathcal{P}_b$8, with $\mathcal{P}_b$9 bits. On Raspberry Pi 4 with a Bluetooth 4.1 link, reported RTT ranges from ~211 ms for $\mathcal{P}_f \neq \mathcal{P}_b$0 to ~554 ms for $\mathcal{P}_f \neq \mathcal{P}_b$1; example operating points include 6.9 kB and mAP 14.5 for $\mathcal{P}_f \neq \mathcal{P}_b$2, 13.8 kB and mAP 29.4 for $\mathcal{P}_f \neq \mathcal{P}_b$3, 20.7 kB and mAP 34.2 for $\mathcal{P}_f \neq \mathcal{P}_b$4, and 27.6 kB and mAP 36.8 for $\mathcal{P}_f \neq \mathcal{P}_b$5 (Assine et al., 2023). Here, SlimPack denotes runtime-adaptable feature packaging under bandwidth and device-compute constraints.

In width-adaptive CNN deployment, "Slimmable ConvNeXt: Width-Adaptive Inference for Efficient Multi-Device Deployment" uses SlimPack as a deployment artifact rather than a training scheduler or communication codec (Haberer et al., 21 May 2026). Width-adaptive inference packages multiple nested subnetworks inside a single set of shared weights. The ConvNeXt design is described as being especially suitable because LayerNorm eliminates switchable batch normalization and the pointwise layers are trivial to slice by channels. Slimming a block to ratio $\mathcal{P}_f \neq \mathcal{P}_b$6 retains the first $\mathcal{P}_f \neq \mathcal{P}_b$7 channels, and a practical SlimPack stores one set of full-width weights plus predefined $\mathcal{P}_f \neq \mathcal{P}_b$8-lists and optional device-specific latency and energy profiles. Reported ImageNet-1k results include Slimmable ConvNeXt-T with 3 subnetworks achieving 80.8% top-1 accuracy at 4.5 GMACs and 77.4% at 1.2 GMACs, Slimmable ConvNeXt-S achieving 82.3% at 8.7 GMACs and 80.2% at 2.3 GMACs, and Slimmable ConvNeXt-B achieving 82.8% at 15.35 GMACs with 82.8% also at 8.8 GMACs for $\mathcal{P}_f \neq \mathcal{P}_b$9 (Haberer et al., 21 May 2026). In this context, SlimPack refers to packaging elastic width configurations into one deployable model artifact.

Across these usages, a plausible common denominator is selective granularity. The object being slimmed differs—parameter updates, intermediate features, sequence slices, or channel widths—but each method replaces uniform treatment of all elements with a compact representation or schedule that targets the elements most relevant to the current systems constraint.