BootSeer: Optimizing LLM Startup

• BootSeer is a system-level framework for LLM training that redefines startup overhead by isolating stages from submission to effective training.
• It employs targeted solutions such as hot block record-and-prefetch, dependency snapshotting, and striped HDFS-FUSE to achieve up to a 50% reduction in startup time.
• Experimental evaluations on large clusters demonstrate BootSeer’s ability to halve startup delays and minimize GPU waste caused by straggling nodes.

Searching arXiv for the specified paper and a few related systems mentioned in the source material. BootSeer is a system-level optimization framework for large-scale LLM training that targets startup overhead rather than steady-state runtime performance. It analyzes the interval from job submission to the beginning of effective training, with particular emphasis on the GPU-consuming portion of startup after resources have already been allocated but before the first training step begins. The system is motivated by production evidence that startup activities can consume more than $3.5\%$ of GPU server-hours in an LLM training cluster, and it addresses three primary bottlenecks: container image loading, runtime dependency installation, and model checkpoint resumption. BootSeer combines detailed startup profiling with three corresponding optimizations—hot block record-and-prefetch, dependency snapshotting, and striped HDFS-FUSE—and was reported to achieve about a $50\%$ reduction in startup overhead on real LLM training workloads (Li et al., 16 Jul 2025).

1. Problem formulation and empirical motivation

BootSeer distinguishes ordinary runtime performance from startup overhead. Prior LLM systems work is described as concentrating on runtime efficiency through parallelism, communication optimization, checkpointing, fault tolerance, kernel optimization, and overlap of compute and communication. BootSeer instead targets the delay between job submission and the first training step, defining startup overhead as the collection of stages from job submission through scheduling and resource allocation, container image loading, runtime environment setup, and model initialization or checkpoint load, up to the beginning of effective training (Li et al., 16 Jul 2025).

The startup decomposition is expressed as

$$T_{\text{startup}} = T_{\text{queue}} + T_{\text{alloc}} + T_{\text{image}} + T_{\text{env}} + T_{\text{model\_init}},$$

where $T_{\text{queue}}$ is waiting in queue, $T_{\text{alloc}}$ is scheduler allocation, $T_{\text{image}}$ is container image loading, $T_{\text{env}}$ is environment setup and dependency installation, and $T_{\text{model\_init}}$ is user-code startup, checkpoint resumption, and RDMA setup. BootSeer focuses on the GPU-consuming subset,

$$T_{\text{startup,GPU}} = T_{\text{image}} + T_{\text{env}} + T_{\text{model\_init}},$$

because these stages occur after GPUs are allocated but remain idle.

The empirical motivation comes from one week of production profiling in an LLM training cluster. During that period, 28,000 jobs were submitted, requesting more than 700,000 GPUs cumulatively, across a hardware mix including NVIDIA H800, A100-80GB, and V100 systems, with nodes containing 8–16 GPUs. Cluster-wide startup activities consumed approximately $3.5\%$ of total GPU server-hours. For jobs using more than 100 GPUs, job-level startup time from submission to training begin was typically 6–7 minutes, with worst cases of at least 15 minutes. Restarts were also common: small jobs with fewer than 100 GPUs usually started once, whereas large jobs often had 2–8 startups, and worst cases exceeded 20 startups for a single logical job due to debugging, configuration changes, and failures (Li et al., 16 Jul 2025).

This characterization suggests that startup latency is not merely a transient inconvenience. In large industrial LLM development, repeated startup costs compound into both measurable GPU waste and increased developer latency, especially when failures, preemption, or iterative reconfiguration cause repeated restarts.

2. Startup anatomy and scaling behavior

BootSeer divides startup into a Scheduler Phase and a Worker Phase. The Scheduler Phase consists of resource queuing and resource allocation and does not consume GPU time. The Worker Phase begins after GPUs are allocated and includes image loading, environment setup, and model initialization, during which GPUs are idle but already reserved (Li et al., 16 Jul 2025).

The profiled stage breakdown identifies resource queuing as having a median of approximately 100 seconds, though it can extend to minutes or hours under contention. Resource allocation takes only a few seconds. Within the GPU-consuming stages, image loading typically requires 20–40 seconds for download and container start, with images of 25–40 GB. Environment setup is the dominant bottleneck, typically lasting 100–300 seconds and exhibiting high variability because of differing dependency graphs and remote package sources. Model initialization usually takes 100–200 seconds and includes launching ranks, setting up distributed training, RDMA connections, and checkpoint loading from HDFS. A cited example is a 25B-parameter MoE model with a 400 GB checkpoint (Li et al., 16 Jul 2025).

A central result is that the dominant contributors among GPU-consuming stages are ordered as environment setup, then model initialization, then image loading. This ranking shapes the design priorities of BootSeer. The system does not merely seek lower mean startup time; it also seeks to reduce variance and tail latency, since synchronized distributed startup makes the slowest nodes determine when training can begin.

Scaling behavior is driven by stragglers. Larger jobs exhibit more variability because more nodes perform concurrent remote accesses, increasing the probability that some nodes will be delayed by network contention or throttling. Node-level startup times are reported to be about one minute shorter than job-level times, with the difference corresponding exactly to waiting for stragglers. The paper defines a Max/Median straggler metric using dependency-install script time as a proxy:

$50\%$0

For jobs using more than 1000 GPUs, the straggler ratio is approximately $50\%$1 on average, with extreme cases above $50\%$2. In one 1,440-node, 11,520-GPU job, most nodes installed dependencies in at most 60 seconds, but fewer than $50\%$3 took about 92 seconds, stalling the entire job (Li et al., 16 Jul 2025).

A plausible implication is that startup optimization at LLM scale must target distribution flattening as much as absolute speedup. In synchronized training systems, even a small fraction of slow nodes can dominate effective startup time.

3. Profiling architecture and system placement

BootSeer consists of both a profiling system and a startup optimizer. As a profiling system, it instruments startup code paths with simple logs such as markers for entering and exiting image loading. Each node runs a Log Parser that watches log files, extracts timestamps for stage events, collects node lifecycle events, and sends structured events to a central Stage Analysis Service. The Stage Analysis Service aggregates events by job and node, computes stage durations, and stores results in a database for dashboarding and offline analysis (Li et al., 16 Jul 2025).

This profiling pipeline underlies the startup characterization and validates the impact of subsequent optimizations. The design is notable for using lightweight instrumentation rather than requiring invasive changes to user training code. BootSeer is described as transparent to user training scripts, with changes confined to infrastructure components such as container runtime behavior, dependency installation and caching, and checkpoint mounting and access paths.

Within the platform stack, BootSeer lives in the infrastructure layer of an internal LLM training platform. It interacts with the cluster scheduler, integrates with the container runtime and image distribution system, interposes on environment setup scripts, and uses HDFS-FUSE to expose HDFS-stored artifacts as local directories. Job submission remains unchanged, and BootSeer is triggered through internal configuration flags, image identifiers, and job metadata used for environment-cache mapping and invalidation (Li et al., 16 Jul 2025).

This placement is significant because it frames BootSeer as an operational systems optimization rather than a training-algorithm modification. No changes are required to PyTorch training scripts, and no image rebuild is required for BootSeer itself.

4. Core mechanisms

BootSeer addresses the three dominant startup bottlenecks with one mechanism per stage: hot block record-and-prefetch for image loading, dependency snapshotting for environment setup, and striped HDFS-FUSE for checkpoint resumption (Li et al., 16 Jul 2025).

For container image loading, the baseline image system already uses flattened images, fixed-size block partitioning, block-level deduplication, and lazy loading, with the cited baseline described as providing up to $50\%$4 faster image distribution than traditional OCI. BootSeer’s observation is that startup touches a small, stable subset of image blocks. It therefore separates blocks into “hot blocks,” accessed during a defined initial startup window, and “cold blocks,” accessed later or not at all. On the first run of a given image, the runtime logs block accesses during startup and sends traces to a controller, which aggregates them into a per-image hot-block list. On subsequent runs, the runtime prefetches all hot blocks before or during startup, using priority fetches from a cluster-level cache or peer-to-peer transfers from other nodes already holding the blocks. Cold blocks are then fetched by background threads after startup. The design explicitly uses peer-to-peer sharing because many nodes in a job request the same image concurrently, and this reduces pressure on a central registry or cache (Li et al., 16 Jul 2025).

For runtime dependency installation, BootSeer targets the environment setup stage, which is described as the largest bottleneck and the primary cause of startup stragglers. The baseline performs full runtime installation on every node, including pip install, apt-get, and internal package fetches. The paper states that dependencies are not fully baked into the Docker image because some packages require runtime specialization by GPU type, OS version, machine type, or region, and because some dependencies change frequently enough that repeated image rebuilds are slow and costly. Under baseline behavior, thousands of nodes may concurrently access package repositories or internal artifact stores, creating “bit storms” and provoking rate limiting or throttling (Li et al., 16 Jul 2025).

BootSeer treats dependency installation as a job-level environment snapshot. It defines a Target Directory, typically the site-packages or dependency directory inside the container. On the first run of a job, BootSeer compares the Target Directory before and after environment setup on a designated node such as worker 0, identifies added or modified files as the environment delta, compresses those files, and uploads the result to HDFS through HDFS-FUSE. The experiment cited an environment delta of approximately 270 MB compressed. On later startups of the same job, including restarts, node replacement, or partial failure, BootSeer checks for an existing cache for the job configuration, mounts the HDFS location, restores the cached files into the Target Directory, and skips dependency installation commands. Cache validity is tied to job parameters such as dependency lists, versions, and environment parameters; if relevant runtime parameters change, the existing cache is marked expired and rebuilt on the next startup (Li et al., 16 Jul 2025).

For checkpoint resumption, BootSeer modifies checkpoint I/O through striped HDFS-FUSE. The paper’s example workload uses an 8-layer MoE with 128 experts and about 25B parameters, producing a 413 GB checkpoint. Standard HDFS block placement is described as limiting parallelism for a single massive checkpoint file because large blocks are replicated on a limited set of DataNodes, which constrains read bandwidth and concentrates contention. BootSeer therefore introduces striping via FUSE: large logical files are split into 1 MB chunks, grouped into physical logical segments, and laid out in HDFS with a stripe size of 4 MB in the design sketch. Chunks from multiple logical files can be interleaved within a single HDFS block. On reads, the striped HDFS-FUSE driver maps file offsets to chunk files and HDFS blocks, issues parallel reads across multiple DataNodes, and overlaps local filesystem I/O with HDFS network I/O. The same chunking and striping are also used on writes for checkpoint save, although that does not directly affect startup (Li et al., 16 Jul 2025).

Taken together, these mechanisms reveal a coherent design principle: reduce synchronized remote demand at startup by identifying the hot path, reusing previously materialized environments, and increasing storage parallelism for large checkpoint loads.

5. Deployment setting and experimental results

BootSeer was evaluated both through production traces and controlled experiments. The controlled evaluation used servers with $50\%$5 Intel Xeon Platinum 8457C processors, each with 96 cores, 2.9 TB of RAM, 8 NVIDIA H800 GPUs per node, InfiniBand interconnect, and an HDFS storage backend. Experiments scaled up to 16 nodes, or 128 GPUs. The workload was an 8-layer MoE model with 128 experts per layer, pipeline parallelism of 2-way, data parallel degrees of $50\%$6, a 413 GB checkpoint, and a 28.62 GB container image (Li et al., 16 Jul 2025).

Methodologically, end-to-end startup was measured from job launch to the point at which training begins, while excluding queue and allocation time in controlled experiments so as to focus on GPU-consuming startup. Baseline and BootSeer were compared across 16, 32, 48, 64, and 128 GPUs. The local image cache was cleared before every run, and each experiment was repeated three times and averaged. Per-stage breakdowns measured image loading, environment setup, and model initialization. Straggler analysis at 128 GPUs collected per-node dependency-install durations and compared distributions between baseline and BootSeer (Li et al., 16 Jul 2025).

The reported end-to-end result is an approximately $50\%$7 startup reduction across all tested GPU scales. Startup time still increased with job size, but the increase was roughly halved relative to baseline, especially from 64 to 128 GPUs, where baseline overhead spiked and BootSeer kept the spike much smaller. Per-stage results attribute this improvement to three distinct effects: image loading became $50\%$8–$50\%$9 faster than baseline lazy loading and remained relatively flat as job size grew; environment setup became roughly $$T_{\text{startup}} = T_{\text{queue}} + T_{\text{alloc}} + T_{\text{image}} + T_{\text{env}} + T_{\text{model\_init}},$$0 faster and much less variable; and model initialization became about $$T_{\text{startup}} = T_{\text{queue}} + T_{\text{alloc}} + T_{\text{image}} + T_{\text{env}} + T_{\text{model\_init}},$$1 faster because of striped HDFS-FUSE (Li et al., 16 Jul 2025).

The straggler analysis is particularly central to BootSeer’s systems argument. In the 128-GPU experiment, baseline dependency installation exhibited a wide spread, long tails, and a substantial gap between median and maximum node times. Under BootSeer, the distribution became tightly clustered with small whiskers, and the paper describes dependency-install stragglers as essentially eliminated at that scale. The paper does not experimentally recompute whole-cluster savings after deployment across the entire production cluster, but it combines the observed $$T_{\text{startup}} = T_{\text{queue}} + T_{\text{alloc}} + T_{\text{image}} + T_{\text{env}} + T_{\text{model\_init}},$$2 pre-optimization GPU-hour waste with the reported $$T_{\text{startup}} = T_{\text{queue}} + T_{\text{alloc}} + T_{\text{image}} + T_{\text{env}} + T_{\text{model\_init}},$$3 startup-overhead reduction to suggest a potential savings of about $$T_{\text{startup}} = T_{\text{queue}} + T_{\text{alloc}} + T_{\text{image}} + T_{\text{env}} + T_{\text{model\_init}},$$4 or more of cluster-wide GPU time, alongside tighter debug and resubmit cycles (Li et al., 16 Jul 2025).

6. Limitations, trade-offs, and broader context

BootSeer is most directly suited to large, multi-node LLM training jobs, especially jobs of 100 or more GPUs, frequent restarts, massive container images, heavy dependency stacks, and checkpoints in the hundreds of gigabytes. The benefits are stated to be strongest when startup optimizations can be amortized over repeated runs and many nodes (Li et al., 16 Jul 2025).

The design also introduces explicit trade-offs. Environment caches consume HDFS capacity, with the experimental environment cache occupying approximately 270 MB compressed per job configuration. Hot-block traces and image metadata add smaller overheads. The infrastructure is more complex because it requires block-level record-and-prefetch in the image service, metadata management for mapping jobs to environment-cache snapshots, and a striped filesystem layer atop HDFS via FUSE. The paper notes that FUSE has overheads, although it also states that prior work shows modern FUSE can be acceptable for such workloads. Correctness depends on proper invalidation: if environment changes are not detected, stale libraries could be used. BootSeer addresses this by tying caches to job parameters and expiring them when relevant parameters change (Li et al., 16 Jul 2025).

The system is described as best-effort rather than correctness-critical. If an environment cache is missing, the fallback is full dependency installation followed by creation of a new cache snapshot. If prefetch fails, the system falls back to baseline lazy image loading. If striped reads encounter issues, errors are handled at the HDFS level and the FUSE layer propagates them. In each case, the stated worst case is performance degradation rather than semantic failure (Li et al., 16 Jul 2025).

Within the broader LLM systems landscape, BootSeer complements work focused on throughput during the training loop. The source material situates it alongside research areas including training algorithms and parallelism such as Megatron-LM and DeepSpeed, runtime communication optimizations and schedulers such as MegaScale and ByteCheckpoint, and reliability and monitoring such as Falcon, XPUTimer, and SuperBench. It also notes conceptual similarity to serverless-oriented ideas such as Catalyzer, MITOSIS, and TrEnv, while emphasizing that BootSeer addresses much larger containers and checkpoints, synchronized startup across thousands of GPUs, and failure patterns characteristic of long-running distributed training (Li et al., 16 Jul 2025).

This suggests a broader systems interpretation: BootSeer reframes startup as a first-class optimization target in industrial LLM development. Instead of assuming that startup cost is amortized by a long uninterrupted training run, it treats restart-heavy development workflows, synchronized boot across many workers, and remote-dependency fan-out as intrinsic features of modern production training.