Hot Block Record-and-Prefetch Optimization

• Hot block record-and-prefetch is a trace-driven optimization that prefetches critical image blocks to reduce startup latency in containerized LLM training environments.
• It records the blocks accessed during the initial startup phase and uses that trace to prefetch these blocks in subsequent runs, mitigating I/O and network bottlenecks.
• This technique has been shown to cut image-loading delays by 4× to 10× in production, significantly reducing wasted GPU time and improving synchronization across nodes.

Hot block record-and-prefetch is a trace-driven optimization for container image startup in large-scale LLM training. Introduced as one of three techniques in BootSeer, it addresses the container image loading bottleneck that arises when many worker nodes simultaneously start very large container images under block-level lazy loading. The method records the image blocks accessed during the startup phase on a first run and uses that trace to prefetch the same blocks on later runs, while remaining cold blocks continue to download in the background. Within the production setting studied by BootSeer, more than 3.5% of GPU time is wasted due to startup overhead alone, which situates image loading as a systems concern rather than a minor initialization detail (Li et al., 16 Jul 2025).

1. Startup bottleneck and workload conditions

BootSeer targets the container image loading bottleneck during LLM training startup. In the production system described in the paper, LLM training jobs use very large container images, typically 25–40 GB, and the evaluation workload uses a 28.62 GB image. When many worker nodes start simultaneously, they all try to pull image content at once, which creates heavy network pressure, heavy I/O pressure, and startup stragglers when some nodes fetch blocks more slowly than others.

The baseline already uses lazy, block-level loading, so containers do not necessarily download the whole image up front. However, lazy loading alone remains insufficient for startup because only a small subset of image data is accessed early, and those accesses are sparse but highly repeatable. If those early-needed blocks are not locally available, the runtime must fetch them from a remote cache or another machine, delaying startup and potentially stalling synchronized workers. The paper explicitly characterizes startup image loading as I/O- and bandwidth-bound, with a sparse access pattern during startup, and identifies this phase as a good candidate for prefetching (Li et al., 16 Jul 2025).

2. Operational meaning of “hot block”

A hot block is a data block in the container image that is accessed during the startup phase, especially in the early portion of container startup. The term does not denote a mathematical score or an explicit optimization variable. Instead, hotness is operationally defined by observed access during startup.

The central observation is that startup repeatedly touches a small, compact, stable set of blocks. Those blocks are “hot” because their timely availability most affects startup latency. This definition is deliberately trace-based rather than model-based. A plausible implication is that the technique depends less on offline profiling or workload semantics than on the empirical regularity of startup accesses across repeated runs (Li et al., 16 Jul 2025).

3. Record phase and trace construction

During the first access to a container image, BootSeer monitors the container runtime’s block accesses during startup and records them. The recorded information includes absolute file paths and block offsets. In effect, the hot-block record is an access trace of which image blocks were touched during startup, together with enough location metadata to retrieve them later.

Recording occurs during the record phase, specifically on the first run or first access to the image. In the evaluation, a 2-minute window is used to record hot blocks, indicating that BootSeer treats the early startup interval as the observation window for determining which blocks are startup-critical. Recording happens in the container runtime on the worker node during startup, and the trace is then uploaded to a remote service. Later, when the same image is used again, another worker node’s runtime can retrieve this record from the controller (Li et al., 16 Jul 2025).

4. Prefetch phase and systems integration

On later runs using the same image, BootSeer retrieves the previously recorded hot-block trace from the controller, identifies the corresponding blocks in the image, and prefetches all identified hot blocks before or during container startup. The purpose is to let the container start without waiting on remote fetches for the blocks most likely to be needed immediately. After container execution begins, the runtime continues downloading the remaining cold blocks in the background.

This foreground/background split preserves the I/O savings of lazy loading while removing delay from startup-critical misses. The mechanism is integrated with the container runtime, the platform’s lazy-loading block image format, a remote service or controller, and cluster-level cache and peer-to-peer sharing for block retrieval. Blocks can be fetched from a cluster-level cache or from peer nodes, so the prefetch path is not restricted to a single origin server.

The image-loading system itself is built around block-level on-demand loading, lazy loading, block deduplication, and a flattened image layout rather than traditional layered OCI images. The platform does not use traditional OCI layered images; it flattens all layers into a single unified layer and manages contents at the block level. This is significant because record-and-prefetch operates on block offsets rather than on higher-level layer metadata (Li et al., 16 Jul 2025).

5. Control flow, heuristics, and implementation characteristics

The paper does not provide formal pseudocode, explicit algorithms, or a named data structure for hot-block selection. It also does not give a LaTeX formula for hotness or a cost model for deciding which blocks are hot. The control flow is nonetheless specified clearly. On a first run, the container starts with lazy loading, the runtime observes which image blocks are accessed during startup, records file path and block offset, and uploads the trace to a remote controller or service. On subsequent runs, the runtime asks the controller for the prior access record for the same image, retrieves the list of hot blocks, prefetches all hot blocks during image loading, starts the container earlier because hot data is already local or available, and fetches the remaining blocks in the background.

The only concrete knobs reported for this technique are a 2-minute window to record hot blocks in evaluation and 8 threads used for background prefetching of cold blocks in evaluation. No explicit hotness score metric is defined; “hot” means “accessed during startup in the prior run.” This makes the method a trace-based heuristic rather than a formally optimized selection policy (Li et al., 16 Jul 2025).

6. Reported effects on startup performance

The evaluation attributes the image-loading improvements primarily to record-and-prefetch. BootSeer overall reduces startup overhead by 2× across job scales, and the image-loading stage is where record-and-prefetch matters most. Specifically, BootSeer outperforms the baseline by 4× to 10× in Image Loading as job scale increases. The baseline lazy-loading setup suffers from cache misses and network pressure, whereas BootSeer’s record-and-prefetch keeps image-loading duration stable regardless of job scale.

The paper explains these gains through the structure of startup access: early startup accesses only a small subset of image blocks; BootSeer records these blocks once; later runs prefetch them; and this avoids remote data access during the critical startup path. The image-loading optimization also reduces the risk of slow nodes caused by remote fetches and bandwidth contention. Since many nodes start together, avoiding remote misses helps reduce synchronization delays. The evaluation further notes that hot blocks are captured with a 2-minute record window and that cold blocks are prefetched in the background with 8 threads. This suggests that the mechanism is not only a cache-hit optimization but also a way to overlap warmup I/O with execution (Li et al., 16 Jul 2025).

7. Limitations, trade-offs, and scope conditions

The effectiveness of hot block record-and-prefetch depends on repeatability. The method works well because startup accesses are described as sparse, compact, and stable across runs. If image startup paths change often, the recorded hot blocks may be less predictive. The method also requires a previous run or prior trace, so it is most effective for repeated restarts of the same job, debugging loops, similar short jobs, and long jobs that restart often. It is less helpful on the very first launch of a new image or workload.

The technique does not remove all image-loading overhead. BootSeer still uses lazy loading and background fetching for cold blocks, so the optimization primarily reduces startup-critical I/O rather than total image transfer. Its end-to-end effect may also be limited when image loading is not the dominant bottleneck. In the production trace discussed in the paper, the Environment Setup stage is the largest bottleneck overall, which means that dependency installation or checkpoint resumption can still constrain total startup even when image loading is improved.

Network and cache contention also remain relevant. Hot blocks still need to be distributed, cold blocks are still fetched in the background, and cluster cache and peer-to-peer paths remain part of the performance story. Finally, the paper does not describe an adaptive threshold, confidence score, or learned model to decide hotness. It uses a simple trace-based heuristic, which is practical and directly aligned with the observed startup access pattern (Li et al., 16 Jul 2025).