Forensic Container Checkpointing (FCC)

• Forensic Container Checkpointing (FCC) is a method to capture a running container’s complete runtime state for forensic analysis, debugging, and workload migration.
• FCC leverages CRIU integrated with container runtimes like CRI-O, Docker, or Podman, relying on privileged, host-mediated checkpointing to preserve memory, process, and network states.
• Kubernetes experiments demonstrate FCC’s potential to reduce downtime and enable deep forensic insights, although challenges remain with filesystem dependencies and cross-host state restoration.

Forensic Container Checkpointing (FCC) denotes the capture of a running containerized workload at a specific instant so that its runtime state can later be inspected, restored, or replayed. In Kubernetes, FCC is described as an experimental feature that exposes container checkpoint/restore capabilities through the Kubernetes API and kubelet; under the hood it typically relies on CRIU integrated into the container runtime, here CRI-O, and captures memory contents, process state, open file descriptors, network state to the extent supported by CRIU/runtime, and kernel resources such as IPC and namespaces (Dinh-Tuan et al., 6 Sep 2025). Read through a broader forensic lens, CRIU-based checkpointing preserves a process or container’s working memory on disk and can deal with both open files and established network connections, making it relevant not only to debugging and failure reproduction but also to offline forensic analysis and, in supported cases, migration of long-running workloads (Andrijauskas et al., 2024).

1. Conceptual basis and design intent

FCC sits at the intersection of checkpoint/restart, container runtime integration, and forensic state preservation. CRIU is a userspace tool for GNU/Linux whose core capability is to stop an application on Linux, save its “working memory on disk,” and restore it later, possibly on another compute node. In the HPC and HTC setting examined for OSG’s OSPool HTCondor setup, the design goal is resilience for long-running jobs subject to preemption, maintenance, or failure. In the Kubernetes setting, the same mechanism is reinterpreted as a forensic primitive: capture a container at a specific instant for offline forensic analysis, debugging, or reproducing a bug, and then reuse that primitive for stateful service migration (Andrijauskas et al., 2024, Dinh-Tuan et al., 6 Sep 2025).

The operational model is explicitly layered. In standalone CRIU usage, checkpoint and restore are invoked through a CLI such as criu dump --shell-job -t 9191 and criu load --shell-job. In Kubernetes, FCC is mediated by the control plane: the API Server receives checkpoint and restore requests, kubelet invokes the local runtime, and CRI-O plus CRIU perform the low-level capture and reconstruction. This architecture matters for forensics because it locates the checkpoint trigger either at a privileged host boundary or at the orchestration layer rather than inside an arbitrary workload (Dinh-Tuan et al., 6 Sep 2025).

Privilege is a first-order property of FCC. CRIU “requires a ‘form’ of root access: sudo, SUID bit, or kernel capabilities,” and a special non-root branch by Adrian Reber was evaluated for unprivileged users. The HPC study shows that this non-root path remains limited, especially in containerized settings. This makes privileged acquisition the normal operating assumption for CRIU-based FCC and explains why Docker, Podman, CRI-O, kubelet, and similar runtime-facing components dominate the practical implementations (Andrijauskas et al., 2024).

2. Captured state and the evidence model

The evidentiary value of FCC derives from the breadth of state that is preserved. Through CRIU, FCC captures memory contents, process state including threads and CPU registers, open file descriptors, network state to the extent supported by the runtime, and kernel resources such as IPC and namespaces. The Kubernetes migration work further states that the mechanism checkpoints all containers in a Pod, freezing the container process tree, dumping memory pages and CPU registers, serializing open file descriptors, and capturing relevant kernel objects and namespace state. In that environment, persistent volumes are not included in the checkpoint and must instead be mounted normally at the target Pod (Dinh-Tuan et al., 6 Sep 2025).

Forensic utility is strongest at the level of user-space runtime state. The CRIU evaluation shows successful dump and restore of simple CPU-only applications, a multi-threaded Pthreads C program, and a forking program, which implies preservation of full user-space memory, process-tree structure, and execution context sufficient for seamless continuation. In an FCC context this enables inspection of decrypted configuration, credentials, in-memory injected code, interpreter data structures, PID relationships, thread counts, and thread states at the moment of capture. The same study also shows that checkpointed binaries compiled with CPU-specific flags can be restored only on machines with the same CPU family/type, indicating that replayability depends on hardware compatibility even when the captured state itself is rich (Andrijauskas et al., 2024).

File-system evidence is preserved indirectly rather than as a complete disk image. CRIU can checkpoint applications with open files, but restoration requires the same directory tree to exist on the restore host: “Restoring a previously checkpointed process requires the same directory paths used during restoration as during checkpointing.” This means the checkpoint contains file descriptors and path references for open files, which are themselves valuable evidence because they identify which configuration files, logs, temporary files, or data paths were live at checkpoint time. It also means that FCC by itself is not a substitute for preserving the underlying filesystem layout (Andrijauskas et al., 2024).

Timers and network-facing blocking state are also represented. On Raspberry Pi 2 Model B devices, a NodeJS function checkpointed halfway through a 60-second sleep and restored 30 seconds later continued with the sleep already completed. In a separate TCP/IP experiment, a container blocked on network I/O while waiting for an HTTP response was checkpointed and suspended; after restore, the existing TCP connection remained usable, the pending response was received, and execution continued. These experiments show that timer state and active TCP/IP sockets can be preserved in supported environments, which is directly relevant to reconstructing incident-time protocol state and partially completed control flows (Karhula et al., 2021).

The evidence model remains incomplete by design. GPU checkpointing is not supported, all GPU-related tests fail, and GPU-resident state is therefore omitted. Kernel-only state is also out of scope: the CRIU study is explicit that the mechanism is userspace and does not address kernel forensic artifacts such as kernel memory or kernel module state. For multi-node MPI applications, checkpointing fails and restore hangs, so FCC based on the evaluated CRIU stack does not yield a coherent distributed snapshot for cluster-scale MPI computations (Andrijauskas et al., 2024).

3. Runtime integration and deployment semantics

FCC is inseparable from container runtime semantics. In the HPC/container evaluation, Docker and Podman support checkpointing only through their own CRIU integration; attempts to attach CRIU directly to a containerized job from the outside failed, and the operational interface is the runtime itself, exemplified by docker checkpoint create looper checkpoint1. Podman is described as having a similar interface. By contrast, Apptainer/Singularity lacks this CRIU integration, and attempts to checkpoint jobs from both outside and inside Apptainer failed for both root and non-root CRIU builds. In nested or container-on-container scenarios, such as HTCondor pilots running in Kubernetes, invoking CRIU inside that environment also fails (Andrijauskas et al., 2024).

These results define the practical acquisition boundary for FCC. The dominant pattern is host- or runtime-mediated checkpointing, not self-checkpointing from inside an unprivileged container. A plausible implication is that FCC should be anchored at the host/container runtime layer whenever Docker, Podman, CRI-O, or kubelet semantics are available, and that unprivileged in-container checkpointing remains unreliable for broad deployment.

At the IoT edge, the integration stack is narrower but more explicit. Functions are executed inside Docker containers on Raspberry Pi 2 Model B devices, Docker community edition is used as the container runtime, CRIU version 3.11 performs checkpoint and restore, and Docker’s experimental CRIU integration provides the operational interface. The implementation fixes on docker-ce 17.03.2 because later versions at the time broke the needed functionality for dumping checkpoint data to arbitrary filesystem locations. The Raspbian kernel had to be recompiled to satisfy CRIU kernel prerequisites, and CONFIG_VDSO had to be disabled for CRIU to work (Karhula et al., 2021).

The Kubernetes realization extends this integration into the orchestration plane. The evaluated stack uses Kubernetes v1.30, CRI-O v1.28 with CRIU enabled, Buildah 1.37.0, Artifact Registry for checkpoint images, RabbitMQ v3.13 for message queues, Ubuntu 20.04 with cgroup v1, and a custom Migration Manager written in Python 3.10. The kubelet on each node implements FCC against the local runtime, Buildah converts checkpoint files into OCI-compliant checkpoint images, and the Migration Manager orchestrates checkpoint, image push, restore, and cleanup through the API Server (Dinh-Tuan et al., 6 Sep 2025).

4. Operational workflows and quantitative behavior

Operationally, FCC appears in three distinct but related workflows: standalone CRIU migration in distributed batch systems, Docker+CRIU suspension and migration of IoT edge functions, and Kubernetes-native migration of stateful microservices.

In the OSPool and HTCondor context, the tested workflow runs a process such as serial LAMMPS, uses criu dump ... to checkpoint it, transfers the state, invokes criu load ... on another host where applicable, and inspects outputs and logs. Some use cases migrate successfully between hosts when they are simple CPU-only workloads with the same filesystem structure and identical CPU type; GPU, MPI, and some networked cases do not (Andrijauskas et al., 2024).

On constrained edge hardware, the Docker+CRIU workflow is explicitly designed to suspend long-running or blocking functions without keeping them resident in RAM. A function executes inside a container and blocks, for example on sleep, an HTTP reply, or an MQTT message. When the device wants to reclaim memory, the container is checkpointed, stopped or removed, and only the checkpoint files remain on disk. Later, the checkpoint is restored into a container, possibly on another device, and execution resumes from the same point. The migration experiment with a NodeJS counter function shows that after checkpointing on one Raspberry Pi and restoring on another, the counter continues from the previous value (Karhula et al., 2021).

The quantitative characteristics of the edge implementation are concrete. For the NodeJS TCP/IP scenario, the base container size is 257 MB, while the checkpoint files total 15 MB. Measured with the Bash built-in time utility on Raspberry Pi 2 Model B, the timings are as follows (Karhula et al., 2021):

Task	Time (s)
Starting a container	1.463
Pause	0.857
Unpause	0.850
Create a checkpoint and save it to disk	1.716
Create a new container	0.438
Start container from the checkpoint	1.763

These timings show that checkpoint creation is only slightly slower than container start, restore is comparable to a fresh start, and pause/unpause are faster but do not release memory or generate disk artifacts. The same study reports that running containers increase memory usage linearly with the number of containers, paused containers consume essentially the same memory as running ones, and checkpointed containers do not cause notable increase in memory because they are not resident (Karhula et al., 2021).

The Kubernetes microservice workflow adds an orchestration layer and a message-replay layer. For an individual Pod, the Migration Manager receives a migration request, retrieves Pod details from the API Server, asks the kubelet on the source node to checkpoint all containers in the Pod, uses Buildah to package the checkpoint files into an OCI image, pushes the image to Artifact Registry, and then creates a new Pod on the target node restored from that checkpoint image. MS2M then replays buffered RabbitMQ messages so that the target instance reconstructs the logical state transitions that occurred after checkpoint time $t_0$ (Dinh-Tuan et al., 6 Sep 2025).

To bound replay overhead under high incoming message rates, the system introduces a Threshold-Based Cutoff Mechanism. With an $M/M/1$ queueing model, the number of accumulated messages and the replay time are written as

$$N_{\text{messages}} = \lambda \times T_{\text{accum}}$$

$$T_{\text{replay}} = \frac{N_{\text{messages}}}{\mu_{\text{target}}} = \frac{\lambda \times T_{\text{accum}}}{\mu_{\text{target}}}$$

subject to

$T_{\text{replay}} \leq T_{\text{replay\_max}}$

which yields the cutoff threshold

$$T_{\text{cutoff}} = T_{\text{accum}} \leq \frac{T_{\text{replay\_max}} \times \mu_{\text{target}}}{\lambda}.$$

Empirically, the baseline Stop-and-Copy migration has an average migration time and downtime of approximately 49.055 s. MS2M for individual Pods reduces average downtime to approximately 1.547 s, and at a low message rate of 4/s the reported downtime reduction is 96.986% compared to a 47.077 s baseline. MS2M with cutoff still reduces downtime by approximately 96.737% at low message rate while making total migration time grow less sharply with message rate. For StatefulSet Pods, the downtime reduction is approximately 24.840% at low message rates, but the benefit diminishes to approximately 0.242% reduction at very high load because StatefulSet semantics do not allow source and target Pods with the same ordinal to run simultaneously (Dinh-Tuan et al., 6 Sep 2025).

5. Constraints, integrity gaps, and reproducibility

FCC is often misunderstood in two opposite directions: as if it were merely a cold snapshot, or as if it were a complete evidentiary image. Neither interpretation is accurate. The Kubernetes study explicitly contrasts cold migration, which stops a container and recreates it from persistent state without in-memory state, with FCC, which captures full process state. At the same time, the CRIU and IoT studies show that checkpoint images are not complete forensic substitutes for the surrounding filesystem, external services, or kernel state (Dinh-Tuan et al., 6 Sep 2025, Andrijauskas et al., 2024).

Environment fidelity is a central constraint. Restore may require the same directory paths used during checkpointing, compatible kernel and runtime support, and the same CPU family/type for CPU-optimized binaries. Cross-host restoration of network applications is especially restricted: simple TCP/UDP send/receive programs can be checkpointed and restored on the same machine, but cross-host migration generally fails because preserving established connections depends on conditions such as firewall configuration and keeping those connections alive during unload/load. In Kubernetes, the feature is experimental rather than general availability, and the evaluated implementation depends on CRIU plus cgroup v1 (Andrijauskas et al., 2024, Dinh-Tuan et al., 6 Sep 2025).

Integrity and chain-of-custody guarantees are absent from the reported mechanisms. The CRIU paper does not address cryptographic integrity of checkpoint images, secure transmission, chain-of-custody, tamper evidence, or protection against a malicious process modifying its own checkpoint image. The IoT paper likewise contains no explicit mechanism for verifying checkpoint integrity, detecting incomplete or corrupted checkpoints, handling power loss during checkpointing, encrypting checkpoints at rest or in transit, or controlling access to checkpoint contents. These omissions are consequential because checkpoints will likely contain sensitive data such as in-memory secrets, TCP session contents, and user data (Andrijauskas et al., 2024, Karhula et al., 2021).

The reported validation is functional rather than formal. Correctness is assessed by restoring applications, checking whether they continue computation or produce correct output, and inspecting system and kernel logs. The studies do not provide image-size-independent formal models of completeness, failure-vs-success rates over many trials, or proofs of behavioral determinism across repeated restores from the same checkpoint. The Kubernetes work similarly does not formalize strict exactly-once semantics; consistency is described relative to FIFO queue ordering and a cutoff message ID rather than through a proof of global correctness (Andrijauskas et al., 2024, Dinh-Tuan et al., 6 Sep 2025).

For evidentiary use, the literature therefore implies that FCC needs external controls around the raw checkpoint mechanism. Recommendations that follow from the CRIU-based analysis include hashing and signing checkpoint directories, logging checkpoint operations, capturing kernel version, CRIU version, container runtime and image hashes, CPU type, system architecture, and filesystem mounts and paths relevant to the container, and standardizing on compatible hardware and OS for replay. These are not built into CRIU or Docker+CRIU in the evaluated systems; they are additional layers required for forensic soundness (Andrijauskas et al., 2024).

6. Relation to transparent user-level checkpointing and broader research directions

FCC research is closely related to older work on transparent user-level checkpointing. DMTCP is a transparent user-level checkpointing package for distributed applications that demonstrates checkpoint and restart for more than 20 applications, including MATLAB, Python, TightVNC, MPICH2, OpenMPI, and runCMS. On 128 distributed cores across 32 nodes, checkpoint and restart times are typically 2 seconds with negligible run-time overhead, and forked checkpointing reduces typical checkpoint times to 0.2 seconds. DMTCP automatically accounts for fork, exec, ssh, mutexes/semaphores, TCP/IP sockets, UNIX domain sockets, pipes, ptys, terminal modes, ownership of controlling terminals, signal handlers, open file descriptors, shared open file descriptors, I/O including the readline library, shared memory via mmap, parent-child process relationships, pid virtualization, and other operating system artifacts. It is unprivileged and does not require special kernel modules or kernel patches [0701037].

This user-space lineage clarifies that FCC is not reducible to one implementation strategy. CRIU-based FCC emphasizes privileged, runtime-mediated capture of containers and Pods, including namespaces and runtime integration. DMTCP emphasizes transparent, user-level capture of distributed processes without kernel modifications. This suggests that FCC occupies a continuum between host-level checkpoint/restart and application- or process-level transparent checkpointing. A plausible implication is that hybrid systems could use CRIU or Kubernetes FCC for the container envelope and DMTCP-style coordination for cross-process or cross-container consistency, especially when a forensic snapshot must span multiple communicating services.

The main open direction is therefore not whether checkpointing can preserve rich runtime state—it can—but how that state should be packaged, authenticated, contextualized, and replayed under forensic constraints. The existing literature establishes that memory, timers, open files, threads, sockets, and namespace-related process context can be preserved with useful fidelity in supported environments; it also establishes that GPU state, kernel-only artifacts, full distributed MPI state, strong integrity guarantees, and fully deterministic replay are not delivered by the checkpoint mechanisms alone (Andrijauskas et al., 2024, Karhula et al., 2021, Dinh-Tuan et al., 6 Sep 2025).