Hardware-Based Halt-Resume Protocol

• The topic hardware-based halt-resume protocol is a mechanism implemented in hardware to pause execution while preserving essential state for accurate recovery.
• It covers diverse designs—from checkpoint-selected FPGA context switching to arbitrary-cycle preemption and persistent-stack recovery in intermittent and NVRAM systems.
• Key challenges include ensuring quiescence, managing inter-domain communication, and balancing hardware versus software control to achieve seamless resumption.

A hardware-based halt-resume protocol is a mechanism for interrupting ongoing execution implemented in hardware, preserving enough execution state to avoid restarting from the beginning, and later restoring that state so execution can continue. In the literature, this objective appears in several distinct forms: deferred checkpointing in reconfigurable circuits, arbitrary-cycle FPGA preemption through configuration readback and writeback, low-voltage-triggered preservation of architectural state in intermittent systems, persistent-stack recovery in NVRAM systems, and enclave checkpoint-transfer-restore built on TEE hardware primitives (Wicaksana et al., 2016, Malik et al., 27 Jan 2025, Chen et al., 2019, Aksenov et al., 2021, Nakatsuka et al., 2022). The resulting field is not a single protocol family with one canonical state model. Rather, it is a spectrum of designs that differ in where execution may be halted, what counts as “context,” whether resumption is instruction-exact or task-level, and how much of the mechanism resides in hardware versus software control planes.

1. Conceptual Scope and Execution Semantics

The core technical question is not merely how to stop a device, but how to stop it without losing the execution state required for correct continuation. In FPGA and SoC settings, the context is described as “an image of the state of the running circuit,” saved during interruption and restored when execution continues (Wicaksana et al., 2016). In intermittent systems, the relevant boundary is power failure: volatile state disappears, while nonvolatile state survives, so halt-resume becomes a question of preserving architectural context before energy is exhausted or reconstructing it after reboot (Chen et al., 2019). In NVRAM execution models, the surviving state is not a raw program counter and register file but a persistent stack of active invocations plus recovery metadata (Aksenov et al., 2021). In TEE migration, enclave memory is intentionally unreadable to the host, so the enclave must export, encrypt, transfer, and later import its own state (Nakatsuka et al., 2022).

A recurrent misconception is that halt-resume necessarily means an immediate, exact freeze of arbitrary machine state. The published designs do not support that as a universal property. The FPGA context-switch method demonstrated for heterogeneous reconfigurable systems is checkpoint-based: interruption may be requested externally, but the actual stop occurs when execution reaches a selected checkpoint state of the circuit finite state machine, and the user-defined “context-switch latency” is the maximum time needed to reach such a checkpoint after a halt request (Wicaksana et al., 2016). By contrast, EPOCH explicitly targets interruption “at any arbitrary clock cycle” by disabling the design clock and then extracting state through configuration readback (Malik et al., 27 Jan 2025). Intermittent hybrid-memory systems preserve mid-execution progress only for “lengthy” tasks whose stack and heap have been moved into NVM, while ordinary tasks restart from the beginning unless they had already committed (Chen et al., 2019). Persistent-stack NVRAM systems resume semantically rather than microarchitecturally: recovery code reconstructs interrupted nested operations in LIFO order instead of restoring exact volatile register state (Aksenov et al., 2021).

The term “hardware-based” is therefore best treated as graded rather than absolute. FPGA clock-gating and frame readback are close to direct hardware context capture (Malik et al., 27 Jan 2025). Threshold-triggered intermittent recovery is better described as hardware-assisted runtime recovery (Chen et al., 2019). SGX checkpoint-transfer-restore is explicitly “software-only” as a retrofit, but relies fundamentally on hardware primitives such as asynchronous exits, state save areas, and attestation (Nakatsuka et al., 2022).

2. Checkpoint-Selected Context Switching in Reconfigurable Systems

In reconfigurable computing, halt-resume first emerged as a resource-management problem. The motivating scenario is a multi-user FPGA environment in which resources are shared and “one must be able to interrupt a running circuit at any given time and continue the task at will” (Wicaksana et al., 2016). The demonstration of a context-switch method for heterogeneous reconfigurable systems did not present the full internals of the protocol, but it made the system-level orchestration concrete. A running hardware task could be interrupted on one FPGA-SoC board, its context extracted and stored, and execution later resumed on another board, including across vendors: a Xilinx ZC706 Evaluation Board and an Altera Arria V SoC Development Kit (Wicaksana et al., 2016).

The technical core of that method is automatic checkpoint selection. The cited earlier work, as summarized in the demonstration paper, performs static analysis of the circuit finite state machine and selects checkpoint states with minimum overhead. The maximum time to reach a checkpoint is defined by the user and considered the context-switch latency. This makes the halt semantics deferred rather than immediate: execution proceeds until a selected FSM state is reached, and only then is context extracted (Wicaksana et al., 2016). The paper does not specify an interrupt signal name, a halt-controller FSM, or quiescence requirements such as drained FIFOs or completed memory transactions. It also does not enumerate the exact contents of the saved context. The only defensible statement is that checkpoint choice depends on FSM analysis and that save/restore overhead depends on the selected states, which implies that different states expose different amounts of live hardware state.

Tooling is central. The checkpoint-selection method is automatic, written in C, and integrated as a plugin in the free and open-source High-Level Synthesis tool AUGH. That places context-switch support at synthesis time rather than in manually instrumented RTL (Wicaksana et al., 2016). The demonstrated runtime uses a server, network storage implemented with NFS, and two SoC-FPGA platforms whose CPUs are tightly coupled to FPGAs. The server generates the circuit configuration for the target platform and stores it, together with test vectors, in network storage. One board loads the configuration and runs; on interruption, its CPU retrieves context from the FPGA and stores it in NFS; another board later detects the bitstream and context in NFS, programs its FPGA, and continues execution (Wicaksana et al., 2016).

The importance of this design lies less in low-level protocol detail than in the architectural pattern it established. Safe halting is reduced to checkpoint reachability; checkpoint density is traded against worst-case suspension latency; and a CPU-side control plane performs configuration management, context extraction, storage, and reattachment. The paper’s own limitations are equally important: it does not define formal correctness conditions, does not specify the restore microarchitecture, and does not address memory consistency, outstanding I/O, DMA engines, or portability of raw context formats across devices (Wicaksana et al., 2016).

3. Arbitrary-Cycle FPGA Preemption and Frame-Level Snapshotting

A more direct hardware preemption model appears in EPOCH, which targets multi-tenant cloud FPGAs and presents a full save-restore-resume path for tenant designs. The design assumes a Xilinx Zynq-7000 SoC in which the processing system runs control software and the programmable logic hosts tenant accelerators or soft processors partitioned into partial reconfiguration slots or P-blocks (Malik et al., 27 Jan 2025). The central claim is that a running tenant can be interrupted at an arbitrary clock cycle, its state snapshot saved with fine-grain granularity, and later restored so execution resumes from the preserved state rather than from the application’s initial state.

EPOCH halts execution by controlling clocks rather than by waiting for precomputed checkpoints. On a preemption request, PS-side software unlocks the protected clock-control register bank with SLCR-UNLOCK and disables the tenant clock through the throttle count control register. This freezes sequential state so that no further transitions modify flip-flops, BRAMs, or other state-holding resources. Once frozen, the system performs configuration readback through PCAP (Malik et al., 27 Jan 2025). The design emphasizes common-clock-domain synchronization among benchmark IP, AXI-GPIO, and GSR handling to reduce timing hazards. The paper therefore characterizes its interruption semantics as effectively precise only under its preferred assumptions of a single/common clock domain or carefully synchronized domains; it explicitly warns that arbitrary interruption in multiple clock-domain designs can break inter-domain relationships unless extra handshake logic is inserted.

The captured context spans fundamental FPGA elements: LUTs, FFs, BRAMs, and DSPs. Granularity is frame-level, addressed through FAR metadata. In Xilinx 7-series devices, one frame contains 101 words, each word is 32 bits, and one frame readback returns 202 words: a padding frame of 101 zero words followed by the actual 101-word frame data. EPOCH discards the padding frame and stores the remaining 101 words in off-chip DRAM (Malik et al., 27 Jan 2025). To expose all relevant state, the readback sequence includes CTL0 writes that unmask LUT-related state and a GCAP command that captures FF values. For restore, saved data are reformatted into a runtime-generated partial-bitstream-like write stream, written back through PCAP, and then made visible to active logic via GSR before the clock is re-enabled (Malik et al., 27 Jan 2025).

Several engineering details distinguish EPOCH from earlier readback-based approaches. Readback and writeback have different padding placement: padding precedes actual data in readback and follows actual data in writeback. Each frame includes a CRC-32 at word position 50, but EPOCH avoids recomputing CRCs by appending a CRC reset command after frame data. BRAM frames require custom bit correction because some bits appear as 1 in readback regardless of whether BRAM content was modified; without patching, writing those bits back can revert the fabric to an old state (Malik et al., 27 Jan 2025). The implementation is highly Xilinx-specific, but the measured performance on a Zynq XC7Z020 SoC is explicit: context save per frame in 62.2 microseconds and context restore per frame in 67.4 microseconds, with zero area overhead on the reconfigurable fabric because the save/restore engine resides in PS software and existing interfaces rather than tenant-side logic (Malik et al., 27 Jan 2025).

EPOCH thus marks an important shift from checkpoint-reachability to direct state capture. It shows that full-context FPGA preemption can be engineered around configuration memory semantics, PCAP readback/writeback, PS-controlled clocking, and frame metadata. At the same time, its scope remains bounded by device-family dependence, limited resource coverage, common-clock assumptions, and the need for design-time metadata extraction and P-block floorplanning (Malik et al., 27 Jan 2025).

4. Intermittent Power, Hybrid Memory, and Persistent-State Execution

In intermittent systems, halt-resume is driven by energy collapse rather than by a scheduler. The central challenge is that power failures destroy volatile state but leave nonvolatile memory intact. One design direction avoids runtime checkpointing entirely by restructuring execution around hybrid memory and task semantics. In the FreeRTOS-based design for failure-resilient intermittent systems, the platform includes a capacitor, a voltage detector, a CPU, volatile memory, and nonvolatile memory such as FRAM. Ordinary tasks do not preserve in-flight execution state across power loss; their progress survives only when a task finishes and atomically commits modified data into NVM. “Lengthy” tasks, whose execution exceeds a power-on interval, instead allocate their stack, heap, and used memory space directly in NVM. When a low-voltage detector raises an interrupt, the recovery handler forces a context switch if the currently running task is lengthy, pushing CPU registers onto the task’s NVM-resident stack. After reboot, the task can continue from its preserved execution point by being re-added to the scheduler’s ready queue (Chen et al., 2019).

This design is primarily OS/runtime software, but its halt-resume boundary is constrained by quantitative hardware conditions. The energy stored in the capacitor is modeled as $\frac{1}{2}CV^2$, and the condition for guaranteeing enough residual energy to complete one context switch is

$$\frac{1}{2}C(V_{th}^2 - V_{op}^2) \geq P \times T_{cs},$$

leading to

$V_{th} \geq \sqrt{\frac{2PT_{cs}}{C} + V_{op}^2}.$

Using measured $P = 5.25$ mW and $T_{cs} = 1$ ms, the implementation derives $V_{th} = 2.42$ V (Chen et al., 2019). The same work provides formal correctness at the task level: conflict-serializability is enforced by backward validation with time complexity $O(N+M)$, where $N$ is the number of data objects accessed by a task and $M$ is the number of concurrently executed tasks (Chen et al., 2019). Experimentally, it reports 0 ms runtime suspension time, 0.6 ms recovery latency, and at least 90% reduction in recovery time relative to checkpointing-based baselines, with up to 43% improvement in computation progress under a relatively strong power source (Chen et al., 2019).

A related but lower-level perspective appears in the persistent-stack execution model for NVRAM programs. Here, the durable control state of each thread is a persistent stack in NVRAM, one stack per thread, containing frames with a unique function identifier, serialized arguments, and a one-byte end marker. The marker encodes the valid prefix of the stack: $0x1$ means “this is the last valid frame,” and $$\frac{1}{2}C(V_{th}^2 - V_{op}^2) \geq P \times T_{cs},$$0 means additional valid frames follow. Function invocation linearizes only when the stack end is moved forward by changing the previous frame’s marker, and function completion linearizes when the stack end is moved backward by restoring the predecessor marker to $$\frac{1}{2}C(V_{th}^2 - V_{op}^2) \geq P \times T_{cs},$$1. Recovery traverses the persistent stack from top to bottom and invokes a recovery function $$\frac{1}{2}C(V_{th}^2 - V_{op}^2) \geq P \times T_{cs},$$2 for each active frame in LIFO order (Aksenov et al., 2021).

The significance of this model is that it replaces raw architectural-state checkpointing with a durable call-structure protocol. Procedure entry requires flushing the new frame before publishing it by moving the stack end forward; procedure return requires persisting the return value before moving the stack end backward. Small return values may be stored in reserved frame space, while larger outputs are returned through caller-allocated persistent heap locations. Repeated failures are handled by leaving a frame durable until its operation or recovery has completed sufficiently to permit safe pop; if a crash occurs during recovery, that frame remains on the stack and recovery is retried after restart (Aksenov et al., 2021). This is a halt-resume protocol at function-call granularity rather than at the level of CPU registers and program counters.

NORM complements these runtime mechanisms by providing an FPGA-based emulation framework for intermittent architectures. It emulates mixed volatility by connecting POWER_RESET to volatile logic but not to emulated nonvolatile registers, and it injects configurable NVM timing through an explicit EN/BUSY handshake. Once a request is accepted, BUSY remains high for the programmed access delay; all memory-related inputs must stay constant while BUSY is high; and if a write operation is accepted, it is completed successfully even if reset arrives during the write, modeling atomic word-level persistence. In the evaluated design, NVR access time is 80 ns, which at 100 MHz corresponds to at least eight clock cycles per request, and “the first operation that all finite state machines perform after a shutdown is the recovery procedure” (Ruffini et al., 2022). NORM does not itself define a full processor-level halt-resume protocol, but it makes explicit the threshold-triggered hazard-detect → backup → reset and reboot → restore-first → continue sequencing that underlies intermittent hardware designs (Ruffini et al., 2022).

5. Host-Orchestrated Halt-Resume for Enclaves and Hardware Emulation

A distinct lineage of halt-resume protocols arises when the protected or accelerated state cannot be captured transparently by a host process. In SGX, enclave memory is unreadable to the host, OS, and migration software, so a migration mechanism must involve the enclave itself. CTR retrofits migration into existing TEE architectures by combining asynchronous exits, enclave-side export/import logic, attestation, and an external Migration Key Service. Arbitrary-point halting is achieved through interrupts that trigger SGX Asynchronous Exit, causing the processor to save thread architectural state into the enclave State Save Area and exit to non-enclave code. The enclave then exports its own data section, heap, stack, and SSA into a serialized image, encrypts that image inside the enclave using AES-GCM with a 256-bit key, and writes only ciphertext to an untrusted host buffer. The migration key is escrowed at the MKS and later released only to an attested destination enclave, which decrypts the checkpoint and restores it into a fresh enclave instance before threads resume (Nakatsuka et al., 2022).

CTR therefore provides arbitrary-point checkpoint-transfer-restore, but it is not a native hardware halt-resume facility. It explicitly depends on existing hardware primitives rather than new CPU instructions: AEX for interruption, SSA for register capture, enclave identity and remote attestation for legitimacy, and ERESUME behavior for continuation. The prototype implementation for Intel SGX reports latency that increases linearly with enclave size: for a 1 GB enclave, overall checkpoint latency is about 0.95 s and overall restore latency about 2.55 s, with destination enclave creation taking about 1.91 s and dominating the cost (Nakatsuka et al., 2022). The design preserves confidentiality and integrity of enclave state during transfer, but it does not claim to migrate all hardware-backed persistent SGX state such as monotonic counters.

Process-level checkpointing of hardware emulation exposes a related boundary problem. DMTCP’s plugin model checkpoints user-space process state, but external state associated with a hardware emulator, license server, or GUI server is not transparently captured. The adapted plugin model therefore allows an application to disconnect from the hardware emulator at checkpoint time and reconnect to a possibly different hardware emulator at restart time. External connections can be marked “external,” ignored by DMTCP during raw checkpointing, and reconstructed later by vendor-specific plugins (Garg et al., 2017). The framework provides barriers, event callbacks at checkpoint and restart, and the ability to disable checkpointing during critical sections, invoke checkpointing from within code, and delay thread resumption until the system reaches a well-behaved state.

The protocol consequence is that safe halt-resume in emulator-assisted systems requires quiescence rather than blind snapshotting. The paper explicitly warns that in-flight transactions at checkpoint time can result in lost data and inconsistent restart state, so the system must be brought to a quiescent state and in-flight data on buses must be drained before saving state (Garg et al., 2017). For shared emulator resources, barriers can be used to elect a leader that saves and restores the connection on behalf of multiple processes. DMTCP thus exemplifies a software-process resume with hardware reattachment, optionally extensible to a hybrid model if a vendor plugin can also save and restore emulator-side state (Garg et al., 2017).

6. Constraints, Misconceptions, and Open Technical Issues

The literature makes clear that halt-resume protocols are constrained first by their state model. In the checkpoint-selected FPGA method, the exact contents of the context are not enumerated, and the paper does not specify whether it includes FSM state registers, datapath registers, local RAM contents, or interface buffers (Wicaksana et al., 2016). In EPOCH, the state model is richer and more explicit—LUTs, FFs, BRAMs, and DSPs—but still excludes arbitrary specialized primitives and depends on device-family-specific command sequences, FAR metadata, and BRAM-specific corrections (Malik et al., 27 Jan 2025). In intermittent hybrid-memory systems, true mid-execution resume applies only to lengthy tasks; ordinary tasks are recreated and rerun unless they had committed, so “resume” is conditional on the task model (Chen et al., 2019). In NVRAM persistent-stack execution, the surviving state is procedural rather than architectural, which changes the meaning of correctness from exact continuation to successful nested recovery (Aksenov et al., 2021).

A second recurring issue is quiescence. Several designs assume or require a safe stop boundary but do not fully formalize it. The heterogeneous FPGA demonstration does not discuss empty pipelines, drained FIFOs, or completed memory transactions (Wicaksana et al., 2016). EPOCH freezes hardware state at a clock boundary and captures it through configuration readback, but warns that multi-clock designs require added handshake logic (Malik et al., 27 Jan 2025). DMTCP explicitly requires draining in-flight bus transactions before checkpointing emulator-attached software (Garg et al., 2017). These differences show that “halt” can denote at least three distinct conditions: arrival at a preselected checkpoint, direct freezing of synchronous logic, or global software quiescence around an external hardware interface.

Security and single-instance continuity introduce a third constraint. CTR uses one-time migration-key release through an MKS to prevent unauthorized duplication or repeated restore of the same encrypted enclave state, but it still depends on trust in the MKS and does not migrate all persistent SGX state (Nakatsuka et al., 2022). EPOCH’s restore path bypasses CRC validation by appending a CRC reset command after frame data, which simplifies writeback but can hide transmission errors (Malik et al., 27 Jan 2025). In multi-tenant FPGA systems, logic isolation derives from slot partitioning and memory isolation from dedicated DRAM regions, yet stronger memory-content security such as encryption or authentication of snapshots is left outside scope (Malik et al., 27 Jan 2025).

The remaining open questions are largely those omitted by demonstration-oriented papers. Production-quality hardware halt-resume protocols still require explicit specification of context contents, consistency conditions for memories and interfaces, side-effect handling for external I/O, portability across device families or vendors, and quantitative characterization of area, timing, and energy overheads. The published designs collectively show that hardware halt-resume is feasible under several models—checkpoint-selected, arbitrary-cycle, power-fail-triggered, persistent-stack, and enclave-cooperative—but they also show that no single mechanism covers all cases. The field remains defined by the boundary it chooses: between clocked state and configuration frames, between volatile and nonvolatile memory, between enclave secrecy and host orchestration, or between process memory and externally attached hardware.