RX-INT: Real-Time In-Memory Threat Detector
- RX-INT is a kernel-assisted system for real-time detection and analysis of fileless and in-memory threats on 64-bit Windows, enabling rapid identification of stealth attacks.
- It employs an event-triggered thread-creation tripwire and stateful VAD scanning to minimize time-of-check-to-time-of-use windows against tactics like manual mapping and module stomping.
- The system integrates kernel and user-mode components, including an automated import resolver, to enhance forensic analysis and outperform conventional tools such as PE-sieve.
RX-INT is a kernel-assisted system for real-time detection and analysis of fileless and in-memory threats on 64-bit Windows. It is designed for adversaries that remain in user mode, potentially with administrative privileges, and evade conventional defenses by executing inside the address space of otherwise legitimate processes. Its defining architectural claim is that it reduces time-of-check-to-time-of-use exposure by coupling a real-time thread-creation tripwire to a stateful Virtual Address Descriptor (VAD) scanner that tracks both private and image-backed memory over time, rather than relying only on Portable Executable artifacts or periodic user-mode scans (Juneja, 5 Aug 2025).
1. Scope, threat model, and problem setting
RX-INT addresses a class of evasive execution techniques that operate entirely in memory. The paper explicitly centers on manual PE mapping, module stomping, threadless injection via mechanisms such as QueueUserAPC, thread hijacking, and fake VEH or callback-based control-flow redirection, as well as post-injection cloaking methods including fake thread start addresses and hiding threads from user-mode debuggers (Juneja, 5 Aug 2025). In this threat model, the attacker avoids disk artifacts and abuses the address space of a trusted process, making conventional signature-based detection intrinsically unreliable.
The system is motivated by two deficiencies in prior approaches. The first is dependence on PE structures. A manually mapped DLL can have its DOS and NT headers erased, its data directories cleaned, and its loader metadata removed, which breaks tools that assume recognizable MZ signatures or normal loader linkage. The second is exposure to TOCTOU races. In module stomping, an attacker can overwrite executable bytes in a legitimate MEM_IMAGE region, execute the payload, and restore the original bytes before a periodic scan arrives. RX-INT is explicitly presented as an answer to both problems: it does not require PE headers for its core detections, and it attempts to shrink the race window by triggering memory analysis directly from execution-related events (Juneja, 5 Aug 2025).
A recurring conceptual difficulty is that executable memory is not inherently malicious. Browsers, games, and managed runtimes legitimately allocate or generate executable MEM_PRIVATE pages. RX-INT therefore does not treat all executable memory as suspicious. Instead, it differentiates between private and image-backed regions, reasons about whether a region is new relative to a baseline, and treats modification of executable MEM_IMAGE pages as an integrity event rather than as a mere allocation event. This distinction is central to its detection semantics.
2. System architecture and operating model
RX-INT has a split design consisting of a user-mode terminal interface client, rx-tui.exe, and a kernel-mode KMDF driver, rxint.sys (Juneja, 5 Aug 2025). The client controls start and stop monitoring, displays status and memory usage, and includes an injection suite for validation. The driver performs the actual inspection logic: thread monitoring, VAD-aware memory scanning, suspicious-region dumping, and import-resolution reporting.
The user/kernel boundary is implemented through a device interface. The driver creates \Device\RxInt and the symbolic link \??\RxInt; the client opens the device with CreateFile and communicates through DeviceIoControl. The control path carries an RXINT_MONITOR_INFO structure containing the target PID and dump-path format. This arrangement is operationally conventional, but the paper’s emphasis is not on the interface itself so much as on the internal coordination between its kernel subsystems.
Three kernel components form the core of the detector: a real-time thread creation monitor, a stateful VAD scanner, and an in-kernel automated import resolver. These are coordinated by a central Detector class (Juneja, 5 Aug 2025). Monitoring begins with baseline construction: the driver records committed regions and hashes executable image-backed regions. It then continues in a hybrid mode, combining periodic scanning with event-triggered scans. The event path is the paper’s main architectural claim. A suspicious thread creation acts as a tripwire; the callback remains lightweight and, rather than performing a full scan itself, wakes the VAD scanner with KeSetEvent, prompting an immediate out-of-band reinspection of the process memory map.
This design is intended to reduce the race window from the full polling interval to roughly the interval immediately after execution begins. The paper does not claim elimination of race conditions in an absolute sense, but it presents this event-triggered wake-up path as the main reason RX-INT can catch module stomping with fast cleanup where periodic user-mode tools may fail (Juneja, 5 Aug 2025).
3. Detection engine, VAD state, and memory semantics
The thread monitor is implemented via PsSetCreateThreadNotifyRoutine, with a callback named OnThreadNotify (Juneja, 5 Aug 2025). When a new thread is created, RX-INT queries the start address using ZwQueryInformationThread and determines the region in which that address lies. If the start address resolves to suspicious executable MEM_PRIVATE not associated with a known module through a PEB walk, the event is treated as an immediate detection. If the start address lies inside known MEM_IMAGE, the callback treats it as a possible module-stomp hint and signals the VAD scanner instead of convicting the event directly. The callback is deliberately kept lightweight because it executes at APC_LEVEL, where long-running scans would be unsafe.
The VAD scanner is implemented as a dedicated worker thread created with PsCreateSystemThread. It traverses the target process memory with ZwQueryVirtualMemory, beginning from a null base address, and records committed regions in a baseline snapshot (Juneja, 5 Aug 2025). RX-INT maintains two kinds of state. For MEM_PRIVATE, it stores structural metadata such as base address, region size, and protection flags. For executable MEM_IMAGE, it copies the region with MmCopyVirtualMemory and computes an XXH64 hash, preserving an integrity baseline for later comparison. The paper’s key formal rule for image-backed code integrity is the hash inequality
which is treated as evidence that an executable image-backed region has been modified (Juneja, 5 Aug 2025).
This yields two primary VAD-level heuristics. The first is the appearance of a new executable MEM_PRIVATE region absent from the baseline. This targets shellcode-style allocation, manual mapping, header-erased private payloads, and some VirtualProtectEx-based execution setups. The second is a hash mismatch in an executable MEM_IMAGE region present in both baseline and current snapshots, which targets module stomping, illicit patching, and inline modification of trusted code (Juneja, 5 Aug 2025). The distinction between MEM_PRIVATE and MEM_IMAGE is not cosmetic; it is the basis for how RX-INT suppresses noise. Private memory is handled structurally because hashing it broadly would be expensive and unstable. Image-backed memory is handled as an integrity problem because its executable contents are expected to remain stable.
The paper also describes one fast path in the callback: a thread start in suspicious MEM_PRIVATE not tied to a known module is elevated directly, while a thread start in known image memory merely forces immediate rescanning. This asymmetry reflects the system’s bias toward memory-state reasoning rather than purely behavioral inference. It also explains why RX-INT is positioned somewhere between memory forensics and EDR telemetry: it is event-triggered, but its decisive evidence is usually a memory-state differential rather than a high-level process-behavior graph.
4. Automated analysis, implementation, and forensic output
A notable part of RX-INT is its in-kernel automated import resolver. The ExportResolver walks the target process’s PEB InLoadOrderModuleList, copies PE headers of loaded modules, and parses the IMAGE_EXPORT_DIRECTORY, Export Address Table, Name Pointer Table, and Ordinal Table (Juneja, 5 Aug 2025). If an export entry resolves to a forwarder string, the resolver records that fact. When a suspicious region is dumped, RX-INT scans the raw memory for candidate 8-byte pointers and matches them against known export addresses from the snapshot. Symbolic resolutions are then written to a companion report.
This mechanism matters because RX-INT is intentionally prepared to dump raw memory that may not contain valid PE headers. In the manual-map case study, the dumped region could be loaded into IDA Pro as raw binary and manually rebased to 0x7FFBFBBE1000; the report then helped resolve an unknown call target, later confirmed in x64dbg as user32.dll!MessageBoxW (Juneja, 5 Aug 2025). The paper frames this as more than a detection aid. It is an analysis aid that shortens triage time for headerless implants and detached payloads.
The implementation itself is presented as a KMDF driver written in modern C++ with heavy use of RAII. The wrappers named in the paper are ProcessReference, which encapsulates PsLookupProcessByProcessId and ObDereferenceObject; SpinLockGuard, which wraps KeAcquireSpinLock and KeReleaseSpinLock; and ProcessAttacher, which wraps KeStackAttachProcess and KeUnstackDetachProcess (Juneja, 5 Aug 2025). This is framed as an engineering choice to reduce kernel resource leaks and synchronization errors rather than as a detection innovation. The same section emphasizes tracked paged-pool and non-paged-pool allocations, callback lightweightness, and deadlock avoidance as practical constraints on detector design.
5. Empirical results and comparison with PE-sieve
The evaluation was conducted on a Windows 11 24H2 virtual machine with 8 GB RAM and 4 vCPUs, on a host with an Intel Core i9-13900H and 64 GB DDR5. Primary testing targeted notepad.exe, with additional tests on chrome.exe, explorer.exe, gmod.exe, and a .NET/mono process. The main comparison baseline was PE-sieve v0.4.1, invoked as pesieve64.exe /pid <PID> (Juneja, 5 Aug 2025).
The paper’s central empirical claim is not that RX-INT supersedes all competing approaches, but that it detects several evasive scenarios missed by PE-sieve. The reported comparison is as follows:
| Attack scenario | RX-INT | PE-sieve |
|---|---|---|
| Standard Load | Detected | Detected |
| Header Erasure | Detected | Detected as detached module |
| PEB Unlinking | Detected | Detected as detached module |
| PEB Unlinking + Header Erasure | Detected | Error: “Could not read remote PE” |
| Manual Map + Headers Intact | Detected | Detected as UNMAPPED |
Threadless Manual Map + Header Erasure (QueueUserAPC) |
Detected | Missed (0 suspicious) |
| Manual Map + Header Erasure | Detected | Missed (0 suspicious) |
| Module Stomping | Detected | Missed (0 suspicious) |
Two case studies are especially important. In the manual-map scenario, a 64-bit DLL was manually mapped and its first 0x1000 bytes were zeroed. PE-sieve reported no implants, whereas RX-INT detected the payload as a new executable MEM_PRIVATE region not present in baseline (Juneja, 5 Aug 2025). In the module-stomping scenario, the Beep function in kernel32.dll was overwritten, executed, and restored quickly. RX-INT succeeded because CreateRemoteThread triggered the thread callback, the callback treated execution from MEM_IMAGE as a stomp hint, and the re-hash of the image region occurred before restoration completed.
The benign-workload tests are more limited but still informative. A 10-minute monitoring session on chrome.exe with multiple tabs and JavaScript-heavy sites produced zero detections, which the paper uses as evidence that the system can tolerate at least some JIT-heavy workloads (Juneja, 5 Aug 2025). By contrast, a false positive occurred on a .NET/mono process because the heuristic “new executable MEM_PRIVATE” also matches benign JIT behavior. This is the paper’s clearest acknowledged false-positive mode.
The reported overhead is modest. Non-paged pool usage was approximately 340 KB; paged pool while monitoring a complex process such as gmod.exe averaged 6.07 MB, largely due to the one-time export-resolver snapshot. CPU cost was reported as less than 0.1% during idle monitoring, 0.79% kernel CPU weight for a DEBUG build during active monitoring of chrome.exe, and 0.46% for the RELEASE build (Juneja, 5 Aug 2025). The paper interprets these values as operationally negligible.
6. Limitations, interpretive boundaries, and future directions
RX-INT is presented as a proof-of-concept architecture rather than as a complete endpoint defense platform. The evaluation is limited to curated attacks, a small process set, and one main public comparison tool. The system does not claim immunity to kernel-mode adversaries; the paper explicitly notes that a malicious kernel component could tamper with callbacks, queried kernel functions, or RX-INT state itself (Juneja, 5 Aug 2025). Its resilience is therefore conditional on the assumption that the kernel has not already been compromised.
The main current semantic limitation is benign executable MEM_PRIVATE. That same heuristic makes RX-INT effective against manual maps and shellcode-style allocation, but it also causes the false positive on a managed runtime process. The paper does not introduce a formal scoring model to resolve this ambiguity. Practical discrimination remains heuristic and state-based: region type, execute permissions, baseline membership, image-hash stability, and anomalous thread start addresses (Juneja, 5 Aug 2025). A common misconception would be to treat kernel visibility as sufficient for perfect classification; the paper does not support that conclusion.
Another limitation is partial dependence on thread creation as a tripwire. The paper emphasizes threadless injection, including QueueUserAPC, as within scope, and its benchmark includes a successful detection of a threadless manual map with header erasure. Even so, attacks that avoid both the thread trigger and obvious differential VAD signals are implicitly more dependent on periodic scanning cadence. The architecture narrows the TOCTOU window, but it does not convert all execution into synchronous trap-based integrity enforcement.
The future direction proposed in the paper is a hypervisor-based redesign using Intel VT-x and Extended Page Tables. In that model, executable module pages could be marked read-only in EPT and a stomp attempt would produce an immediate EPT_VIOLATION, moving detection out of the guest kernel and into VMX root mode (Juneja, 5 Aug 2025). This suggests a stronger integrity model than software hashing and callback-triggered rescans, and a plausible implication is that RX-INT’s core idea is not tied to KMDF as such, but to a broader design principle: stateful memory integrity monitoring should be coupled to execution-adjacent triggers and should avoid sole dependence on PE structure recovery.
Within the paper’s own framing, RX-INT’s practical domains are anti-cheat, memory security, malware detection, and reverse engineering (Juneja, 5 Aug 2025). Its significance lies less in any single heuristic than in the way it combines kernel residency, event-triggered wake-up, VAD-differential state, executable-image hashing, and automated symbolic enrichment of raw dumps into one detector. That architecture is the central contribution.