ranDecepter: In-Host Ransomware Defense

• ranDecepter is an in-host runtime framework for Windows that intercepts API calls to detect and block user-space ransomware before file damage occurs.
• It employs staged detection using file, cryptographic, and semantic indicators—culminating in ransom note validation—without relying on decoys or ML classifiers.
• The system neutralizes destructive operations with fake-success responses and depletes attacker resources by forcing the exfiltration of fabricated victim data.

ranDecepter is an in-host runtime framework for user-space ransomware on Windows that combines active cyber deception with real-time analysis to identify ransomware at the API level, prevent destructive file-system effects through FakeSuccess responses, and then isolate confirmed samples in a controlled deceptive environment where binary rewriting forces repeated generation and exfiltration of fake victim state to attacker infrastructure (Sajid et al., 1 Aug 2025). Its design is explicitly decoy-free and ML-free in the sense used by the paper: it does not rely on honeyfiles or trained classifiers, but instead stages detection through file, crypto, and semantic indicators, culminating in ransom-note validation. The same architecture is also presented as a mechanism for attacker resource depletion, because repeated looping of the malware causes attacker backends to accumulate large volumes of fabricated per-victim entries (Sajid et al., 1 Aug 2025).

1. Scope, threat model, and motivation

ranDecepter targets a specific attack model: ransomware has already bypassed traditional defenses and is running on the victim host. The system assumes a conventional ransomware workflow consisting of file enumeration and opening through CreateFile and ReadFile, encryption through standard crypto APIs, statically linked libraries, or custom cryptographic code, writing encrypted content through WriteFile, deleting or renaming originals through DeleteFile, MoveFile, or MoveFileWithProgress, exfiltrating keys and victim identifiers to a C2 / backend, and producing ransom notes as files or wallpaper changes (Sajid et al., 1 Aug 2025). Kernel-mode ransomware that bypasses user-mode APIs is explicitly declared out of scope.

The paper motivates this focus by arguing that several defense families remain structurally insufficient. Static, signature-based, and ML-based code analysis are vulnerable to polymorphism, packing, obfuscation, and zero-day samples. Classic sandboxing is often offline and can be evaded by malware that detects sandbox artifacts. Behavioral host-based defenses such as CryptoLock and ShieldFS often trigger only after file modifications are observed, and runtime ML methods may generate false positives under heavy but benign I/O workloads. Recovery systems such as PayBreak, SSD-insider, and caching-based schemes may facilitate restoration but cannot prevent the initial damage. Backup-oriented defenses likewise do not prevent operational disruption and may themselves be targeted by backup-aware ransomware (Sajid et al., 1 Aug 2025).

A central motivation is the limitation of deception mechanisms based on decoy or honeyfiles. The paper lists false positives, operational complexity, evasion by targeted ransomware that ignores decoys, and reactive activation after partial file loss. In the comparisons reported, RWGuard loses about 288 files on average, R-Sentry allows up to 10 files, and R-Trap about 18 files, while other systems also permit partial damage (Sajid et al., 1 Aug 2025). ranDecepter is therefore positioned around four stated goals: eliminate decoys entirely, detect and contain ransomware before any user file is encrypted or deleted in almost all samples, achieve zero false positives in the reported evaluation, and extend beyond defense into offensive deception that drains attacker resources.

2. API-level identification and staged deception

The endpoint component is a Deception DLL injected into new processes via Windows Defender’s file system filter WdFilter.sys, with API hooking implemented through EasyHook in user mode (Sajid et al., 1 Aug 2025). Once injected, the DLL intercepts file, crypto, system-information, and related operations. The key architectural claim is that ranDecepter does not merely observe suspicious behavior; it actively manipulates return values so that ransomware believes destructive operations succeeded while original data remains intact.

The staged logic is central to that design.

Stage	Evidence or trigger	Main action
1	CreateFile, MoveFile, MoveFileWithProgress; suspicious extensions or renaming	Immediate termination for known ransomware extensions, otherwise escalate monitoring
2	Crypto APIs, CFS match, or high entropy	Set Encryption Flag; enable containment and FakeSuccess behavior
3	WriteFile semantics	Allow tracked writes to new files; drop overwrite-original writes and return success
4	DeleteFile after new encrypted writes	Return TRUE without deleting original
5	Ransom-note file or wallpaper semantics	Final ransomware classification; delete encrypted outputs and preserve originals

At Stage 1, ranDecepter hooks CreateFile, MoveFile, and MoveFileWithProgress, checks dwCreationDisposition for CREATE_ALWAYS and CREATE_NEW, and compares file extensions against a curated ransomware extension list denoted [ransomext]. If a known ransomware extension is used, the process is immediately terminated and handed off to the reset phase. Suspicious renaming patterns instead raise monitoring intensity rather than forcing immediate classification (Sajid et al., 1 Aug 2025).

At Stage 2, the system searches for evidence of encryption capability. This evidence can arise from standard crypto APIs such as CryptEncrypt, AES_encrypt, or [AES](https://www.emergentmind.com/topics/automatic-essay-scoring-aes)::Encryption; from Cryptographic Function Signatures (CFS) for statically linked or proprietary crypto; or from entropy-based heuristics for custom cryptographic routines. The entropy measure is described in terms of standard Shannon entropy,

$H(X) = - \sum_{i=1}^{n} p(x_i)\log_2 p(x_i)$

where $p(x_i)$ is the estimated probability of byte value $x_i$ in a buffer. High, uniform entropy consistent with bulk encryption triggers the Encryption Flag even when no known crypto API or CFS match is present (Sajid et al., 1 Aug 2025).

At Stage 3, ranDecepter distinguishes between two behavioral forms. For new-file encryption, writes to a new destination created by CreateFile(D) are allowed but tracked, so that they can later be deleted if the process is confirmed as ransomware. For overwrite-original behavior, identified by SetFilePointer(O) followed by WriteFile(O) on the same handle, the system drops the write and returns success when the Encryption Flag is set and Application Restart Control (ARC) is enabled. If ARC is disabled, a temporary shadow copy is made before overwrite (Sajid et al., 1 Aug 2025).

At Stage 4, DeleteFile(O) calls following writes to encrypted new files are intercepted and made to return TRUE without deleting the original. When classification is later resolved, ranDecepter either re-applies the deferred deletion for benign software or deletes the tracked encrypted destination files for confirmed ransomware (Sajid et al., 1 Aug 2025).

At Stage 5, semantic confirmation occurs. ranDecepter monitors CreateFile in user-sensitive directories such as Desktop, Documents, and Downloads; scans created files for keywords including “ransom”, “encrypt(ed)”, “decrypt(ed)”, “payment”, “bitcoin”, “delete(d)”, and “lose”; and tracks wallpaper changes, applying OCR via Google Cloud Vision API followed by keyword matching. Final ransomware classification requires this semantic layer in combination with the earlier behavioral evidence (Sajid et al., 1 Aug 2025).

This staging has a specific methodological implication. Because destructive writes and deletes have already been neutralized, the framework can wait for a stronger signal than pure encryption behavior alone. That is the basis for the paper’s claim that encryption tools such as 7zip and VeraCrypt are not misclassified despite using crypto and sometimes altering files (Sajid et al., 1 Aug 2025).

3. Offline knowledge construction and binary orchestration

The full architecture is divided into three major phases: an Offline Phase, a Real-time Identification Phase, and a Reset Phase (Binary Orchestration) (Sajid et al., 1 Aug 2025). The offline phase builds the knowledge that informs runtime interception and later binary rewriting.

During Data Collection & Dynamic Analysis, the authors execute 521 ransomware samples from 30 families in a customized Cuckoo Sandbox, trace WinAPI and crypto call sequences, and construct Malicious Subgraphs (MSGs). In these graphs, nodes are API calls such as WinAPI, CryptoAPI, OpenSSL, or Crypto++ functions, and edges encode data-flow dependencies such as a key handle passed from CryptGenKey to CryptEncrypt, or file handles flowing from CreateFile to WriteFile to DeleteFile (Sajid et al., 1 Aug 2025). MSGs are used to capture behavioral workflows including file-encryption chains and key-transfer routines.

The Cryptographic Function Signatures component addresses statically linked or proprietary routines that cannot be recognized through dynamic linkage. The paper describes fingerprints of about 32 bytes, built from characteristic constants such as AES S-Boxes, instruction patterns such as aesenc, and control-flow skeletons of cryptographic routines. These signatures support lightweight in-memory matching during runtime, particularly around WriteFile (Sajid et al., 1 Aug 2025).

The deception plan derived from MSGs emphasizes behavioral semantics rather than a fixed set of library calls. Two canonical file-encryption patterns are extracted. In Pattern 1: Write-to-new-file, the chain is CreateFile(O) → CreateFile(D) → Encrypt(CO) → WriteFile(D) → CloseHandle(O,D) → DeleteFile(O), and the deception plan is to allow the chain except for DeleteFile(O), which is intercepted and faked. In Pattern 2: Overwrite-original, the chain is CreateFile(O) → ReadFile(O) → Encrypt(CO) → SetFilePointer(O) → WriteFile(O) → CloseHandle(O), and the deception plan is to drop the overwrite while returning success (Sajid et al., 1 Aug 2025). Because this logic is defined at the level of behavior, the paper argues that it applies across CryptoAPI, OpenSSL, Crypto++, and related implementations.

The Reset Phase extends the system from prevention into active deception against attacker infrastructure. It is performed in a controlled deceptive environment and consists of three components: a Symbolic Binary Analyzer (SBA) built on gExtractor with selective symbolic execution and support for roughly 390 APIs; an Actuating Agent that reconstructs execution chains and identifies where the “Transfer Key(s)” routine ends; and a second Deceptor DLL that performs data hooking and control-flow redirection (Sajid et al., 1 Aug 2025).

The SBA identifies the binary entry point, traces user-space API calls together with arguments, return values, call stacks, and return addresses, and then locates the final API in the key-exfiltration chain. The Actuating Agent classifies each sample into EC1, where “Transfer Key(s)” occurs before file encryption operations, or EC2, where it occurs after the encryption/deletion loop (Sajid et al., 1 Aug 2025). This distinction determines whether the later patch must also skip the file-encryption chain.

The reset Deceptor DLL then performs two manipulations. First, it hooks fingerprinting and randomness APIs for MAC address, system time, RNG, OS information, and related calls, returning randomized or misleading values each loop so that each iteration appears to be a new victim. Second, it inserts a JMP at the return site of the last “Transfer Key” API, redirecting execution back to the binary entry point while restoring stack base and limit values saved from the initial suspended launch. For EC2, it additionally inserts a JMP that skips the FindFirstFile → ... → DeleteFile encryption loop so that files are not repeatedly touched while keys and victim IDs continue to be sent (Sajid et al., 1 Aug 2025).

A common misconception would be that ranDecepter fabricates cryptographic material by re-implementing the malware’s cryptography. The paper explicitly states the opposite: it does not explicitly re-write keys, but instead allows the ransomware to generate legitimate keys and victim IDs based on fake system parameters, so that the resulting records correspond to fake victims rather than altered cryptographic payloads (Sajid et al., 1 Aug 2025). This design avoids breaking the malware’s own logic and plausibly increases the chance that the backend will store the entries as normal infections.

4. Containment policy, deployment modes, and operational semantics

The runtime framework exposes two containment policies under Application Restart Control (ARC). Partial containment intercepts destructive overwrites but allows encryption to new files, preserving originals while deferring deletions. Full containment blocks both new encryptions and deletions, at the cost that benign applications may require restarting after classification (Sajid et al., 1 Aug 2025). The platform also maintains whitelisting, where processes can be manually or automatically whitelisted after n consecutive clean runs with no ransomware detection; associated metadata is encrypted and protected against tampering (Sajid et al., 1 Aug 2025).

The system is deployable in two legal and operational modes. In split deployment, offline analysis and the reset phase run on trusted or authorized infrastructure such as DoD/CISA/HSI servers or other LEA-controlled systems, while the real-time DLL is deployed on endpoints. In honeypot deployment, the entire stack runs on LEA-managed honeypots or sacrificial hosts (Sajid et al., 1 Aug 2025). This separation is important because the endpoint module is defensive and data-preserving, whereas the reset phase has an explicitly counter-offensive character.

From an implementation perspective, ranDecepter is focused on Windows, though the paper states that the approach is conceptually portable to POSIX calls on Linux. The research stack includes EasyHook for user-mode hooking, WdFilter.sys for process injection, a customized Cuckoo Sandbox for offline extraction, gExtractor for symbolic binary analysis, Google Cloud Vision API for wallpaper OCR, and WinAppDriver for automated interaction and response-time measurement of benign applications (Sajid et al., 1 Aug 2025). The prototype and artifacts are described as being available in an anonymized open repository titled “ranDecepter Artifact and Code Repository” (Sajid et al., 1 Aug 2025).

Operationally, the design supports integration into endpoint and SOC workflows even though direct EDR/IDS/SIEM connectors are not explicitly implemented. The paper states that Deception DLL logs could be federated to a SIEM and that confirmed Stage 5 events could trigger incident-response playbooks such as host isolation, memory capture, or binary upload to analysis infrastructure (Sajid et al., 1 Aug 2025). This suggests a deployment model in which ranDecepter functions as both a prevention mechanism and a telemetry source.

5. Evaluation, empirical results, and resource-depletion effects

The principal ransomware evaluation uses 1,134 real-world samples from 30 families, including WannaCry, CryptoWall, Cerber, GandCrab, LockBit, Dharma, TeslaCrypt, Maze, Conti, and NotPetya, collected from VirusTotal and Any.Run and limited to the last five years (Sajid et al., 1 Aug 2025). The benign corpus is described in two ways in the provided material: the abstract states evaluation using twelve benign applications, while the detailed evaluation section states 45 popular Windows applications, with 12 “representative” ones emphasized, including 7zip, Firefox, WinSCP, Opera, MySQL Workbench, VeraCrypt, and MS Word (Sajid et al., 1 Aug 2025). Both figures appear in the paper materials and therefore describe different reporting granularities rather than necessarily contradictory experiments.

The ransomware detection results are broken down by stage. 204 samples were detected at Stage 1 through CreateFile with a known ransomware extension and incurred 0 file loss. 895 samples were detected through the combination of encryption/deletion behavior and ransom-note semantics, also with 0 file loss because writes and deletes were faked under ARC and FakeSuccess. 16 samples using custom or embedded cryptography were detected via high entropy and incurred an average loss of 7 files; these are the only cases with non-zero damage. 19 samples never reached encryption because a C2 trigger was missing, though they opened network ports, and they caused no file loss (Sajid et al., 1 Aug 2025).

The aggregate result is that 98.5% (1115/1134) of samples resulted in 0 file loss, while 1.5% (16/1134) lost ≤7 files. The paper reports 100% identification accuracy for ransomware, and the detailed evaluation reports 0 false positives across 45 benign applications (Sajid et al., 1 Aug 2025). The paper attributes this outcome to a conservative final decision rule: encryption plus deletion behavior alone does not trigger definitive ransomware classification; a ransom note or wallpaper semantics are also required, and the system can afford to wait because destructive effects have already been neutralized (Sajid et al., 1 Aug 2025).

Performance measurements were obtained by running benign workloads with and without ranDecepter and measuring task completion time via WinAppDriver. The reported overheads for the subset shown in Table 4b are 5.73% for 7zip, 5.44% for Firefox, 4.61% for WinSCP, 3.79% for Opera, and 3.69% for MySQL Workbench, with other applications ranging from 0.40% to 2.99% (Sajid et al., 1 Aug 2025). The paper therefore characterizes runtime impact as minimal, including for heavy operations such as compressing a 125MB folder.

The resource-depletion experiments use 4 open-source ransomware projects with accessible C2 code: NekRos (r1), Cryptonite (r2), Jasmin (r3), and Mauri870’s PoC ransomware (r4) (Sajid et al., 1 Aug 2025). For each family, the reset phase extracted key-transfer endpoints, patched the binary, and then executed it with 50 agents over 24 hours while counting entries and storage on the attacker-side database. The per-family results are as follows:

Family	Entries with 50 agents (1h / 12h / 24h)	Storage (1h / 12h / 24h)
r1	290K / 3,420K / 6,910K	~43 / 527 / 1054 MB
r2	131K / 1,582K / 3,081K	~15 / 189 / 385 MB
r3	413K / 4,829K / 9,223K	~52.9 / 603 / 1189 MB
r4	195K / 2,257K / 4,357K	~22.7 / 272 / 544 MB

These experiments are reported to confirm that binary orchestration correctly located and looped the “Transfer Key(s)” logic in 100% of tested cases and that a moderate number of agents can bloat attacker databases substantially (Sajid et al., 1 Aug 2025). The maximal reported figure is 9,223K entries in 24 hours for r3, corresponding to about 1.1 GB of attacker storage (Sajid et al., 1 Aug 2025).

6. Relation to prior defenses, limitations, and contested implications

In the paper’s comparison against RWGuard, R-Sentry, R-Trap, and R-Locker, ranDecepter differs in four stated ways: it uses no decoys, performs final validation through ransom note check plus staged behaviors, reports no false positives in its own benign tests, and relies on rule-based staging with CFS and entropy heuristics rather than ML models (Sajid et al., 1 Aug 2025). The reported file-loss comparison is particularly central: ranDecepter yields 0 files lost in 98.5% of samples and ≤7 files in the remaining 1.5%, whereas the cited values for alternatives are much larger or partially reported (Sajid et al., 1 Aug 2025).

This positioning supports the paper’s broader claim that the framework decouples detection time from file damage. Because WriteFile and DeleteFile can be neutralized before confirmation, the framework can use a later, more conservative semantic signal without sacrificing user data. A plausible implication is that this design targets a longstanding trade-off in ransomware defense: earlier triggers often reduce damage but increase false positives, whereas later triggers improve specificity but usually tolerate more damage. ranDecepter’s API-level deception is presented as a way to relax that trade-off (Sajid et al., 1 Aug 2025).

The paper also states several technical assumptions and limitations. The system presumes user-space API visibility, continued use of WinAPI or recognizable crypto patterns, and the presence of a file-based ransom note or wallpaper change. Malware that uses kernel APIs, SSDT manipulation, or direct disk writes without passing through hooked user-mode APIs could evade detection. Advanced malware might also use anti-hooking or self-integrity checks such as IAT/EAT verification or inline integrity checks to detect or remove hooks. The current design focuses mainly on parent-child relationships, so inter-process or multi-process ransomware using side channels may be less visible (Sajid et al., 1 Aug 2025).

The resource-depletion component has its own assumptions. It depends on attackers maintaining persistent per-victim state in a backend rather than operating in a trivially stateless fashion. The authors acknowledge that sophisticated adversaries might introduce sanity checks, limit infections per IP address, or move toward more stateless architectures. Even under those conditions, the paper argues that endpoint-level neutralization still remains effective and that resource depletion is an additional offensive benefit, not the sole defensive outcome (Sajid et al., 1 Aug 2025).

Finally, the counter-offensive element raises legal and ethical questions. The paper explicitly asks who is authorized to manipulate attacker infrastructure and whether such activity could be construed as “hacking back.” It therefore proposes LEA or government cybersecurity organizations, including DoD/CISA/HSI, as the primary operators of the reset phase (Sajid et al., 1 Aug 2025). Another stated concern is that polluting attacker infrastructure might interfere with key retrieval for victims who have already paid. The architecture’s separation between the endpoint defense module and the reset infrastructure is intended to make it easier to confine offensive operations to legally authorized environments (Sajid et al., 1 Aug 2025).

Taken together, ranDecepter defines a ransomware-defense paradigm centered on API-centric prevention, semantic confirmation, and controlled binary reuse. Within the scope claimed by the paper, it is a system for both preserving endpoint data integrity and exploiting ransomware’s dependence on per-victim key management as a deception surface (Sajid et al., 1 Aug 2025).