---
title: 'ranDecepter: In-Host Ransomware Defense'
url: https://www.emergentmind.com/topics/randecepter
type: topic
---

# ranDecepter: In-Host Ransomware Defense

ranDecepter is an in-host runtime framework for **user-space ransomware on Windows** that combines **active cyber deception** with **real-time analysis** to identify ransomware at the API level, prevent destructive file-system effects through **FakeSuccess** responses, and then isolate confirmed samples in a controlled deceptive environment where binary rewriting forces repeated generation and exfiltration of fake victim state to attacker infrastructure [2508.00293]. Its design is explicitly **decoy-free** and **ML-free** in the sense used by the paper: it does not rely on honeyfiles or trained classifiers, but instead stages detection through file, crypto, and semantic indicators, culminating in ransom-note validation. The same architecture is also presented as a mechanism for **attacker resource depletion**, because repeated looping of the malware causes attacker backends to accumulate large volumes of fabricated per-victim entries [2508.00293].

## 1. Scope, threat model, and motivation

ranDecepter targets a specific attack model: **ransomware has already bypassed traditional defenses and is running on the victim host**. The system assumes a conventional ransomware workflow consisting of file enumeration and opening through `CreateFile` and `ReadFile`, encryption through standard crypto APIs, statically linked libraries, or custom cryptographic code, writing encrypted content through `WriteFile`, deleting or renaming originals through `DeleteFile`, `MoveFile`, or `MoveFileWithProgress`, exfiltrating keys and victim identifiers to a **C2 / backend**, and producing **ransom notes** as files or wallpaper changes [2508.00293]. **Kernel-mode ransomware** that bypasses user-mode APIs is explicitly declared **out of scope**.

The paper motivates this focus by arguing that several defense families remain structurally insufficient. Static, signature-based, and ML-based code analysis are vulnerable to polymorphism, packing, obfuscation, and zero-day samples. Classic sandboxing is often offline and can be evaded by malware that detects sandbox artifacts. Behavioral host-based defenses such as CryptoLock and ShieldFS often trigger only **after file modifications** are observed, and runtime ML methods may generate **false positives** under heavy but benign I/O workloads. Recovery systems such as PayBreak, SSD-insider, and caching-based schemes may facilitate restoration but **cannot prevent the initial damage**. Backup-oriented defenses likewise do not prevent operational disruption and may themselves be targeted by backup-aware ransomware [2508.00293].

A central motivation is the limitation of **deception mechanisms based on decoy or honeyfiles**. The paper lists false positives, operational complexity, evasion by targeted ransomware that ignores decoys, and reactive activation after partial file loss. In the comparisons reported, **RWGuard** loses about **288 files on average**, **R-Sentry** allows up to **10 files**, and **R-Trap** about **18 files**, while other systems also permit partial damage [2508.00293]. ranDecepter is therefore positioned around four stated goals: eliminate decoys entirely, detect and contain ransomware **before any user file is encrypted or deleted** in almost all samples, achieve **zero false positives** in the reported evaluation, and extend beyond defense into **offensive deception** that drains attacker resources.

## 2. API-level identification and staged deception

The endpoint component is a **Deception DLL** injected into new processes via **Windows Defender’s file system filter `WdFilter.sys`**, with API hooking implemented through **EasyHook** in **user mode** [2508.00293]. Once injected, the DLL intercepts file, crypto, system-information, and related operations. The key architectural claim is that ranDecepter does not merely observe suspicious behavior; it **actively manipulates return values** so that ransomware believes destructive operations succeeded while original data remains intact.

The staged logic is central to that design.

| Stage | Evidence or trigger | Main action |
|---|---|---|
| 1 | `CreateFile`, `MoveFile`, `MoveFileWithProgress`; suspicious extensions or renaming | Immediate termination for known ransomware extensions, otherwise escalate monitoring |
| 2 | Crypto APIs, **CFS** match, or high entropy | Set **Encryption Flag**; enable containment and **FakeSuccess** behavior |
| 3 | `WriteFile` semantics | Allow tracked writes to new files; drop overwrite-original writes and return success |
| 4 | `DeleteFile` after new encrypted writes | Return `TRUE` without deleting original |
| 5 | Ransom-note file or wallpaper semantics | Final ransomware classification; delete encrypted outputs and preserve originals |

At **Stage 1**, ranDecepter hooks `CreateFile`, `MoveFile`, and `MoveFileWithProgress`, checks `dwCreationDisposition` for `CREATE_ALWAYS` and `CREATE_NEW`, and compares file extensions against a curated ransomware extension list denoted **[ransomext]**. If a known ransomware extension is used, the process is immediately terminated and handed off to the reset phase. Suspicious renaming patterns instead raise monitoring intensity rather than forcing immediate classification [2508.00293].

At **Stage 2**, the system searches for evidence of **encryption capability**. This evidence can arise from standard crypto APIs such as `CryptEncrypt`, `AES_encrypt`, or `AES::Encryption`; from **Cryptographic Function Signatures (CFS)** for statically linked or proprietary crypto; or from **entropy-based heuristics** for custom cryptographic routines. The entropy measure is described in terms of standard Shannon entropy,

$$
H(X) = - \sum_{i=1}^{n} p(x_i)\log_2 p(x_i)
$$

where \( p(x_i) \) is the estimated probability of byte value \(x_i\) in a buffer. High, uniform entropy consistent with bulk encryption triggers the **Encryption Flag** even when no known crypto API or CFS match is present [2508.00293].

At **Stage 3**, ranDecepter distinguishes between two behavioral forms. For **new-file encryption**, writes to a new destination created by `CreateFile(D)` are allowed but tracked, so that they can later be deleted if the process is confirmed as ransomware. For **overwrite-original** behavior, identified by `SetFilePointer(O)` followed by `WriteFile(O)` on the same handle, the system drops the write and returns success when the **Encryption Flag** is set and **Application Restart Control (ARC)** is enabled. If ARC is disabled, a temporary **shadow copy** is made before overwrite [2508.00293].

At **Stage 4**, `DeleteFile(O)` calls following writes to encrypted new files are intercepted and made to return `TRUE` without deleting the original. When classification is later resolved, ranDecepter either re-applies the deferred deletion for benign software or deletes the tracked encrypted destination files for confirmed ransomware [2508.00293].

At **Stage 5**, semantic confirmation occurs. ranDecepter monitors `CreateFile` in user-sensitive directories such as Desktop, Documents, and Downloads; scans created files for keywords including “ransom”, “encrypt(ed)”, “decrypt(ed)”, “payment”, “bitcoin”, “delete(d)”, and “lose”; and tracks wallpaper changes, applying OCR via **Google Cloud Vision API** followed by keyword matching. Final ransomware classification requires this semantic layer in combination with the earlier behavioral evidence [2508.00293].

This staging has a specific methodological implication. Because destructive writes and deletes have already been neutralized, the framework can wait for a **stronger signal** than pure encryption behavior alone. That is the basis for the paper’s claim that encryption tools such as **7zip** and **VeraCrypt** are not misclassified despite using crypto and sometimes altering files [2508.00293].

## 3. Offline knowledge construction and binary orchestration

The full architecture is divided into three major phases: an **Offline Phase**, a **Real-time Identification Phase**, and a **Reset Phase (Binary Orchestration)** [2508.00293]. The offline phase builds the knowledge that informs runtime interception and later binary rewriting.

During **Data Collection & Dynamic Analysis**, the authors execute **521 ransomware samples from 30 families** in a customized **Cuckoo Sandbox**, trace WinAPI and crypto call sequences, and construct **Malicious Subgraphs (MSGs)**. In these graphs, nodes are API calls such as WinAPI, CryptoAPI, OpenSSL, or Crypto++ functions, and edges encode data-flow dependencies such as a key handle passed from `CryptGenKey` to `CryptEncrypt`, or file handles flowing from `CreateFile` to `WriteFile` to `DeleteFile` [2508.00293]. MSGs are used to capture behavioral workflows including file-encryption chains and key-transfer routines.

The **Cryptographic Function Signatures** component addresses statically linked or proprietary routines that cannot be recognized through dynamic linkage. The paper describes fingerprints of about **32 bytes**, built from characteristic constants such as AES S-Boxes, instruction patterns such as `aesenc`, and control-flow skeletons of cryptographic routines. These signatures support lightweight in-memory matching during runtime, particularly around `WriteFile` [2508.00293].

The deception plan derived from MSGs emphasizes **behavioral semantics** rather than a fixed set of library calls. Two canonical file-encryption patterns are extracted. In **Pattern 1: Write-to-new-file**, the chain is `CreateFile(O) → CreateFile(D) → Encrypt(CO) → WriteFile(D) → CloseHandle(O,D) → DeleteFile(O)`, and the deception plan is to allow the chain except for `DeleteFile(O)`, which is intercepted and faked. In **Pattern 2: Overwrite-original**, the chain is `CreateFile(O) → ReadFile(O) → Encrypt(CO) → SetFilePointer(O) → WriteFile(O) → CloseHandle(O)`, and the deception plan is to drop the overwrite while returning success [2508.00293]. Because this logic is defined at the level of behavior, the paper argues that it applies across CryptoAPI, OpenSSL, Crypto++, and related implementations.

The **Reset Phase** extends the system from prevention into active deception against attacker infrastructure. It is performed in a **controlled deceptive environment** and consists of three components: a **Symbolic Binary Analyzer (SBA)** built on **gExtractor** with selective symbolic execution and support for roughly **390 APIs**; an **Actuating Agent** that reconstructs execution chains and identifies where the **“Transfer Key(s)”** routine ends; and a second **Deceptor DLL** that performs data hooking and control-flow redirection [2508.00293].

The SBA identifies the binary entry point, traces user-space API calls together with arguments, return values, call stacks, and return addresses, and then locates the final API in the key-exfiltration chain. The Actuating Agent classifies each sample into **EC1**, where “Transfer Key(s)” occurs before file encryption operations, or **EC2**, where it occurs after the encryption/deletion loop [2508.00293]. This distinction determines whether the later patch must also skip the file-encryption chain.

The reset Deceptor DLL then performs two manipulations. First, it hooks **fingerprinting** and randomness APIs for MAC address, system time, RNG, OS information, and related calls, returning **randomized or misleading values** each loop so that each iteration appears to be a new victim. Second, it inserts a **`JMP`** at the return site of the last “Transfer Key” API, redirecting execution back to the binary **entry point** while restoring stack base and limit values saved from the initial suspended launch. For **EC2**, it additionally inserts a `JMP` that skips the `FindFirstFile → ... → DeleteFile` encryption loop so that files are not repeatedly touched while keys and victim IDs continue to be sent [2508.00293].

A common misconception would be that ranDecepter fabricates cryptographic material by re-implementing the malware’s cryptography. The paper explicitly states the opposite: it **does not explicitly re-write keys**, but instead allows the ransomware to generate **legitimate keys and victim IDs based on fake system parameters**, so that the resulting records correspond to **fake victims** rather than altered cryptographic payloads [2508.00293]. This design avoids breaking the malware’s own logic and plausibly increases the chance that the backend will store the entries as normal infections.

## 4. Containment policy, deployment modes, and operational semantics

The runtime framework exposes two containment policies under **Application Restart Control (ARC)**. **Partial containment** intercepts destructive overwrites but allows encryption to new files, preserving originals while deferring deletions. **Full containment** blocks both new encryptions and deletions, at the cost that benign applications may require restarting after classification [2508.00293]. The platform also maintains **whitelisting**, where processes can be manually or automatically whitelisted after **n consecutive clean runs** with no ransomware detection; associated metadata is encrypted and protected against tampering [2508.00293].

The system is deployable in two legal and operational modes. In **split deployment**, offline analysis and the reset phase run on trusted or authorized infrastructure such as **DoD/CISA/HSI servers** or other **LEA-controlled** systems, while the real-time DLL is deployed on endpoints. In **honeypot deployment**, the entire stack runs on **LEA-managed honeypots** or sacrificial hosts [2508.00293]. This separation is important because the endpoint module is defensive and data-preserving, whereas the reset phase has an explicitly counter-offensive character.

From an implementation perspective, ranDecepter is focused on **Windows**, though the paper states that the approach is conceptually portable to POSIX calls on Linux. The research stack includes **EasyHook** for user-mode hooking, `WdFilter.sys` for process injection, a customized **Cuckoo Sandbox** for offline extraction, **gExtractor** for symbolic binary analysis, **Google Cloud Vision API** for wallpaper OCR, and **WinAppDriver** for automated interaction and response-time measurement of benign applications [2508.00293]. The prototype and artifacts are described as being available in an anonymized open repository titled “ranDecepter Artifact and Code Repository” [2508.00293].

Operationally, the design supports integration into endpoint and SOC workflows even though direct EDR/IDS/SIEM connectors are not explicitly implemented. The paper states that Deception DLL logs could be federated to a SIEM and that confirmed Stage 5 events could trigger incident-response playbooks such as host isolation, memory capture, or binary upload to analysis infrastructure [2508.00293]. This suggests a deployment model in which ranDecepter functions as both a prevention mechanism and a telemetry source.

## 5. Evaluation, empirical results, and resource-depletion effects

The principal ransomware evaluation uses **1,134 real-world samples from 30 families**, including **WannaCry, CryptoWall, Cerber, GandCrab, LockBit, Dharma, TeslaCrypt, Maze, Conti, and NotPetya**, collected from **VirusTotal** and **Any.Run** and limited to the last five years [2508.00293]. The benign corpus is described in two ways in the provided material: the abstract states evaluation using **twelve benign applications**, while the detailed evaluation section states **45 popular Windows applications**, with **12 “representative” ones emphasized**, including **7zip, Firefox, WinSCP, Opera, MySQL Workbench, VeraCrypt, and MS Word** [2508.00293]. Both figures appear in the paper materials and therefore describe different reporting granularities rather than necessarily contradictory experiments.

The ransomware detection results are broken down by stage. **204 samples** were detected at **Stage 1** through `CreateFile` with a known ransomware extension and incurred **0 file loss**. **895 samples** were detected through the combination of encryption/deletion behavior and ransom-note semantics, also with **0 file loss** because writes and deletes were faked under ARC and FakeSuccess. **16 samples** using custom or embedded cryptography were detected via high entropy and incurred an average loss of **7 files**; these are the only cases with non-zero damage. **19 samples** never reached encryption because a C2 trigger was missing, though they opened network ports, and they caused **no file loss** [2508.00293].

The aggregate result is that **98.5% (1115/1134)** of samples resulted in **0 file loss**, while **1.5% (16/1134)** lost **≤7 files**. The paper reports **100% identification accuracy** for ransomware, and the detailed evaluation reports **0 false positives across 45 benign applications** [2508.00293]. The paper attributes this outcome to a conservative final decision rule: encryption plus deletion behavior alone does not trigger definitive ransomware classification; a ransom note or wallpaper semantics are also required, and the system can afford to wait because destructive effects have already been neutralized [2508.00293].

Performance measurements were obtained by running benign workloads with and without ranDecepter and measuring task completion time via **WinAppDriver**. The reported overheads for the subset shown in Table 4b are **5.73% for 7zip**, **5.44% for Firefox**, **4.61% for WinSCP**, **3.79% for Opera**, and **3.69% for MySQL Workbench**, with other applications ranging from **0.40% to 2.99%** [2508.00293]. The paper therefore characterizes runtime impact as **minimal**, including for heavy operations such as compressing a **125MB folder**.

The resource-depletion experiments use **4 open-source ransomware projects with accessible C2 code**: **NekRos (r1)**, **Cryptonite (r2)**, **Jasmin (r3)**, and **Mauri870’s PoC ransomware (r4)** [2508.00293]. For each family, the reset phase extracted key-transfer endpoints, patched the binary, and then executed it with **50 agents** over **24 hours** while counting entries and storage on the attacker-side database. The per-family results are as follows:

| Family | Entries with 50 agents (1h / 12h / 24h) | Storage (1h / 12h / 24h) |
|---|---|---|
| r1 | 290K / 3,420K / 6,910K | ~43 / 527 / 1054 MB |
| r2 | 131K / 1,582K / 3,081K | ~15 / 189 / 385 MB |
| r3 | 413K / 4,829K / 9,223K | ~52.9 / 603 / 1189 MB |
| r4 | 195K / 2,257K / 4,357K | ~22.7 / 272 / 544 MB |

These experiments are reported to confirm that binary orchestration correctly located and looped the **“Transfer Key(s)”** logic in **100% of tested cases** and that a moderate number of agents can bloat attacker databases substantially [2508.00293]. The maximal reported figure is **9,223K entries in 24 hours** for **r3**, corresponding to about **1.1 GB** of attacker storage [2508.00293].

## 6. Relation to prior defenses, limitations, and contested implications

In the paper’s comparison against **RWGuard**, **R-Sentry**, **R-Trap**, and **R-Locker**, ranDecepter differs in four stated ways: it uses **no decoys**, performs final validation through **ransom note check plus staged behaviors**, reports **no false positives** in its own benign tests, and relies on **rule-based staging with CFS and entropy heuristics rather than ML models** [2508.00293]. The reported file-loss comparison is particularly central: ranDecepter yields **0 files lost in 98.5% of samples** and **≤7 files in the remaining 1.5%**, whereas the cited values for alternatives are much larger or partially reported [2508.00293].

This positioning supports the paper’s broader claim that the framework **decouples detection time from file damage**. Because `WriteFile` and `DeleteFile` can be neutralized before confirmation, the framework can use a later, more conservative semantic signal without sacrificing user data. A plausible implication is that this design targets a longstanding trade-off in ransomware defense: earlier triggers often reduce damage but increase false positives, whereas later triggers improve specificity but usually tolerate more damage. ranDecepter’s API-level deception is presented as a way to relax that trade-off [2508.00293].

The paper also states several technical assumptions and limitations. The system presumes **user-space API visibility**, continued use of WinAPI or recognizable crypto patterns, and the presence of a **file-based ransom note or wallpaper change**. Malware that uses kernel APIs, SSDT manipulation, or direct disk writes without passing through hooked user-mode APIs could evade detection. Advanced malware might also use **anti-hooking** or self-integrity checks such as IAT/EAT verification or inline integrity checks to detect or remove hooks. The current design focuses mainly on parent-child relationships, so **inter-process or multi-process ransomware** using side channels may be less visible [2508.00293].

The resource-depletion component has its own assumptions. It depends on attackers maintaining **persistent per-victim state** in a backend rather than operating in a trivially stateless fashion. The authors acknowledge that sophisticated adversaries might introduce sanity checks, limit infections per IP address, or move toward more stateless architectures. Even under those conditions, the paper argues that endpoint-level neutralization still remains effective and that resource depletion is an **additional offensive benefit**, not the sole defensive outcome [2508.00293].

Finally, the counter-offensive element raises legal and ethical questions. The paper explicitly asks who is authorized to manipulate attacker infrastructure and whether such activity could be construed as **“hacking back.”** It therefore proposes **LEA or government cybersecurity organizations**, including **DoD/CISA/HSI**, as the primary operators of the reset phase [2508.00293]. Another stated concern is that polluting attacker infrastructure might interfere with key retrieval for victims who have already paid. The architecture’s separation between the endpoint defense module and the reset infrastructure is intended to make it easier to confine offensive operations to legally authorized environments [2508.00293].

Taken together, ranDecepter defines a ransomware-defense paradigm centered on **API-centric prevention**, **semantic confirmation**, and **controlled binary reuse**. Within the scope claimed by the paper, it is a system for both preserving endpoint data integrity and exploiting ransomware’s dependence on per-victim key management as a deception surface [2508.00293].

Source: https://www.emergentmind.com/topics/randecepter