REAL-IoT: Real-World IoT Evaluation

• REAL-IoT is a comprehensive framework defining realistic IoT security by unifying heterogeneous NetFlow datasets into a common benchmark for GNN-based intrusion detection.
• It employs adaptive stratified sampling, schema standardization, and physical testbed data to simulate dynamic network conditions and practical adversarial scenarios.
• Empirical results reveal that varying GNN architectures show distinct robustness gaps under distribution drift and adversarial perturbations, stressing the need for realistic evaluation.

REAL-IoT denotes, in the strictest current usage, a comprehensive framework for evaluating the robustness of graph neural network-based network intrusion detection systems in Internet of Things environments under distribution drift and practical adversarial attack (Zhan et al., 14 Jul 2025). In a broader and increasingly common research sense, the term also aligns with a methodological shift away from stationary benchmarks and laboratory-only assumptions toward heterogeneous deployments, evolving traffic, realistic attacker behavior, real-time system constraints, and deployment-oriented evaluation. This broader usage is visible in work on concept-drift-resilient IoT botnet detection, honeypot ecosystems for real attacker observation, real-world device identification, real-time stream processing benchmarks, and operational spatial IoT infrastructures (Wasswa et al., 27 Dec 2025).

1. Definition, scope, and terminological usage

REAL-IoT is not a single universal dataset, nor is it a single model family. The named framework in "REAL-IoT: Characterizing GNN Intrusion Detection Robustness under Practical Adversarial Attack" is explicitly a measurement and evaluation pipeline rather than a new GNN architecture. Its purpose is to standardize multiple canonical NetFlow intrusion datasets, construct a unified graph benchmark, add a physical IoT intrusion dataset collected from a testbed, evaluate representative GNN-based NIDS under drift and adversarial attack, and study whether LLMs can mitigate node-injection effects by filtering suspicious graph elements (Zhan et al., 14 Jul 2025).

The phrase also functions more loosely across adjacent literature. "Toward Real-World IoT Security" does not define a dataset literally named REAL-IoT, but it explicitly argues that stationary datasets such as Bot-IoT, N-BaIoT, and CICIoT2022 are unrealistic for dynamic IoT environments, and it evaluates cross-network concept drift using heterogeneous home- and enterprise-like traffic. This suggests that REAL-IoT has become an umbrella descriptor for evaluation settings in which device populations change, attack scripts evolve, and network conditions drift over time rather than remaining fixed (Wasswa et al., 27 Dec 2025).

A related misconception is that REAL-IoT refers only to cybersecurity. In practice, adjacent work uses closely related formulations such as "real-world IoT device identification," "real-time IoT benchmark," and "real-time IoT data" to describe open-world vendor inference over 2,015 vendors, streaming benchmarks with 27 IoT tasks and 4 composite applications, and agentic search over live IoT databases. The common denominator is operational realism rather than a single task definition (Mahmood et al., 24 Sep 2025).

2. Unified robustness-evaluation framework

The core REAL-IoT framework begins from the claim that GNN-based NIDS are usually evaluated on one dataset under the implicit assumption that the data distribution is stationary, and that adversarial robustness is often assessed with perturbations that do not respect protocol semantics or realistic graph structure. To close that measurement gap, the framework selects three NetFlow v2 datasets—NF-BoT-IoT v2, NF-CSE-CIC-IDS2018 v2, and NF-UNSW-NB15 v2—and transforms them into a unified benchmark through adaptive stratified sampling, schema standardization, label harmonization, and graph construction (Zhan et al., 14 Jul 2025).

Adaptive stratified sampling is used because the source datasets are large and strongly imbalanced. For each class $c$, the sampling rate $r_c$ depends on whether the class is rare, uncommon, or common, with full retention when $N_c < N_{\min}$. In the notation given for the framework,

$r_c = \begin{cases} \max(r_{\text{high}}, r_{\text{base}} \cdot M_{\text{rare}}) & \text{if } N_c < \theta_{\text{rare}},\[4pt] \min(1.0, r_{\text{base}} \cdot M_{\text{uncommon}}) & \text{if } \theta_{\text{rare}} \leq N_c < \theta_{\text{uncommon}},\[4pt] r_{\text{base}} \cdot M_{\text{common}} & \text{if } N_c \geq \theta_{\text{uncommon}}. \end{cases}$

This preserves rare attack samples while reducing overall scale, which is especially important for realistic IoT attack distributions (Zhan et al., 14 Jul 2025).

After sampling, all datasets are mapped into a common NetFlow schema that includes IPV4_SRC_ADDR, IPV4_DST_ADDR, L4_SRC_PORT, L4_DST_PORT, PROTOCOL, L7_PROTO, IN_BYTES, OUT_BYTES, IN_PKTS, OUT_PKTS, FLOW_DURATION_MILLISECONDS, TCP_FLAGS, binary Attack, categorical Label, flow_id, and dataset_source. Numeric features are standardized with Z-score scaling using training statistics only:

$$\mathbf{x}' = \frac{\mathbf{x} - \boldsymbol{\mu}_{\text{train}}}{\boldsymbol{\sigma}_{\text{train}}}.$$

Attack labels from the original datasets are harmonized into a unified taxonomy, with unmatched classes assigned to "Other" (Zhan et al., 14 Jul 2025).

The graph abstraction is deliberately simple and uniform. Each unique IP address becomes a node, each NetFlow record becomes a directed edge from source IP to destination IP, and standardized NetFlow fields become edge features. The result is an IP-centric communication graph that can be fed into all evaluated models under the same preprocessing regime. This design matters because it removes much of the dataset-specific tailoring that otherwise makes published GNN results difficult to compare (Zhan et al., 14 Jul 2025).

3. Physical testbed, threat model, and adversarial methodology

REAL-IoT extends beyond canonical datasets by building a physical IoT testbed. The hardware consists of 5 Raspberry Pi 4B units and 1 MacBook Pro. Four Raspberry Pis run Raspberry Pi OS as IoT device simulators, one Raspberry Pi runs Kali Linux 2024.4 as the attacker, and the MacBook Pro acts as Wi-Fi access point, DHCP server, and capture point on the bridge100 interface. Simulated IoT devices use Node-RED 4.0.9 to generate HTTP traffic and MQTT-like traffic on port 8883 to an AWS IoT Greengrass broker. A real Amazon Echo Dot was initially attempted, but its traffic was instead recorded and reproduced in Node-RED because of OS and performance constraints (Zhan et al., 14 Jul 2025).

The captured attack scenarios are intentionally practical rather than purely gradient-based abstractions. Benign phases generate only HTTP and MQTT traffic. Attack B is an IP spoofing DoS using hping3 --rand-source against one target, with randomized source IPs and incrementing source ports beginning at 60002. Attack C is an nmap -sS -T4 -p [range] -g 60003 TCP SYN scan against another target. Attack D is a combined multi-vector DoS using concurrent TCP SYN, UDP, and ICMP flooding with source ports beginning at 60004. Additional day-3 phases include UDP and ICMP DoS. All packets are captured with Wireshark, converted to bidirectional flows with CICFlowMeter, and labeled using timestamps, target IPs, target ports, base source ports, and flag patterns, yielding 4,161 NetFlow records (Zhan et al., 14 Jul 2025).

The framework also applies three adversarial attack families to the standardized graphs. PGD is a white-box feature attack with objective

$$\max_{\delta} \mathcal{L}(f(x+\delta), y) \quad \text{s.t.} \quad \|\delta\|_{\infty} \le \epsilon,$$

with model-specific $\epsilon$ ranges. EdgeRemove is a black-box structural perturbation that randomly removes a fraction of graph edges. NodeInject adds new nodes and edges, modeling spoofed sources, bot insertion, or flooding behavior. The framework explicitly treats PGD as a useful upper bound but less realistic operationally, whereas node injection is closer to natural IoT attack surfaces because IP spoofing and distributed flooding literally alter graph structure (Zhan et al., 14 Jul 2025).

A further methodological component is LLM-based filtering. Here an LLM receives a system prompt describing the task of analyzing one node with its raw traffic and local neighborhood, and outputs a confidence score in $[0,1]$ together with a short justification. Nodes flagged as victims or anomalous are pruned along with incident edges, producing an "LLM-fixed" graph on which the downstream GNN is reevaluated. The LLM therefore acts as an expert pre-filter rather than as the primary classifier (Zhan et al., 14 Jul 2025).

4. Empirical results and what REAL-IoT measures

REAL-IoT’s central empirical result is that robustness and generalization are frequently overestimated when GNN-based NIDS are evaluated only on their home datasets. On the standardized and unified benchmarks, the three representative models—Anomal-E, E-GraphSAGE, and CAGN-GAT Fusion—display sharply different behavior. Anomal-E achieves $F1 = 0.641$ on standardized NF-BoT-IoT v2 but $F1 = 0.977$ and accuracy $r_c$0 on the unified dataset; E-GraphSAGE reaches accuracy $r_c$1 and $r_c$2 on standardized NF-BoT-IoT v2 but only accuracy $r_c$3 and $r_c$4 on standardized NF-CICIDS2018 v2, and accuracy $r_c$5 with $r_c$6 on the unified dataset; CAGN-GAT Fusion obtains accuracy $r_c$7 and $r_c$8 on standardized NF-BoT-IoT v2 but collapses to accuracy $r_c$9 and $N_c < N_{\min}$0 on standardized NF-UNSW-NB15 v2, recovering only partially on the unified dataset with accuracy $N_c < N_{\min}$1 and $N_c < N_{\min}$2 (Zhan et al., 14 Jul 2025).

These results are significant because they show that heterogeneity does not affect all architectures in the same way. The self-supervised Anomal-E can benefit from more diverse flows, while mean-aggregation or attention-based supervised architectures may become miscalibrated across sources. REAL-IoT therefore measures not only raw classification quality but also architecture-specific susceptibility to distributional mismatch (Zhan et al., 14 Jul 2025).

Under synthetic adversarial attack, PGD is consistently the most damaging perturbation family. For Anomal-E, PGD with $N_c < N_{\min}$3 reduces accuracy to approximately $N_c < N_{\min}$4. For E-GraphSAGE, PGD with $N_c < N_{\min}$5 up to $N_c < N_{\min}$6 reduces accuracy from $N_c < N_{\min}$7 to $N_c < N_{\min}$8 and $N_c < N_{\min}$9 from $r_c = \begin{cases} \max(r_{\text{high}}, r_{\text{base}} \cdot M_{\text{rare}}) & \text{if } N_c < \theta_{\text{rare}},\[4pt] \min(1.0, r_{\text{base}} \cdot M_{\text{uncommon}}) & \text{if } \theta_{\text{rare}} \leq N_c < \theta_{\text{uncommon}},\[4pt] r_{\text{base}} \cdot M_{\text{common}} & \text{if } N_c \geq \theta_{\text{uncommon}}. \end{cases}$0 to $r_c = \begin{cases} \max(r_{\text{high}}, r_{\text{base}} \cdot M_{\text{rare}}) & \text{if } N_c < \theta_{\text{rare}},\[4pt] \min(1.0, r_{\text{base}} \cdot M_{\text{uncommon}}) & \text{if } \theta_{\text{rare}} \leq N_c < \theta_{\text{uncommon}},\[4pt] r_{\text{base}} \cdot M_{\text{common}} & \text{if } N_c \geq \theta_{\text{uncommon}}. \end{cases}$1. For CAGN-GAT Fusion, PGD with $r_c = \begin{cases} \max(r_{\text{high}}, r_{\text{base}} \cdot M_{\text{rare}}) & \text{if } N_c < \theta_{\text{rare}},\[4pt] \min(1.0, r_{\text{base}} \cdot M_{\text{uncommon}}) & \text{if } \theta_{\text{rare}} \leq N_c < \theta_{\text{uncommon}},\[4pt] r_{\text{base}} \cdot M_{\text{common}} & \text{if } N_c \geq \theta_{\text{uncommon}}. \end{cases}$2 up to $r_c = \begin{cases} \max(r_{\text{high}}, r_{\text{base}} \cdot M_{\text{rare}}) & \text{if } N_c < \theta_{\text{rare}},\[4pt] \min(1.0, r_{\text{base}} \cdot M_{\text{uncommon}}) & \text{if } \theta_{\text{rare}} \leq N_c < \theta_{\text{uncommon}},\[4pt] r_{\text{base}} \cdot M_{\text{common}} & \text{if } N_c \geq \theta_{\text{uncommon}}. \end{cases}$3 reduces accuracy from $r_c = \begin{cases} \max(r_{\text{high}}, r_{\text{base}} \cdot M_{\text{rare}}) & \text{if } N_c < \theta_{\text{rare}},\[4pt] \min(1.0, r_{\text{base}} \cdot M_{\text{uncommon}}) & \text{if } \theta_{\text{rare}} \leq N_c < \theta_{\text{uncommon}},\[4pt] r_{\text{base}} \cdot M_{\text{common}} & \text{if } N_c \geq \theta_{\text{uncommon}}. \end{cases}$4 to $r_c = \begin{cases} \max(r_{\text{high}}, r_{\text{base}} \cdot M_{\text{rare}}) & \text{if } N_c < \theta_{\text{rare}},\[4pt] \min(1.0, r_{\text{base}} \cdot M_{\text{uncommon}}) & \text{if } \theta_{\text{rare}} \leq N_c < \theta_{\text{uncommon}},\[4pt] r_{\text{base}} \cdot M_{\text{common}} & \text{if } N_c \geq \theta_{\text{uncommon}}. \end{cases}$5, $r_c = \begin{cases} \max(r_{\text{high}}, r_{\text{base}} \cdot M_{\text{rare}}) & \text{if } N_c < \theta_{\text{rare}},\[4pt] \min(1.0, r_{\text{base}} \cdot M_{\text{uncommon}}) & \text{if } \theta_{\text{rare}} \leq N_c < \theta_{\text{uncommon}},\[4pt] r_{\text{base}} \cdot M_{\text{common}} & \text{if } N_c \geq \theta_{\text{uncommon}}. \end{cases}$6 from $r_c = \begin{cases} \max(r_{\text{high}}, r_{\text{base}} \cdot M_{\text{rare}}) & \text{if } N_c < \theta_{\text{rare}},\[4pt] \min(1.0, r_{\text{base}} \cdot M_{\text{uncommon}}) & \text{if } \theta_{\text{rare}} \leq N_c < \theta_{\text{uncommon}},\[4pt] r_{\text{base}} \cdot M_{\text{common}} & \text{if } N_c \geq \theta_{\text{uncommon}}. \end{cases}$7 to $r_c = \begin{cases} \max(r_{\text{high}}, r_{\text{base}} \cdot M_{\text{rare}}) & \text{if } N_c < \theta_{\text{rare}},\[4pt] \min(1.0, r_{\text{base}} \cdot M_{\text{uncommon}}) & \text{if } \theta_{\text{rare}} \leq N_c < \theta_{\text{uncommon}},\[4pt] r_{\text{base}} \cdot M_{\text{common}} & \text{if } N_c \geq \theta_{\text{uncommon}}. \end{cases}$8, and AUC from $r_c = \begin{cases} \max(r_{\text{high}}, r_{\text{base}} \cdot M_{\text{rare}}) & \text{if } N_c < \theta_{\text{rare}},\[4pt] \min(1.0, r_{\text{base}} \cdot M_{\text{uncommon}}) & \text{if } \theta_{\text{rare}} \leq N_c < \theta_{\text{uncommon}},\[4pt] r_{\text{base}} \cdot M_{\text{common}} & \text{if } N_c \geq \theta_{\text{uncommon}}. \end{cases}$9 to $$\mathbf{x}' = \frac{\mathbf{x} - \boldsymbol{\mu}_{\text{train}}}{\boldsymbol{\sigma}_{\text{train}}}.$$0. By contrast, even 20–30% random edge removal leaves metrics largely unchanged, whereas node injection causes steady and more realistic degradation, including $$\mathbf{x}' = \frac{\mathbf{x} - \boldsymbol{\mu}_{\text{train}}}{\boldsymbol{\sigma}_{\text{train}}}.$$1 decline from $$\mathbf{x}' = \frac{\mathbf{x} - \boldsymbol{\mu}_{\text{train}}}{\boldsymbol{\sigma}_{\text{train}}}.$$2 to $$\mathbf{x}' = \frac{\mathbf{x} - \boldsymbol{\mu}_{\text{train}}}{\boldsymbol{\sigma}_{\text{train}}}.$$3 for E-GraphSAGE and from $$\mathbf{x}' = \frac{\mathbf{x} - \boldsymbol{\mu}_{\text{train}}}{\boldsymbol{\sigma}_{\text{train}}}.$$4 to $$\mathbf{x}' = \frac{\mathbf{x} - \boldsymbol{\mu}_{\text{train}}}{\boldsymbol{\sigma}_{\text{train}}}.$$5 for CAGN-GAT Fusion as injection ratio increases from 5% to 20% (Zhan et al., 14 Jul 2025).

On the physical IoT testbed, E-GraphSAGE achieves AUC $$\mathbf{x}' = \frac{\mathbf{x} - \boldsymbol{\mu}_{\text{train}}}{\boldsymbol{\sigma}_{\text{train}}}.$$6, accuracy $$\mathbf{x}' = \frac{\mathbf{x} - \boldsymbol{\mu}_{\text{train}}}{\boldsymbol{\sigma}_{\text{train}}}.$$7, $$\mathbf{x}' = \frac{\mathbf{x} - \boldsymbol{\mu}_{\text{train}}}{\boldsymbol{\sigma}_{\text{train}}}.$$8, precision $$\mathbf{x}' = \frac{\mathbf{x} - \boldsymbol{\mu}_{\text{train}}}{\boldsymbol{\sigma}_{\text{train}}}.$$9, and recall $$\max_{\delta} \mathcal{L}(f(x+\delta), y) \quad \text{s.t.} \quad \|\delta\|_{\infty} \le \epsilon,$$0, while CAGN-GAT Fusion achieves AUC $$\max_{\delta} \mathcal{L}(f(x+\delta), y) \quad \text{s.t.} \quad \|\delta\|_{\infty} \le \epsilon,$$1, accuracy $$\max_{\delta} \mathcal{L}(f(x+\delta), y) \quad \text{s.t.} \quad \|\delta\|_{\infty} \le \epsilon,$$2, $$\max_{\delta} \mathcal{L}(f(x+\delta), y) \quad \text{s.t.} \quad \|\delta\|_{\infty} \le \epsilon,$$3, precision $$\max_{\delta} \mathcal{L}(f(x+\delta), y) \quad \text{s.t.} \quad \|\delta\|_{\infty} \le \epsilon,$$4, and recall $$\max_{\delta} \mathcal{L}(f(x+\delta), y) \quad \text{s.t.} \quad \|\delta\|_{\infty} \le \epsilon,$$5. These figures remain strong, but they are lower than many single-dataset reports, which is precisely the point of the framework’s realism claim (Zhan et al., 14 Jul 2025).

LLM-based filtering shows mixed but nontrivial promise. On a 100-node graph with 20 injected nodes, LLaMA 4 Maverick 17B improves downstream CAGN-GAT performance to accuracy $$\max_{\delta} \mathcal{L}(f(x+\delta), y) \quad \text{s.t.} \quad \|\delta\|_{\infty} \le \epsilon,$$6 and $$\max_{\delta} \mathcal{L}(f(x+\delta), y) \quad \text{s.t.} \quad \|\delta\|_{\infty} \le \epsilon,$$7, while GPT-4o reaches accuracy $$\max_{\delta} \mathcal{L}(f(x+\delta), y) \quad \text{s.t.} \quad \|\delta\|_{\infty} \le \epsilon,$$8 and $$\max_{\delta} \mathcal{L}(f(x+\delta), y) \quad \text{s.t.} \quad \|\delta\|_{\infty} \le \epsilon,$$9. On a 1000-node graph with 200 injected nodes, LLaMA 4 reaches accuracy $\epsilon$0 and $\epsilon$1, and GPT-4o reaches accuracy $\epsilon$2 and $\epsilon$3. Poorer LLM filtering can be harmful because high recall with many incorrect flags removes benign nodes and damages graph integrity. REAL-IoT therefore treats LLM mitigation as promising but not yet stable (Zhan et al., 14 Jul 2025).

5. Related REAL-IoT directions beyond GNN intrusion detection

A major adjacent direction is concept-drift-resilient IoT botnet detection. "Toward Real-World IoT Security" argues that existing models depend on stationary datasets that fail to reflect the dynamic nature of real-world IoT NetFlow traffic, then proposes a pipeline with a historical VAE, a second VAE for current traffic, an MLP alignment model minimizing Wasserstein distance between latent distributions, and a fixed GAT classifier over $\epsilon$4-nearest-neighbor graphs with $\epsilon$5 and Euclidean distance. In its cross-network setting, training on ACI-IoT-2023 and testing on IoT-NID yields accuracy $\epsilon$6, precision $\epsilon$7, recall $\epsilon$8, and $\epsilon$9; after latent-space alignment, all four metrics rise to approximately $[0,1]$0. This establishes concept drift as a central REAL-IoT concern and suggests a deployment model in which the classifier is trained once and adaptation occurs through representation alignment rather than continuous retraining (Wasswa et al., 27 Dec 2025).

Another direction emphasizes direct observation of real attacker behavior. "A First Step Towards Understanding Real-world Attacks on IoT Devices" develops a multi-phased honeypot ecosystem with HoneyShell, HoneyWindowsBox, and HoneyCamera, deployed on-premise and in AWS and Azure across France, the UK, India, Australia, Canada, and Singapore. Over roughly one year it records 18,972,475 total hits, including 17,343,412 HoneyShell hits, 1,618,906 HoneyWindowsBox hits, and 10,157 HoneyCamera hits. HoneyShell shows a successful login rate of about $[0,1]$1, but only $[0,1]$2 of successful sessions execute at least one command; SSH banner analysis attributes $[0,1]$3 of attacks to bots and $[0,1]$4 to likely human-driven activity. HoneyCamera captures CVE-targeted attacks against D-Link, Hikvision, Netwave, AIVI Tech, Foscam, and Shellshock-style CGI interfaces. This line of work grounds REAL-IoT in attacker-facing measurement rather than only offline classification (Tabari et al., 2020).

A third direction concerns open-world device identity. "LLMs for Real-World IoT Device Identification" reframes vendor inference as a language modeling problem over passive network metadata from the IoT Inspector corpus. After preprocessing 2.68M flow-level rows into 216K canonical device signatures and generating pseudo-labels over 2,015 vendors, an instruction-tuned quantized Meta-Llama-3.1-8B-Instruct model reaches $[0,1]$5 top-1 accuracy and $[0,1]$6 macro accuracy on the full 216K-device phase, with $[0,1]$7 accuracy on tail vendors with $[0,1]$8 samples. The same study reports resilience to missing fields, VPN obfuscation, protocol drift, spoofed user labels, spoofed DHCP hostnames, and misleading hostname tokens. This extends REAL-IoT beyond NIDS toward open-world asset visibility under incomplete, noisy, and adversarial metadata (Mahmood et al., 24 Sep 2025).

The real-time systems literature broadens the term further. RIoTBench defines a "Realtime IoT Benchmark" with 27 reusable tasks, 4 composite applications—ETL, STATS, TRAIN, and PRED—and 4 workloads with peak rates from 500 to 10,000 messages/sec, showing that ETL and PRED can operate at tens-of-milliseconds end-to-end latency while STATS and TRAIN occupy seconds-to-minutes regimes (Shukla et al., 2017). Real-time geoinformation systems for agri-environment research document 3 major and 14 minor system versions, 2,727 devices manufactured, and active or planned deployment across four continents, emphasizing the transition from TRL 4–5 prototypes to TRL 7–8 operational fleets with PostGIS-backed QA/QC, local buffering, solar power management, and Grafana-based monitoring (Runck et al., 2024). "Agentic Search Engine for Real-Time IoT Data" adds an application-layer perspective: a SensorsConnect-based architecture with a unified data model, Sentence-BERT service embeddings, HNSW vector search, MongoDB geospatial queries, and LangGraph orchestration achieves $[0,1]$9 top-1 accuracy in retrieving intent-based services in a Toronto scenario, with $F1 = 0.641$0 of responses below 2.10 s and $F1 = 0.641$1 below 4.12 s (Elewah et al., 15 Mar 2025).

Security and control-plane work also maps naturally onto the same agenda. A zero-trust real-time lightweight access-control protocol for mobile cloud-based IoT sensors combines ECDH, symmetric encryption, hash chains, Merkle-tree-based trust scores, KP-ABE, and IBBE so that sensors perform only lightweight cryptography while the cloud and coordinators handle policy and trust evaluation. Separately, blockchain reviews aimed at real-time IoT frame the design space around end-to-end latency $F1 = 0.641$2, throughput $F1 = 0.641$3, reliability $F1 = 0.641$4, and energy $F1 = 0.641$5, and argue that edge/fog integration, Layer-2 mechanisms, sharding, and hybrid consensus are needed because conventional blockchains do not meet millisecond-to-sub-second control requirements (Mohseni-Ejiyeh, 2023).

6. Limitations, controversies, and future directions

The most important limitation of REAL-IoT as a named framework is scale and coverage. Its physical testbed dataset contains 4,161 NetFlow records and attack scenarios that are mainly DoS and scan variants. Many published GNN-NIDS are not included because code is missing, outdated, or heavily tied to proprietary preprocessing, and LLM mitigation experiments are conducted on sampled subgraphs because API cost and latency are substantial. The framework is therefore best understood as a rigorous measurement step rather than a completed universal benchmark (Zhan et al., 14 Jul 2025).

A second limitation concerns standardization versus novelty. REAL-IoT’s unified dataset is designed to expose generalization gaps, but that same standardization can suppress dataset-specific semantics that some architectures implicitly exploit. This is not a flaw so much as a deliberate controversy in measurement philosophy: whether a benchmark should reward adaptation to one environment or robustness across many. The framework sides explicitly with the latter (Zhan et al., 14 Jul 2025).

Broader REAL-IoT research inherits additional unresolved issues. The latent-alignment botnet framework does not yet include integrated drift detection and still depends on representative current traffic for training the second VAE and the alignment model (Wasswa et al., 27 Dec 2025). LLM-based vendor identification reports that explanations are not guaranteed to be causally faithful and can hallucinate false business relationships under sparse metadata (Mahmood et al., 24 Sep 2025). IoT-ASE does not provide a detailed security or privacy architecture for public exposure of IoT data, even though it proposes a WWW-like substrate for real-time search (Elewah et al., 15 Mar 2025). The zero-trust access-control protocol identifies lightweight user-attribute revocation as future work (Mohseni-Ejiyeh, 2023). Blockchain-oriented real-time IoT reviews continue to emphasize the unresolved scalability-security-decentralization trade-off, along with interoperability, privacy preservation, and sustainable resource management (Rhidoy et al., 29 Mar 2026).

What emerges from these threads is a consistent research agenda rather than a settled standard. REAL-IoT, as used today, marks a transition from static, closed-world, single-dataset evaluation toward IoT systems that must withstand heterogeneous device populations, protocol evolution, cross-site distribution drift, realistic attacker behavior, operational latency budgets, and deployment constraints. The named framework for GNN robustness evaluation is one concrete instantiation of that transition; the surrounding literature suggests that the term’s long-term significance will depend on whether such realism can be made reproducible, scalable, and comparable across security, systems, networking, and application layers.