EdgeBench

• EdgeBench is a series of benchmark suites evaluating edge computing platforms, workflows, and multi-tenant resource interference at Edge
• Developed in various years, including 2018, 2020 and 2025, each with distinct foci, such as comparing AWS Greengrass and Azure IoT Edge.
• EdgeBench has applications in domains such as video analytics, IoT hubs, and real-world agent learning, with varied metrics also including latency, resource utilization, and cost.

EdgeBench is a name used for several distinct benchmark suites and frameworks for evaluating edge computing, embedded artificial intelligence, multi-tenant edge nodes, and, more recently, long-horizon agent learning. The most prominent uses are EdgeBench: Benchmarking Edge Computing Platforms, a 2018 benchmark comparing AWS Greengrass, Azure IoT Edge, and cloud-only pipelines (Das et al., 2018); EdgeBench: A Workflow-based Benchmark for Edge Computing, a 2020 framework for configurable IoT–edge–cloud workflows (Yang et al., 2020); and a 2025 open-source framework for automated multi-tenancy evaluation on edge compute nodes (Georgiou et al., 12 Jun 2025). The same name is also used by a 2026 benchmark of agent learning from real-world environments (Zhu et al., 6 Jul 2026). These projects are related by their concern with edge-system behavior but are not a single continuously developed benchmark.

1. Terminology and historical development

The first EdgeBench usage addresses managed commercial edge platforms. It compares Amazon AWS Greengrass, which executes locally deployed Lambda functions, with Microsoft Azure IoT Edge, which executes Docker-compatible containers called edge modules. The benchmark also compares each platform with its provider’s cloud-only implementation (Das et al., 2018). Its purpose is to measure latency, computation time, bandwidth, CPU and memory use, and approximate cost under common workloads.

A second EdgeBench, introduced in 2020, broadens the unit of evaluation from a provider platform to a configurable application workflow. It allows users to define workflow logic, storage backends, and the distribution of workflow stages among IoT devices, edge servers, and cloud resources (Yang et al., 2020). The framework uses OpenFaaS functions and persistent storage services to measure workflow-level and function-level behavior.

A third use of the name denotes an automated multi-tenancy framework. This EdgeBench evaluates performance interference among co-located workloads on real edge nodes, using containerized streaming analytics, databases, machine-learning inference, and component stressors (Georgiou et al., 12 Jun 2025). It emphasizes resource contention, workload antagonism, power, temperature, cold-start behavior, and performance degradation.

The name has also been applied to DeepEdgeBench, which evaluates neural-network inference on embedded SoCs and microcontrollers (Baller et al., 2021), and to a 2026 suite of 134 long-horizon tasks measuring how agents improve through sustained interaction with real-world environments (Zhu et al., 6 Jul 2026). The latter is conceptually unrelated to edge-computing infrastructure despite sharing the name.

2. EdgeBench for commercial edge platforms

The 2018 EdgeBench suite contains three applications:

1. Audio/speech-to-text processing, using PocketSphinx and 104 Tatoeba Corpus audio samples converted to 16-kHz, 16-bit mono WAV.
2. Image recognition, using OpenCV resizing, MXNet, and pretrained SqueezeNet on 500 ILSVRC2012/ImageNet images resized to $224\times224\times3$.
3. Scalar sensor emulation, which generates random scalar values, groups them over a configurable interval, and sends the group to cloud storage.

Each workload processes data locally on an edge device and sends its output through the relevant IoT service to cloud storage. The suite deliberately combines computationally meaningful workloads with an almost computation-free scalar pipeline. This separates local-computation benefits from messaging, storage, and platform overhead.

The measured architecture contains an edge device, local application execution, cloud IoT ingestion, cloud routing, and cloud storage. AWS uses Greengrass Core and Lambda functions, AWS IoT services, Rules, and S3. Azure uses IoT Edge modules, edgeAgent, edgeHub, Azure IoT Hub, and Blob Storage.

AWS Greengrass functions were configured with 256 MB RAM and long-lived execution. Greengrass supports Python 2.7, Node.js 6.10, Java 8, C, C++, and other languages capable of importing C libraries. External dependencies must be compiled for the target environment and packaged into deployment archives.

Azure IoT Edge uses Docker-compatible containers. Its container model supports arbitrary user code, Azure Functions, Streaming Analytics components, and Azure Machine Learning models. The authors found containerized dependency management cleaner for applications with substantial native libraries. Azure also supports MQTT and AMQP, whereas the reported AWS communication path primarily uses MQTT.

A major platform difference concerns cloud storage semantics. AWS can create one S3 object per incoming message. Azure Blob Storage batches messages before writing a blob, with a minimum reported time window of 60 seconds or chunk size of 10 MB. Under a 60-second batching interval, the average time in Azure IoT Hub was approximately 90 seconds; under a 90-second interval, it was approximately 93–94 seconds. This batching substantially affected end-to-end latency.

3. Workloads, metrics, and experimental results

The benchmark records three timestamps:

• $T_1$: inserted immediately before the edge application sends a result;
• $T_2$: inserted when the message is enqueued in the cloud IoT Hub;
• $T_3$: the creation timestamp of the final cloud-storage blob.

Compute time is measured locally. Time in flight is:

$T_2-T_1.$

End-to-end latency is defined as:

$C_{\mathrm{edge}}+(T_3-T_1).$

The Raspberry Pi 3B was used as the edge device. AWS and Azure cloud regions were both located in US East, Virginia. Cloud-only AWS Lambda used 3008 MB, while Azure used the Consumption Host Plan. The edge platform was connected through a wireless router operating in the 2.4-GHz spectrum.

The explicitly reported AWS and Greengrass average end-to-end latencies are:

Workload	AWS cloud-only	AWS Greengrass
Audio	1.79 s	5.36 s
Image	0.87 s	1.10 s
Scalar	0.936 s	0.66 s

The audio workload favored AWS cloud-only because the cloud used substantially more powerful compute resources. Greengrass required approximately 4.77 seconds of local audio computation, compared with an average audio duration of approximately 2.4 seconds, making real-time processing questionable on the tested Raspberry Pi.

Image recognition was a stronger edge case. Greengrass latency was 1.10 seconds, compared with 0.87 seconds for AWS cloud-only, while transmitting substantially less data. For 500 images, AWS edge traffic was 0.90 MB compared with 73.10 MB for cloud-only. In the traffic-camera estimate, the edge image pipeline cost approximately $1.5584 per month compared with $8.027 per month for AWS cloud-only, while monthly network traffic was approximately 253.125 MB for edge execution and 35.38 GB for cloud-only execution (Das et al., 2018).

The results show that edge execution does not universally reduce latency. Its principal advantages arise when raw inputs are large, outputs are small, bandwidth is costly or constrained, and local computation is feasible. Cloud execution remains advantageous when the edge device is too weak for the workload or when the input volume is small.

4. Workflow-based EdgeBench

The 2020 workflow-based EdgeBench treats an edge application as a configurable graph of functions and storage operations (Yang et al., 2020). Users can customize workflow logic, input and output storage, successor functions, synchronous or asynchronous invocation, periodic execution, data sizes, concurrency, and stage placement.

The principal configuration mechanism is a per-function YAML template containing fields such as:

• input;
• output;
• next_function;
• next_tier;
• sync;
• cron.

The framework wraps user-defined handlers with logic that loads input, invokes the handler, stores output, and invokes a successor through a REST API when appropriate. The conceptual execution sequence is:

$$\text{load input}\rightarrow\text{invoke handler}\rightarrow\text{store output}\rightarrow\text{invoke successor}.$$

Four workflow patterns are predefined:

1. Pipeline: sequential execution of $f_1\rightarrow f_2\rightarrow\cdots\rightarrow f_n$.
2. One-to-many: one function invokes multiple successors.
3. Branching: an output condition selects one successor.
4. Cron workflow: a function or workflow executes periodically.

The prototype uses OpenFaaS for function execution. Storage is treated as a first-class design variable and is accessed through load() and store() operations. Supported backends include Minio, Amazon S3, and Apache Kafka. The representative workflows additionally use InfluxDB, Amazon Timestream, and VerneMQ/MQTT.

Video analytics workflow

The video workflow consists of:

$$\text{video generator}\rightarrow\text{motion detection}\rightarrow\text{face detection}\rightarrow\text{face recognition}.$$

FFmpeg generates video chunks, OpenCV performs motion detection, SSD performs face detection, and a CNN plus k-nearest neighbors performs face recognition. The first two stages process every video chunk and generate much of the I/O, while only approximately 18% of frames reach face recognition.

At 50 concurrent streams, CPU utilization reaches 99.7%. Face recognition has the highest per-invocation latency. The evaluated placements include IoT-plus-edge, IoT-plus-cloud, and a three-tier configuration. Cloud execution provides lower individual stage latency, but communication can dominate end-to-end performance. The three-tier placement, with earlier processing on IoT and edge resources and later recognition in the cloud, achieves the best reported end-to-end latency.

IoT hub workflow

The IoT hub represents large numbers of devices emitting records containing location, temperature, moisture, power consumption, and health status. It includes:

1. sensor-data generation;
2. LSTM training;
3. LSTM prediction;
4. database querying.

Sensor data is delivered through MQTT and stored in InfluxDB at IoT and edge tiers or Amazon Timestream in the cloud. LSTM models and predictions use Minio or S3.

LSTM training is the most CPU-intensive job, queries consume the most memory, and sensor generation produces the most I/O. When training is enabled, average CPU usage rises by 54% and memory usage by 69%. Prediction latency increases from 1.57 seconds to 2.63 seconds, a 67% increase, when training runs concurrently. The results support placing computationally intensive training in the cloud while retaining latency-sensitive or data-local operations nearer to the source.

The testbed consists of Raspberry Pi 4 IoT devices, ASU edge servers, and AWS EKS cloud workers. The benchmark uses Kubernetes, OpenFaaS, Prometheus, Minio, and InfluxDB. Each experiment is run five times, although the paper reports representative runs rather than complete confidence intervals.

5. Multi-tenancy and resource interference

The 2025 EdgeBench framework automates the evaluation of multi-tenancy on real edge compute nodes (Georgiou et al., 12 Jun 2025). Its objective is to characterize performance interference when multiple containerized workloads share CPU, memory, disk, network, power, and thermal resources.

Experiments are specified through YAML. A description identifies the output record, repetition count, duration, concurrent workloads, target nodes, workload-specific parameters, startup delays, idle intervals, and container orchestrator. The framework bootstraps nodes through SSH, installs Docker and monitoring components, validates experiment descriptions, generates deployment specifications, starts and stops workloads, checks health, repeats trials, and exports CSV results.

The framework uses Docker containers built for both x86-64 servers and ARM Raspberry Pi devices. Experiments use Docker Swarm, although the architecture is intended to support other Docker-based orchestrators, including Kubernetes.

Monitoring is provided by containerized Netdata agents, Prometheus, and power probes. Measurements are collected every five seconds. Metrics include:

• non-idle CPU utilization;
• memory usage in MiB;
• disk I/O in KiB;
• network I/O in bytes;
• average power in watts;
• CPU temperature;
• streaming tuples and latency;
• database operation rates;
• completed ML queries and batches per second;
• ML latency and accuracy;
• network traffic;
• cold-start or model-loading duration.

Supported workloads include component stressors, Yahoo Streaming Benchmark, MongoDB/YCSB operations, and MLPerf-derived image inference. The streaming workload uses Kafka, Redis, and Apache Storm, Flink, or Spark. The database workload uses MongoDB or Redis with configurable read, update, and insert operations and uniform, latest, or Zipfian distributions. The ML workload supports local and streaming execution with ONNX, NCNN, and TensorFlow backends.

The framework compares isolated and co-located execution. It can pair ML inference with CPU, memory, disk-I/O, or network stress, MongoDB/YCSB, streaming analytics, or combinations of these workloads.

The results show that interference is resource-specific. On a medium server, isolated ML startup takes approximately 244 seconds. CPU and disk-I/O stress increase startup time by approximately 35–40%, while memory stress increases it to approximately 505 seconds. ML throughput falls from 7.29 batches/s in isolation to approximately 4.5–5.5 batches/s with several co-tenants, 4.04 batches/s under CPU stress, and 2.54 batches/s under memory stress.

The framework also compares hardware. ML throughput is reported as 1.12 batches/s on a Raspberry Pi, 9.5 on a small server, 7.2 on a medium server, and 16.6 on a large server. Average inference power is approximately 5 W on the Raspberry Pi, 90 W on the medium server, 144 W on the large server, and 150 W on the small server. The reported energy per batch is 4.52 J on the Raspberry Pi, 8.65 J on the large server, 12.5 J on the medium server, and 15.73 J on the small server.

These results establish that performance, power, and temperature must be evaluated jointly. Network stress is comparatively non-antagonizing with respect to CPU and temperature, while memory and CPU stress are strongly antagonizing. Local ML execution maximizes throughput but consumes more CPU and memory; streaming execution reduces local resource use while increasing network traffic and lowering throughput.

6. Related benchmark families

EdgeBench is part of a broader edge-benchmarking ecosystem. CAVBench supplies domain-specific workloads such as ORB-SLAM2, SSD object detection, CIWT tracking, EVBattery diagnostics, DeepSpeech, and OpenALPR. Its key distinction is the combination of application latency with QoS–resource-utilization curves and Matching Factor metrics (Wang et al., 2018).

DeFog evaluates cloud-only, edge-only, and cloud-edge deployments using six applications, including YOLOv3, PocketSphinx, Aeneas, iPokeMon, FogLAMP, and RealFD. It emphasizes service placement, stress, concurrent users, communication latency, computation latency, and real-platform execution (McChesney et al., 2019).

Edge AIBench extends the scope to complete client–edge–cloud AI scenarios, including ICU monitoring, surveillance cameras, smart homes, and autonomous vehicles. It includes training, inference, data collection, and federated-learning goals, but the paper provides limited quantitative methodology and no numerical experimental results (Hao et al., 2019).

DeepEdgeBench focuses on embedded neural-network inference across the ASUS Tinker Edge R, Raspberry Pi 4, Google Coral Dev Board, NVIDIA Jetson Nano, and Arduino Nano 33 BLE. It measures inference time, power, energy, and accuracy. Its principal result is conditional: the Coral Dev Board performs best for compatible fully quantized TensorFlow Lite models, while Jetson Nano is more flexible for floating-point models (Baller et al., 2021).

BenchFaaS provides a reproducible FaaS testbed across Raspberry Pis and virtual machines, with K3s, OpenFaaS, Nebula, Linux netem, and JMeter. It evaluates overhead, compute-intensive functions, payload size, scalability, and chained functions under centralized and distributed WAN conditions (Carpio et al., 2022).

The survey “A Survey on Edge Performance Benchmarking” classifies edge benchmarks by system under test, analyzed techniques, and benchmark runtime. It identifies persistent gaps in full device–edge–cloud evaluation, accelerators, storage, networks, energy, orchestration, mobility, multi-tenancy, and reproducibility (Varghese et al., 2020).

7. Scope, limitations, and interpretation

The different EdgeBench projects should not be conflated. The 2018 suite is primarily a commercial platform comparison. The 2020 suite is a configurable workflow framework. The 2025 suite is an automated multi-tenancy and interference-evaluation framework. DeepEdgeBench is an embedded-AI inference benchmark, while the 2026 EdgeBench measures long-horizon agent learning rather than edge infrastructure.

Across the edge-computing usages, several methodological limitations recur:

• Incomplete end-to-end semantics: edge-only measurements may terminate at local delivery even when globally synchronized visibility is required.
• Limited hardware diversity: many evaluations use one Raspberry Pi, one CPU platform, or a narrow set of servers.
• Insufficient tail statistics: means and medians frequently replace p95, p99, deadline-miss ratios, or confidence intervals.
• Network simplification: experiments often omit wireless variability, mobility, packet loss, bandwidth contention, or multi-edge routing.
• Incomplete energy analysis: some frameworks report power, while others infer energy benefits without direct measurement.
• Application-specific coverage: benchmark workloads may not generalize across industrial IoT, robotics, healthcare, databases, distributed storage, and AI inference.
• Platform and software confounding: cloud runtimes, container systems, model libraries, compilers, and accelerator backends are often not equivalent.
• Reproducibility gaps: exact versions, input streams, preprocessing, warm-up rules, repetitions, hardware counters, and raw traces are not always supplied.

A mature EdgeBench architecture would therefore combine several layers: low-level component characterization, application and workflow execution, resource and network measurement, energy and power accounting, service placement, multi-tenancy, fault and mobility testing, and statistically rigorous reporting. It would report raw latency and resource distributions alongside any aggregate score, distinguish local from globally synchronized completion, and preserve workload-specific quality metrics such as accuracy, prediction error, tracking quality, or deadline compliance.

The central principle shared by the strongest EdgeBench-related work is that edge performance is multidimensional. Execution time alone cannot explain why cloud, edge, and hybrid deployments differ. Workload structure, data volume, storage semantics, hardware heterogeneity, network placement, concurrency, resource contention, power, and application quality jointly determine whether a platform is suitable for a particular edge deployment.