GraphFlash: Serverless Graph Processing

• GraphFlash is a serverless, subgraph-centric framework for large-scale graph analytics that externalizes state management to shared storage.
• It offers two execution modes—rotating for resource-constrained environments and pinned for performance—balancing elasticity with efficiency.
• Advanced optimizations like partition-aware key aggregation and intra-function co-location reduce I/O overhead and enhance scalability.

Searching arXiv for the specified paper and closely related serverless graph-processing work. GraphFlash is a fully serverless, subgraph-centric graph processing framework designed to make large-scale graph analytics practical on serverless infrastructure while preserving stateless, ephemeral function execution and externalized state management through shared storage (Zhao et al., 12 May 2026). It targets the mismatch between iterative, stateful graph workloads and function-as-a-service environments, where existing graph systems either depend on statically provisioned clusters or incur high overhead from fine-grained external I/O. Within this setting, GraphFlash combines BSP-style supersteps, a GRAPE-based subgraph-centric programming model, shared external storage for coordination and communication, and two execution modes—rotating and pinned—to balance elasticity against performance under real serverless constraints (Zhao et al., 12 May 2026).

1. Problem Setting and Design Objectives

Large-scale graph analytics in domains such as social networks, web graphs, and recommendation systems is commonly executed on distributed graph processing frameworks such as Giraph and GraphScope. In the formulation used by GraphFlash, these systems require statically provisioned clusters, assume relatively long-lived jobs and stable workloads, and are tuned for high throughput rather than rapid elasticity (Zhao et al., 12 May 2026). The resulting issues are poor elasticity, over- or under-provisioning, and high operational overhead from machine management, deployment, configuration, scaling, and teardown.

Serverless computing is attractive because it offers automatic scaling, fine-grained billing, and no cluster management. The central difficulty is that graph workloads are iterative and stateful, whereas serverless functions are stateless and ephemeral. Persisting vertex and edge state across supersteps requires external storage; communication also typically passes through external storage because direct function-to-function communication is absent. Existing serverless graph systems therefore incur substantial latency and I/O amplification, especially when state is maintained at vertex granularity (Zhao et al., 12 May 2026).

GraphFlash is motivated by the need for a truly serverless graph processing system that achieves performance comparable to distributed frameworks, retains elasticity and pay-per-use execution, and operates under real serverless constraints: stateless functions, external storage, limited memory and CPU per function, cold starts, short timeouts, and concurrency limits (Zhao et al., 12 May 2026). Its stated design goals are high performance, true serverless compatibility, elasticity and practicality, cost efficiency, and scalability to graphs with hundreds of millions of vertices and billions of edges (Zhao et al., 12 May 2026).

2. Architecture and Computation Model

GraphFlash consists of three principal components: coordinator functions, worker functions, and MaaS (Memory-as-a-Service) (Zhao et al., 12 May 2026). The coordinator is instantiated once per graph task and is responsible for initializing metadata in MaaS, invoking workers, implementing BSP-style barriers using an unfinished-partition counter, maintaining the keep_computing and finish flags, and advancing the superstep counter until convergence. Coordinators are lightweight and do not perform heavy computation.

Worker functions execute as serverless functions on Knative or AWS Lambda. They load partitions, messages, and partial results from MaaS; execute algorithm logic through PEval in superstep 0 and IncVal in subsequent supersteps; write updated state, messages, and metadata back to MaaS; and atomically update shared counters and flags (Zhao et al., 12 May 2026). Workers are multi-threaded, and each partition within a worker may run on its own thread.

MaaS provides the persistence and synchronization substrate. In the Knative implementation, a low-latency key-value store based on Dragonfly stores metadata, aggregated messages, and results, while MinIO stores partitioned graph binaries and potentially bulk messages or other large objects. In the AWS design, the corresponding roles are played by S3 and an AWS-compatible low-latency key-value store (Zhao et al., 12 May 2026). Atomic operations are required for correctness of barriers and flags.

GraphFlash adopts the subgraph-centric model from GRAPE. The graph is partitioned into subgraphs, each containing inner vertices owned by the partition and outer vertices that act as read-only ghost copies of vertices owned elsewhere (Zhao et al., 12 May 2026). Computation proceeds in synchronous BSP supersteps. In each superstep, workers process assigned partitions, write outgoing messages and possibly state back to MaaS, and then wait for the coordinator to observe that all partitions have completed. If no worker has set keep_computing, the coordinator sets finish; otherwise it increments the superstep and triggers the next round (Zhao et al., 12 May 2026).

The developer-facing API is GRAPE-style C++ and exposes abstractions such as VertexRange(), GetEdges(v), GetValue(v), and SetValue(v, value), together with algorithm hooks PEval() and IncVal() (Zhao et al., 12 May 2026). The programming model is templated over graph and value types and supports algorithms including BFS, PageRank, CDLP, and WCC. The use of subgraph-centric execution is central: most edges are internal to partitions, so communication is dominated by boundary effects rather than per-edge messaging. This reduces the number of external storage operations, which GraphFlash identifies as the principal bottleneck in serverless graph processing (Zhao et al., 12 May 2026).

3. Execution Modes and State Management

GraphFlash supports two execution modes chosen by the configuration parameter max_worker (Zhao et al., 12 May 2026). If max_worker >= #partitions, the system enters pinned mode. If max_worker < #partitions, it enters rotating mode.

Rotating mode is designed for resource-constrained environments. Each worker processes multiple partitions sequentially in a superstep. For each assigned partition, the worker loads the partition binary and partial results from MaaS, loads incoming messages, executes PEval or IncVal, and writes updated messages and partial results back to MaaS (Zhao et al., 12 May 2026). Partitions rotate across a fixed worker pool across supersteps. An additional optimization allows idle workers, near the end of a superstep, to prefetch the next superstep’s partition and partial results, reducing startup stall in the following round (Zhao et al., 12 May 2026). This mode supports very small max_worker values and datasets larger than the memory of a single function, but it incurs higher repeated I/O and is more exposed to cold-start effects.

Pinned mode is designed for higher performance when resources are sufficient. Each partition is assigned to a specific worker for the entire job. On the first superstep, the worker loads its partition into memory. On later supersteps it reads only messages and writes only messages; it neither reloads partitions nor writes partial results (Zhao et al., 12 May 2026). This reduces I/O, avoids repeated cold-start overhead if functions are reused, and is especially effective when combined with intra-function partition co-location. Its constraints are that the number of workers must be at least the number of partitions and each worker must have sufficient memory to keep its partition resident (Zhao et al., 12 May 2026).

Function-level statelessness is preserved in both modes because all persistent state lives in MaaS. Graph partitions are stored as pre-partitioned binary files in object storage and are balanced by total vertex degree. In rotating mode, intermediate state is written back every superstep; in pinned mode, partial state remains in memory and only messages are externalized (Zhao et al., 12 May 2026). Messages are not stored per vertex. Instead, GraphFlash aggregates them per destination partition, serializes them in binary format, and may compress them with Zstd (Zhao et al., 12 May 2026).

The coordination semantics are explicit. After processing its assigned partitions, a worker flushes messages and partial state to MaaS, sets keep_computing if any vertex remains active, and atomically decrements unfinished_partition. The coordinator polls until unfinished_partition reaches 0, checks keep_computing, and either sets finish or increments superstep and resets the shared metadata for the next round (Zhao et al., 12 May 2026). This yields recovery at superstep boundaries because the necessary persisted state is externalized.

4. Serverless-Specific Optimizations

A central optimization is partition-aware key aggregation. In a naive serverless design with per-vertex keys, a worker may perform $O(v)$ key accesses per superstep, where $v$ is the total number of vertices and $v_i$ those in partition $P_i$. GraphFlash instead annotates each vertex with the set of adjacent partitions and aggregates all updates destined for partition $P_j$ into a single payload $m_{ij}$ (Zhao et al., 12 May 2026). Incoming communication for worker $w_i$ is therefore bounded by at most $p-1$ keys from other workers, where $p$ is the number of partitions, reducing per-worker key accesses per superstep from $O(v)$ to $v$0, more precisely to at most 2(p-1) (Zhao et al., 12 May 2026). Reported speedups from this optimization include up to 4.06× on G5/CDLP, 3.81× on G5/WCC, and 2.57× on G3/BFS (Zhao et al., 12 May 2026).

A second optimization is intra-function partition co-location. GraphFlash uses multi-threaded workers that process multiple partitions concurrently within a single function. This enables shared vertex arrays, reduces duplication of boundary state, and allows updates across co-located partitions to be applied directly in local memory rather than via MaaS (Zhao et al., 12 May 2026). For cross-worker communication, GraphFlash aggregates messages per worker using bitmaps. Each vertex maintains an adjacent partition bitmap; each worker maintains a worker partition bitmap; bitwise intersection identifies whether that worker should receive a message (Zhao et al., 12 May 2026). Under a fixed total vCPU budget, increasing threads per function from t = 1 to t = 6 yields speedups of 1.32× for G3/BFS, 1.34× for G3/CDLP, 1.31× for G3/WCC, 1.22× for G6/BFS, and 1.12× for G6/WCC, while memory usage falls from 543MB to 254MB on G3 and from 52.3GB to 23.2GB on G6 (Zhao et al., 12 May 2026).

A third optimization is superstep-aware activation. Rather than tracking active vertices from the start, GraphFlash enables activation beginning at a user-chosen superstep. Before the threshold, all vertices are processed; after it, only active partitions or vertices are processed (Zhao et al., 12 May 2026). The rationale is that many algorithms have high activity in early supersteps and sparse activity only later. Reported improvements include CDLP on G3 from 22.4s to 20.1s, WCC on G3 from 4.63s to 3.71s, and WCC on G6 from 21.8s to 17.6s (Zhao et al., 12 May 2026).

Additional optimizations include binary serialization instead of JSON, Zstd compression, prefix compression plus varint encoding for vertex IDs, message batching, and use of Cilium CNI in the Knative deployment to address network bottlenecks (Zhao et al., 12 May 2026). Collectively, these mechanisms are designed to reduce external I/O volume, packet count, and serialization overhead. This suggests that GraphFlash’s performance gains arise not from relaxing serverless assumptions, but from systematically restructuring communication and state movement under those assumptions.

5. Implementation and Empirical Evaluation

GraphFlash has two implementations. The Knative version uses lightweight HTTP servers as functions, with Knative autoscaling fixed to 1 request per instance. Control metadata is stored on three Dragonfly servers for superstep counters, active flags, and worker counters, while graph partitions reside in MinIO (Zhao et al., 12 May 2026). The reported cluster consists of four nodes, each with a 32-core AMD EPYC 9474F @ 3.6GHz, 128GB DRAM, and a 25Gbps inter-node network; the software stack includes Ubuntu 24.04.5, gcc 13.3, Kubernetes 1.30.8, Knative Serving 1.16.0, Docker 28.0.4, Dragonfly 1.62.0, MinIO 2024-12-18, and Cilium CNI (Zhao et al., 12 May 2026). The AWS Lambda version uses AWS Custom Runtime for C++, S3 for partitions and large messages, and a low-latency key-value store mirroring the MaaS design (Zhao et al., 12 May 2026).

The evaluation uses both real and synthetic datasets: dota-league (DL), com-friendster (CF), graph500-23 (G3), graph500-25 (G5), graph500-26 (G6), graph500-27 (G7), datagen-9_2-zf (ZF), and graph500-28 (G8) (Zhao et al., 12 May 2026). Algorithms are BFS, PageRank, CDLP, and WCC. Baselines are Graphless, FaaSGraph, GraphScope, and Giraph (Zhao et al., 12 May 2026).

On Knative in pinned mode, using DL, G3, G5, and G7, GraphFlash is reported as fastest across all algorithms on smaller graphs such as DL and G3. On larger graphs such as G5 and G7, it matches or beats GraphScope and FaaSGraph depending on the algorithm and generally outperforms Giraph (Zhao et al., 12 May 2026). Graphless runs only on the smallest dataset, DL, and GraphFlash is reported as 12×–127× faster than Graphless on DL, with CDLP on DL achieving up to 127× speedup (Zhao et al., 12 May 2026).

For cost efficiency on Knative in rotating mode, using DL with functions configured at 1 core and 2GB memory, all GraphFlash configurations with max_worker from 1 to 4 finish faster than Graphless for all algorithms and use significantly fewer concurrent functions (Zhao et al., 12 May 2026). With a single function, reported reductions in resource consumption include 96.8% for WCC and up to 98.8% for PageRank (Zhao et al., 12 May 2026). Execution time decreases as max_worker increases.

The effect of partition count on G6 in pinned mode is described as U-shaped for execution time and monotonically increasing for cost measured in core·seconds (Zhao et al., 12 May 2026). Too few partitions cause under-parallelization, whereas too many increase edge cuts and message volume. Reported minima are approximately 18 partitions for PageRank and 42 for CDLP (Zhao et al., 12 May 2026).

On AWS Lambda, using DL and a single 768MB GraphFlash function instance, GraphFlash is reported as approximately 9× faster than Graphless on BFS and more than 48× faster on PageRank, with overall speedups up to 48× and cost reduction up to 99.97% (Zhao et al., 12 May 2026). For larger datasets where Graphless cannot run, GraphFlash reports the following execution times: on CF with 64 partitions, BFS 102.95s, PageRank 154.14s, CDLP 281.94s, WCC 126.02s; on ZF with 256 partitions, BFS 226.46s, PageRank 328.18s, CDLP 321.48s, WCC 241.83s; on G8 with 256 partitions, BFS 167.07s, PageRank 311.61s, CDLP 321.48s, WCC 199.87s (Zhao et al., 12 May 2026). The paper characterizes these results as demonstrating stable performance and scalability on AWS Lambda.

6. Relation to Prior Systems, Limitations, and Significance

GraphFlash is conceptually close to subgraph-centric distributed systems such as GraphScope and Gemini in its use of partition-level computation and BSP synchronization, but it differs architecturally in running entirely on FaaS and using MaaS for communication and control (Zhao et al., 12 May 2026). Relative to serverless graph systems, the distinction is sharper. Graphless is purely serverless and vertex-centric, storing each vertex state as a key in external storage; GraphFlash retains the externalized-state principle but replaces fine-grained vertex-centric execution with coarse-grained subgraph-centric execution and adds partition-aware aggregation, co-location, and activation (Zhao et al., 12 May 2026). FaaSGraph attains high performance through shared memory and proxies across co-located containers, but GraphFlash characterizes this as incompatible with true FaaS assumptions and not directly deployable on AWS Lambda-like platforms (Zhao et al., 12 May 2026).

The limitations identified for GraphFlash are dependence on external storage for all cross-function communication, the barrier cost of the synchronous BSP model, the need to tune the activation threshold, overheads from cold starts and metadata traffic on very small graphs, and dependence on the memory, CPU, and concurrency limits of the underlying serverless platform (Zhao et al., 12 May 2026). Some optimizations, such as multi-threading within a function, also presume that the platform permits multiple vCPUs per function (Zhao et al., 12 May 2026).

Future directions mentioned for the system include serverless platforms with direct function-to-function communication, asynchronous or hybrid BSP/asynchronous execution, adaptive partitioning and dynamic repartitioning under skew, and more automated tuning of partition count, max_worker, and activation thresholds (Zhao et al., 12 May 2026). A plausible implication is that GraphFlash’s broader significance lies not only in graph analytics, but in showing that iterative, stateful workloads can be restructured for serverless execution through coarse-grained partitioning, explicit external coordination, and in-function parallelism without abandoning the stateless execution model.

GraphFlash therefore occupies a specific position in the serverless systems landscape: it is a fully serverless graph processing framework whose core claim is that performance comparable to traditional distributed frameworks can be obtained without static cluster provisioning, while preserving elasticity, fine-grained billing, and operational simplicity (Zhao et al., 12 May 2026). Its main technical lesson is that the dominant serverless bottlenecks in graph processing are not inherent to graph workloads alone, but to the granularity at which state and communication are externalized.