Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Comprehensive Benchmarking Analysis of Fault Recovery in Stream Processing Frameworks (2404.06203v3)

Published 9 Apr 2024 in cs.DC and cs.SE

Abstract: Nowadays, several software systems rely on stream processing architectures to deliver scalable performance and handle large volumes of data in near real-time. Stream processing frameworks facilitate scalable computing by distributing the application's execution across multiple machines. Despite performance being extensively studied, the measurement of fault tolerance-a key feature offered by stream processing frameworks-has still not been measured properly with updated and comprehensive testbeds. Moreover, the impact that fault recovery can have on performance is mostly ignored. This paper provides a comprehensive analysis of fault recovery performance, stability, and recovery time in a cloud-native environment with modern open-source frameworks, namely Flink, Kafka Streams, and Spark Structured Streaming. Our benchmarking analysis is inspired by chaos engineering to inject failures. Generally, our results indicate that much has changed compared to previous studies on fault recovery in distributed stream processing. In particular, the results indicate that Flink is the most stable and has one of the best fault recovery. Moreover, Kafka Streams shows performance instabilities after failures, which is due to its current rebalancing strategy that can be suboptimal in terms of load balancing. Spark Structured Streaming shows suitable fault recovery performance and stability, but with higher event latency. Our study intends to (i) help industry practitioners in choosing the most suitable stream processing framework for efficient and reliable executions of data-intensive applications; (ii) support researchers in applying and extending our research method as well as our benchmark; (iii) identify, prevent, and assist in solving potential issues in production deployments.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (24)
  1. Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark. In Proceedings of the 2018 International Conference on Management of Data. ACM, 601–613.
  2. Chaos Engineering. IEEE Software 33, 3 (2016), 35–41.
  3. Apache Flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 36, 4 (2015).
  4. Jeffrey Dean and Sanjay Ghemawat. 2008. MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM 51, 1 (Jan. 2008), 107–113.
  5. A survey on the evolution of stream processing systems. The VLDB Journal 33, 2 (2024), 507–541.
  6. Khaos: Dynamically optimizing checkpointing for dependable distributed stream processing. In FedCSIS. IEEE, 553–561.
  7. Sören Henning and Wilhelm Hasselbring. 2022. A Configurable Method for Benchmarking Scalability of Cloud-Native Applications. Empirical Software Engineering 27, 6 (Aug. 2022).
  8. Sören Henning and Wilhelm Hasselbring. 2024. Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud. Journal of Systems and Software 208 (2024), 111879.
  9. ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks. In Proceedings of the 15th ACM/SPEC International Conference on Performance Engineering. ACM. In press.
  10. A survey on automatic parameter tuning for big data processing systems. ACM Comput Surv 53, 2 (2020), 1–37.
  11. A catalog of stream processing optimizations. ACM Comput Surv 46, 4, Article 46 (March 2014), 34 pages.
  12. A Performance Comparison of Open-Source Stream Processing Platforms. In GLOBECOM. IEEE, 1–6.
  13. Stream Bench: Towards benchmarking modern distributed stream computing frameworks. In UCC. IEEE, 69–78.
  14. Benchmarking modern distributed streaming platforms. In ICIT. IEEE, 592–598.
  15. Streams and Tables: Two Sides of the Same Coin. In Proceedings of the International Workshop on Real-Time Business Intelligence and Analytics (BIRTE ’18). ACM, Article 1, 10 pages.
  16. Bianca Schroeder and Garth A Gibson. 2009. A large-scale study of failures in high-performance computing systems. IEEE TDSC 7, 4 (2009), 337–350.
  17. CheckMate: Evaluating Checkpointing Protocols for Streaming Dataflows. arXiv preprint arXiv:2403.13629 (2024).
  18. Li Su and Yongluan Zhou. 2021. Fast recovery of correlated failures in distributed stream processing engines. In Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems. 66–77.
  19. Scabbard: Single-node fault-tolerant stream processing. VLDB Endowment 15, 2 (2021), 361–374.
  20. Giselle van Dongen and Dirk van den Poel. 2021. A Performance Analysis of Fault Recovery in Stream Processing Frameworks. IEEE Access 9 (2021), 93745–93763.
  21. Self-adaptation on parallel stream processing: A systematic review. Concurr Comp 34, 6 (2022).
  22. A systematic mapping of performance in distributed stream processing systems. In Euromicro Conference on Software Engineering and Advanced Applications. IEEE.
  23. Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka. In SIGMOD/PODS ’21. ACM, 2602–2613.
  24. A comprehensive study on fault tolerance in stream processing systems. Front Comput Sci 16, 2 (2022), 162603.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now