- The paper demonstrates that shifting from G1GC to ZGC, alongside kernel and storage tuning, reduces median publish latency from 18.1 ms to 3.88 ms while reaching 1.5M messages per second.
- It identifies critical latency sources such as journal fdatasync delays and undocumented kernel block-layer effects that impact NVMe performance.
- The study provides actionable recommendations for hardware, JVM, and OS optimizations to improve Pulsar throughput and minimize latency spikes.
Benchmarking and Latency Optimization of Apache Pulsar: Technical Analysis
Introduction and Problem Context
This paper provides a rigorous examination of Apache Pulsar's performance characteristics at enterprise scale, focusing on latency reduction and validating high-throughput capabilities on bare-metal Kubernetes deployments. The study addresses the request-to-acknowledgment path in Pulsar, elucidating latency sources encountered in real-time event streaming applications across distributed physical security infrastructure. The heterogeneity of underlying storage and the JVM's garbage collection (GC) behavior, compounded by OS-level factors, are dissected through production-grade profiling and targeted experimentation.
Production-System Investigation and Baseline Characterization
The initial empirical analysis scrutinizes multiple live Pulsar clusters, identifying persistent median publish latency of 13-18 ms even at moderate ingestion rates (700–9,000 msg/s). Latency spikes up to 213 ms were evident, with latency breakdown revealing that journal fdatasync on SSDs/NVMe and BookKeeper's internal processing dominated the publish pipeline. Only minor contributions (<15%) originated in the broker/network and tunable group synchronization mechanisms. The data suggest that hardware-induced delays from aged storage and JVM GC pauses are primary performance limiters, unaffected by conventional broker-level configuration changes.
Root Cause Analysis Using JFR and OS Profiling
Three critical latency sources were isolated via Java Flight Recorder (JFR)-driven introspection and OS workload analysis:
- G1GC Pauses: Empirical results showed that G1GC, under 32 GB heap conditions, incurs unpredictable major stop-the-world pauses—manifesting as 213 ms spikes. Transitioning to ZGC Generational eliminated these pathological tail latencies, reducing P99 consumer latency from 197 ms to 18 ms.
- Journal fdatasync on Worn Media: Validated across production environments, worn SSDs and NVMes induced journal commit delays (5.1 ms P50 typical, up to 6.6 ms on degraded hardware), directly inflating overall publish latency. Only state-of-the-art NVMe (<0.02 ms fsync) could support sub-millisecond persistence demands.
- Kernel Page Cache Writeback (Novel Mechanism): Previously undocumented, it was shown that ledgers and journals on physically discrete NVMe volumes still suffer synchronized peak latency due to shared Linux kernel block-layer resource contention. During BookKeeper’s log flushes,
ForceWriteThread experienced system-level fdatasync inflation (from <1 ms to 15–22 ms). This resource coupling rebuts the assumption of device-level isolation for I/O critical sections and is not addressed in the Pulsar or BookKeeper documentation.
Latency Mitigation Experiments
Successive experiments confirmed the efficacy of several mitigations:
- Garbage Collection Algorithm Shift: ZGC eliminated GC-induced publish tail spikes, outperforming G1GC across P50 and P99 metrics at scale. The generational mode of ZGC on Java 21 was decisive for consistent low-latency operation.
- Write Cache Flush Tuning: Adjusting BookKeeper’s write-cache flush interval from 60s to 30s reduced median latency by 35% without increasing tail latencies, by minimizing the magnitude and frequency of dirty page build-up.
- Linux Kernel Parameter Tuning: Reducing
dirty_ratio to 2 capped dirty page accumulation, reducing kernel-induced burst I/O latency—especially critical at flush boundaries.
Integrating these changes with modern NVMe storage shrank median publish latency by 79% (18.1 ms to 3.88 ms) while enabling a 50-fold increase in throughput on the same hardware.
Benchmark Validation: 1.5 Million Messages per Second
The optimized architecture—leveraging 3 bare-metal nodes with dedicated NVMe journals, ZGC Generational, OS kernel tuning, and Kubernetes-structured topology—sustained 1,499,947 messages per second with 3.88 ms median publish latency over a ten-minute window, with zero failures. P99% publish latency remained within 6.5 ms, while end-to-end latency (including consumer) remained below 25 ms for 99% of messages. Compute resources were non-limiting (65–82% headroom); network saturation on 10 Gbps NICs was the sole bottleneck.
Scaling Architecture and Theoretical Throughput Projections
The study outlined a scalable architecture capable of linearly increasing throughput with minimal changes:
- Upgrading network links from 10 Gbps to 25 Gbps (no node count increase) would double aggregate throughput to 3 million msg/s.
- Extending to five such clusters with Pulsar’s native key-based partition routing (each managing 128 partitions spread by account/cluster) offers 15 million msg/s throughput on 15 nodes. This provides both horizontal scaling and per-cluster failure isolation without external load balancing.
These configurations demonstrate that Pulsar can achieve massive message throughput with minimal hardware and without architectural complexity if critical latency sources are systematically mitigated.
Key Implications and Recommendations
Strong claims include:
- A previously undocumented Linux kernel block-layer effect that can multiply I/O latency across physically discrete NVMe drives under concurrent flush.
- Median publish latency reduced to 3.88 ms while achieving 50x the original production throughput on commodity hardware with no failures.
- Projected linear scaling to 1.3 trillion messages/day on 15 nodes without external load balancers.
Recommended engineering practices:
- Dedicate contemporary NVMe exclusively for journal operations.
- Adopt ZGC Generational over G1GC on all JVMs to eliminate GC tail latency.
- Aggressively tune kernel dirty ratios (as low as 2), mitigating flush-induced tail latency.
- Reduce BookKeeper cache flush intervals to balance steady-state and P99 performance.
- Use KeyShared subscriptions and bundle auto-splitting for adaptive, high-cardinality workloads.
Practical and Theoretical Implications
This work provides actionable guidance for both Pulsar operators and distributed system architects. By uncovering OS-level interactions not captured in official documentation, it challenges assumptions about device isolation in high-performance disk I/O. Additionally, it suggests that JVM and OS kernel evolution (e.g., ZGC, more granular dirty page controls) are now critical enablers of low-latency operation at scale in cloud-native streaming systems.
Future research should focus on extending kernel-level instrumentation to forecast I/O interference in NVMe-dense environments, generalizing partition-routing strategies for hybrid-cloud federation, and exploring further hardware-software codesign (e.g., user-space I/O stacks) to push latency bounds.
Conclusion
The paper delivers a comprehensive, empirical analysis that not only optimizes Apache Pulsar’s end-to-end latency and throughput but also extends the understanding of kernel-application interactions affecting distributed log persistence. Documented engineering interventions and architectural recommendations provide a template for both immediate deployment improvement and future scalability considerations in high-throughput messaging systems. The identification of kernel-layer I/O contention advances both operational best practice and theoretical knowledge in the deployment of distributed event streaming infrastructure (2603.29113).
