BtrLog: Low-Latency Logging for Cloud Database Systems
Abstract: Cloud database systems cannot rely on instance-local disks for write-ahead logging (WAL) durability, forcing WAL onto remote storage. Existing options are unsatisfying: remote block storage like EBS is easy to adopt but adds substantial write latency and cost, while object storage offers excellent durability and low storage cost but is impractical for OLTP due to high latency and per-append cost. Many cloud-native databases, therefore, depend on purpose-built logging backends, which are typically proprietary and tightly coupled to engine-specific replication and recovery protocols, limiting reuse. We present BtrLog, a reusable cloud logging service that combines low-latency durable appends with low-cost archival for the common single-writer architecture. BtrLog replicates log records across a quorum of SSD-backed log nodes in a single network round trip, reducing sensitivity to stragglers in commit latency. To minimize storage cost, log nodes archive records to object storage as large segments, which are written asynchronously and off the latency-critical write path. In our evaluation, BtrLog achieves lower latency than EBS and enables higher end-to-end transaction throughput when integrated into a DBMS.
Summary
- The paper introduces BtrLog, a cloud-native WAL logging system that reduces append latency by up to 4× compared to AWS EBS.
- Its design separates a fast, SSD-backed staging layer from asynchronous archival in object storage to optimize performance and cost.
- Evaluation shows BtrLog delivers up to 2× throughput improvement in LeanStore while reducing per-append costs through effective batching and replication.
BtrLog: A Low-Latency, Cost-Efficient Logging Service for Cloud Database Systems
Introduction and Motivation
This paper presents BtrLog, a reusable cloud-native logging backend that targets the transactionally critical write-ahead log (WAL) for OLTP database systems. The motivation is grounded in a fundamental challenge: in cloud environments, local disks are ephemeral, compelling WALs to reside on persistent remote storage. However, the production WAL workload is characterized by latency sensitivity and high frequency of small, strictly ordered appends generated by a single writer per log stream. Existing solutions are unsatisfactory: remote block storage (e.g., AWS EBS) provides the traditional block abstraction but incurs high latency and cost, while object storage (e.g., S3) is impractical for high-velocity logging due to its latency profile and per-write costs. Industry has responded with proprietary, engine-specific logging subsystems with varying architectures, but there is a conspicuous lack of open, reusable designs with formal performance and durability analysis.
Related System Landscape and Design Rationale
The architecture of shared log and WAL systems is extensively mapped and compared (Figure 1). Consensus- and sequencing-based systems (e.g., Paxos, Raft, Corfu, Scalog) introduce additional network hops and centralization points, inflating commit latency and introducing straggler sensitivity. LazyLog and BookKeeper optimize the single-writer case with quorum replication and single round-trip latency, at the expense of coupling or lack of WAL-specific guarantees. BtrLog takes direct inspiration from these, but targets cloud-native requirements—namely, high multi-tenancy, locality-aware deployment, dynamic resource utilization, and explicit archival stratification.
Figure 1: Comparison of log architectures highlighting latency overheads in leader-sequenced systems and the efficient path in BookKeeper and BtrLog.
In the design space for cloud WAL systems, durability, latency, availability, and cost are core dimensions. BtrLog exploits the workload's single-writer invariant: log records are replicated directly, with explicit commit watermarks and LSN monotonicity handled at the client, eliminating the need for global log sequencing. By separating a low-latency, SSD-backed staging layer from asynchronous archival in object storage, BtrLog consolidates fast path durability with cost-optimal long-term storage.
System Architecture
The principal components of BtrLog are: the DBMS-integrated client library, a cluster of log nodes with NVMe-backed DRAM/SSD staging, a highly durable object storage archive, and a shared metadata store (e.g., S3 with CAS semantics). The append path is engineered for a single network round-trip: the client assigns LSNs and propagates appends to all log nodes, which synchronously persist to SSD and acknowledge upon fsync. Commit is signaled after a quorum of nodes have replied. BtrLog’s protocol maintains WAL serialization by ensuring in-order acknowledgment and commit watermarks, facilitating precise ARIES-style recovery semantics. Log nodes batch and asynchronously flush large segments to object storage, amortizing API invocation costs and decoupling cold data archival.
Figure 2: Overview of BtrLog’s decoupled write/read paths across the log node staging layer and object storage.
Figure 3: State evolution for a log stream in the fault-free case: token acquisition, durable replication, commit advancement, and segment archival.
Read protocols distinguish between hot data (retained in DRAM/SSD, servicing aborts and log shipping) and cold segments (streamed from object storage during recovery). BtrLog supports both hot quorum reads and direct streaming of flushed segments.
Fault Tolerance, Failover, and Protocol Strategies
Client-driven quorum replication masks stragglers and single-node failures, but cloud environments require explicit handling of non-crash-failure modes and correlated faults (e.g., AZ-wide or storage stack bugs). The failover protocol leverages escalating write tokens (wtoken) stored in strongly consistent object metadata to fence and serialize writers across failures (Figure 4). On takeover, a new client queries log nodes, reconstructs the maximal contiguous committed log tail, performs necessary repairs by re-replicating under-replicated records, and advances the commit epoch ensuring monotonicity and atomicity of log progression. Reader logic and lazy consolidation protocols address asynchronous segment flushes, object storage overlap, and duplicate record resolution, ensuring the ARIES invariants are not violated post-failure.
Figure 4: Illustration of client failover, fencing, and repair via token acquisition and tail reconciliation.
BtrLog addresses unreliable network delivery and node churn by adopting a hybrid of on-path detection (peer heartbeats) and cheap, segment-level flushes to object storage as durable repair mechanism. The object store acts as both slow path repair backend and high-bandwidth recovery mechanism.
Implementation Details
The implementation targets commodity cloud hardware: network round-trip latencies (~40 μs) and SSD persist latencies (~30 μs) represent a hard lower bound. The full asynchronous stack is implemented in Rust with a custom io_uring-based framework for efficient, event-driven IO on Linux. The networking stack is UDP-based (obviating TCP’s overhead), with symmetric thread-to-port associations, cross-thread internal load balancing, and a multi-tenant per-core runtime that schedules independent append streams and coordinates unflushed segment tail retention.
Figure 5: BtrLog log node architecture, managing concurrency across append requests, CPU cores, and physical SSDs.
Segment retention and eviction are parameterized by a sliding LSN window, permitting tail-following replicas and high-frequency logs to avoid premature archival under concurrent workloads.
Evaluation
Comprehensive evaluation on AWS clusters demonstrates the dominant impact of architecture on latency and cost. For 128-byte appends, BtrLog consistently achieves p50 latency of 70–110 μs at realistic WAL throughputs (30–500 k appends/s), outperforming EBS io2 by 4× and Apache BookKeeper by up to 2× at the same hardware tier and benchmarking conditions. Single-AZ and multi-AZ deployments are compared, with multi-quorum deployments trading a moderate absolute latency increase for additional availability and durability.
Figure 6: Median and p99 latency with increasing system load for EBS, BookKeeper, and BtrLog.
Figure 7: End-to-end latency before and after log node failure, showing modest increases due to hedging loss.
Figure 8: Latency implications of (2-of-3) vs. (3-of-3) quorum; relaxed quorums reduce both median and tail latency.
When integrated as the WAL backend in LeanStore, BtrLog enables up to 2× improvement in transactional throughput under YCSB-A compared to BookKeeper and exhibits clear cost-efficiency gains compared to both BookKeeper and EBS backends.
Figure 9: LeanStore transactional throughput with alternative WAL backends, affirming throughput improvements via BtrLog.
Cost analysis reveals that BtrLog's design sharply narrows the performance/cost differential between local SSD and cloud-native backends. Per-append cost, assuming full utilization, is minimized by batching segment flushes and amortizing IOPS across high concurrency; BtrLog achieves $\$0.00125$ per 1M appends under typical deployment models, with multi-AZ deployments incurring proportionally more for inter-AZ traffic. These properties are visualized in direct comparison with EBS, S3/S3 Express, and BookKeeper.
Figure 10: Cost, latency, and availability tradeoffs across cloud WAL solutions; BtrLog occupies the optimal frontier.
Implications and Future Directions
BtrLog brings explicit separation between the latency-critical path of transaction log durability and the large-scale archival properties needed for recovery, backup, and analytical rollup. Its open design and protocol formalization (including TLA+ validation/modeling) facilitate integration in both OLTP DBMSes and other single-writer, strong-durability log workloads (e.g., streaming engines, coordination services).
Practically, the results show that public cloud block storage offerings are not optimized for WAL semantics, and the performance/cost gains of WAL-optimized, multi-tenant log backends are robust across hyperscalers. The modular structure allows for future extensions such as multi-writer protocol layers, direct support for externalized authentication/gRPC APIs, and programmable archival transforms.
Theoretically, BtrLog provides an explicit, reusable baseline for the latency/durability/cost tradeoff envelope in cloud WAL systems. Its protocol and engineering choices are situated for extension to new cloud-native DBMS paradigms, including disaggregated and autonomous recovery systems. The source code and TLA+ artifacts are available for reproducibility and further protocol research.
Conclusion
BtrLog establishes a new reference design for WAL logging in the cloud: it achieves strictly lower append latency and cost than EBS and BookKeeper while providing high durability, flexible deployment (single- and multi-AZ), and strong ARIES/WAL semantics required for modern DBMSs. The architectural insights and explicit tradeoff analysis materially contribute to the systematization and composability of cloud database infrastructure.
Reference: "BtrLog: Low-Latency Logging for Cloud Database Systems" (2606.27051).
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
BtrLog: Low‑Latency Logging for Cloud Database Systems — A Simple Explanation
1. What is this paper about?
This paper introduces BtrLog (pronounced “better log”), a new way for cloud databases to save their “journal” of changes quickly and safely. Think of it like a diary where a database writes down every change so it can recover if something goes wrong. The goal is to make these diary writes very fast and affordable in the cloud, while still keeping them safe.
2. What questions did the researchers ask?
In simple terms, the paper asks:
- Can we design a shared logging service for many databases that is both very fast and low-cost in the cloud?
- Can we get strong safety (data doesn’t get lost) without paying high prices or waiting a long time?
- Can we keep recent changes super fast to write, and still store older data cheaply?
- Does this design actually beat popular cloud storage options in speed and cost?
3. How does BtrLog work?
Imagine you’re writing important notes and you want to:
- Save each note quickly so it won’t be lost.
- Keep costs low over time.
- Be able to read recent notes fast.
Here’s the everyday-language version of BtrLog’s approach:
- One writer at a time:
- Many modern databases split work so that each part has exactly one “writer” making changes. That means there’s no arguing about the order of notes; they’re written in the order they’re made.
- Write to several “friends” at once (quorum replication):
- When the database writes a note, BtrLog sends it to 3 fast machines (with NVMe SSDs — like super-fast USB drives).
- As soon as 2 of the 3 confirm they’ve saved it (that’s the “quorum”), BtrLog says “you’re safe to continue.” This avoids getting stuck if one machine is slow.
- Pressing “Save” for real:
- Each log machine doesn’t just keep the note in memory; it actually writes it to its SSD and “fsyncs” it (think: really pressing Save so it survives a power cut). This adds a tiny bit of time, but makes your note much safer.
- Batch and archive to cheap storage:
- Instead of sending every tiny note to the cloud’s object storage (like Amazon S3) one by one (which would be slow and costly), BtrLog bundles many notes into a big “box” (a large segment, e.g., 16 MB) and ships that box later, in the background. This keeps the “hot path” (the part that must be fast) very quick while making long-term storage cheap.
- Keep the recent “tail” handy:
- BtrLog keeps the most recent notes in memory (and on SSD) so they can be read quickly (for example, by read-only replicas that want to stay in sync).
Some terms explained with analogies:
- Latency: How long you wait between writing a note and getting a thumbs-up that it’s safe.
- Quorum: Waiting for “enough” friends (like 2 out of 3) to confirm they saved your note before you move on.
- SSD vs. object storage: SSDs are like a fast personal locker near you; object storage is like a super-cheap warehouse far away that’s great for long-term bulk storage but slower per item.
- Single round trip: You send the note to the 3 friends once and get replies back once — that’s very fast.
Why not just use other cloud storage?
- Remote block storage (like AWS EBS) is easy to plug in, but often adds several times more latency for small writes.
- Object storage (like S3) is great for cheap, durable archives, but too slow and too expensive per tiny write.
- BtrLog combines the best of both: fast local SSD staging + cheap S3 archiving.
4. What did they find, and why is it important?
The paper tested BtrLog in real cloud setups and found:
- Lower write latency than high-end cloud block storage (EBS): BtrLog’s appends were up to about 4× faster for the small, frequent writes databases make to their logs.
- Higher database throughput: When plugged into a database system, BtrLog’s faster logging helped the database process more transactions per second.
- Lower cost per write: Because BtrLog batches many small writes into bigger segments before sending them to object storage, the per-write cost drops a lot.
- Strong durability: BtrLog only acknowledges a write after saving it to SSD on multiple machines, which protects against crashes and most failures.
- Handles real-world hiccups: If one machine misses a note, readers can still get the full story by asking a quorum of machines. Repairs can be done by fetching missing pieces from peers or during recovery from the archive.
Why this matters:
- For apps that need quick transactions (like shopping carts, payments, or games), shaving off milliseconds on every commit adds up to smoother, faster experiences.
- Lower cost per write means you can scale to more users without exploding your budget.
5. What’s the impact of this research?
- A reusable foundation: Many big cloud companies quietly built their own private logging services. BtrLog is a public, reusable design and open-source implementation that others can adopt and learn from.
- Faster, cheaper cloud databases: By separating “fast now” (SSD quorum) from “cheap later” (object storage), databases can get better performance and lower costs at the same time.
- Better reliability and flexibility: The design supports single‑writer logs common in today’s cloud databases, helps replicas keep up, and remains robust against common failures.
- Informs future systems: The paper explains why general-purpose services (like remote block storage) can be slower for this specific “tiny-append” workload and shows how a specialized design beats them. It also points to future work like streaming reads to replicas, built-in security (encryption/authentication), and exploring alternatives like NVMe‑over‑TCP.
In short, BtrLog shows a practical way to make the critical “write-ahead log” both quick and affordable in the cloud—helping databases run faster and cheaper without giving up safety.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The following list summarizes concrete gaps and unresolved questions that the paper leaves open. These items are framed so that future work can act on them.
- Missing-record repair on the tail: The prototype lacks mechanisms to repair per-node gaps in the in-memory/SSD tail (e.g., due to UDP packet loss). Evaluate and implement (a) client-side refetch from alternate nodes vs. (b) node-to-node gossip, including correctness, complexity, bandwidth overheads, and effects on ack/commit latency under controlled loss.
- Streaming read support to replicas: End-to-end support for low-latency log streaming to read replicas (including gap detection, retransmission, and flow control) is not implemented. Design and benchmark a tailing protocol with backpressure that handles packet loss and slow consumers without resorting to object storage.
- Formal fencing and failover protocol: The paper assumes single-writer semantics but does not specify or evaluate a concrete fencing/ownership-transfer mechanism (e.g., BookKeeper-like fencing). Define the protocol, prove safety (no split brain), and measure failover latency.
- Metadata consistency and failure handling: The design relies on S3 (or compatible object store) conditional writes for metadata, but the exact guarantees, failure modes (e.g., S3 partial unavailability, retries, conditional-write conflicts), and their effects on writer election and log stream lifecycle are not specified or evaluated.
- Durability model and MTTDL analysis: The paper argues for SSD persistence before ack but does not provide a quantitative durability analysis across failure modes (single-AZ vs multi-AZ, correlated failures, instance termination vs reboot, SSD failure). Provide an MTTDL/RPO model comparing BtrLog to EBS/S3 under realistic failure rates and flush timeouts.
- Single-AZ tail risk vs. flush timeout: The trade-off between low-latency single-AZ deployments and the risk of losing the unflushed tail before archival is qualitatively discussed but not quantified. Characterize data-loss probability and cost for different flush timeouts and AZ configurations.
- Exact read semantics and linearizability: While the paper refers to cLSN and quorum reads, it does not formalize the read/scan consistency model (e.g., linearizable reads below cLSN, monotonic scans across nodes). Specify guarantees and verify via model checking or Jepsen-style testing.
- LSN window sizing and flow control: The “LSN window” guaranteeing an in-memory tail is described but lacks concrete policies for sizing, admission control, and backpressure when writers outpace flush/archival. Develop adaptive window control and evaluate its impact on tail latency and memory utilization across many streams.
- Multi-tenant isolation and fairness: The prototype notes memory pressure and throughput cliffs with many streams but does not implement or evaluate tenant isolation (fair queuing, rate limiting, priority, per-tenant memory quotas) or protection from noisy neighbors.
- Security, authentication, and encryption: The current datapath uses UDP without implemented authentication/encryption. Design and evaluate secure transport options (e.g., DTLS, QUIC, in-proc TLS offload, or a gateway), quantifying latency overheads and CPU costs.
- Recovery from S3 with divergent segment variants: The paper sketches a merge/consolidation strategy for multiple segment variants on S3 (due to holes), but does not detail the algorithm, correctness proofs, or worst-case costs (extra GETs, tail latency on recovery). Specify, implement, and evaluate consolidation at scale.
- Garbage collection and retention: Policies and mechanisms for retention, compaction, and GC of old segments in S3 (including duplicate or hole-containing variants) are not described. Define lifecycle management, correctness (no premature deletion), and cost/latency impacts.
- Object-store dependency and portability: The design and metadata layer assume S3 semantics (If-Match, strong consistency). Assess portability to GCS/Azure Blob (conditional ops, consistency), and quantify the cross-cloud behavior and performance differences for both data and metadata.
- Cost model assumptions and sensitivity: The cost analysis assumes full utilization and best-case IOPS, excludes EBS capacity cost, and fixes appends-per-I/O batching factors. Provide sensitivity analyses (utilization, batch size distribution, cross-AZ bandwidth, read traffic) and amortization under real workloads.
- Deployment across AZs/regions: Multi-AZ configurations are mentioned but the write quorum choices, placement policies, latency/availability trade-offs, and cross-AZ bandwidth costs are not fully specified nor evaluated under failure scenarios (e.g., AZ loss).
- UDP reliability under adverse conditions: The design banks on rare packet loss in modern DC networks; behavior under congestion, microbursts, or loss bursts is not explored. Evaluate reliability and implement loss recovery/ECN-based congestion control or consider QUIC/TCP fallback.
- End-to-end DBMS integration breadth: Results are shown for LeanStore; generality across engines with different WAL behaviors (e.g., PostgreSQL with full-page writes, group-commit settings, multi-partition commits) is not evaluated. Provide integrations and benchmarks across diverse DBMSs.
- Large-record handling and fragmentation: The system assumes sub-4 KiB appends are common; handling of larger records (e.g., full-page writes, >16 KiB) and their impact on batching, SSD I/O, and latency are not quantified or optimized.
- Hotspot mitigation and stream placement: Policies for mapping many log streams to nodes, rebalancing hot streams, and scaling out/in are not presented. Design placement, load balancing, and migration strategies with minimal impact on commit latency.
- Backpressure to clients: The write path and windowing imply the need for backpressure when nodes are saturated or archival lags; the mechanism and client behavior (retries, throttling, exponential backoff) are not specified.
- Protocol correctness and verification: No formal specification or model checking of the client-driven quorum replication protocol (including cLSN computation, tail repair, and recovery from mixed node states) is provided. Develop a formal model and verify safety/liveness.
- Data integrity and scrubbing: Beyond naming segments by hashes, end-to-end integrity (per-record checksums, on-SSD corruption detection, periodic scrubbing, and S3 data verification) is not described. Specify integrity mechanisms and measure detection/repair overheads.
- Operational concerns (upgrades and rolling restarts): Procedures for versioning, wire compatibility, and zero-downtime rolling upgrades of log nodes are not discussed. Define upgrade protocols and test under load.
- SLA definition and SLO compliance: The paper reports median/tail latencies but does not define service-level objectives (e.g., p99 append latency, durability lag budget) or show compliance under diurnal patterns and failure injections.
- NVMe-over-TCP exploration: NVMe/TCP is discussed as an alternative but remains unimplemented. Prototype a hybrid (server-managed control plane with NVMe/TCP dataplane) and compare performance, complexity, and failure handling vs. BtrLog’s current architecture.
- Gateway trade-offs: The proposed gRPC/HTTP gateway would add an RTT; its design, caching, batching strategies, security integration, and impact on latency/throughput are not evaluated.
- Recovery time objectives (RTO): The performance and predictability of restart recovery (scan/read from S3, reconstructing segments, rewarming the tail) are not measured. Provide RTO measurements across log sizes and failure modes.
- Write skew and hedging: The paper alludes to straggler mitigation, but specific hedging strategies (duplicate sends, adaptive quorum selection) and their effects on tail latency and cost are not detailed or evaluated.
Practical Applications
Immediate Applications
The following applications can be deployed today using BtrLog’s design and prototype capabilities, or with modest integration work. They leverage single-writer WAL semantics, client-driven quorum replication to NVMe-backed log nodes, and asynchronous archival to object storage (e.g., S3).
- Low-latency WAL backend for single-writer cloud databases
- Sectors: software, cloud databases (DBaaS), finance (OLTP), e-commerce, gaming
- What: Replace EBS-based WAL with BtrLog in single-AZ or multi-AZ deployments to cut commit latency (single network round-trip) and improve throughput.
- Tools/products/workflows:
- Client library integration into DB engines with pluggable WAL modules (e.g., Rust/Java client stub), LeanStore-like engines, custom OLTP engines
- Deployment on NVMe-equipped instances (e.g., c6id.* class), S3 for archival
- Operational dashboards for quorum health and S3 flushing status
- Assumptions/dependencies:
- Single-writer per log stream (typical per-partition/leader patterns)
- UDP-based prototype; need to add authentication/encryption (e.g., DTLS/TLS) for production
- Dependence on instance-local NVMe and low-loss datacenter networking
- Writer fencing/ownership transfer on failover must be integrated with the DB’s leader election
- Cost-optimized logging for high-rate OLTP workloads
- Sectors: finance (payments, ledgers), ad-tech, gaming, retail
- What: Reduce per-append and per-GB costs by staging on SSD and asynchronously archiving large segments to object storage; tune flush timeouts to control S3 PUT frequency.
- Tools/products/workflows:
- Cost calculators using the paper’s per-append normalization
- Tunable flush timeouts per stream; per-tenant pricing aligned to memory window size and S3 PUT volume
- Assumptions/dependencies:
- Workloads dominated by small (<4 KiB) appends
- Aggressive batching to reduce S3 PUTs (e.g., 16 MB segments)
- Accurate capacity planning for memory-backed tail windows
- Faster read-replica log shipping within a single AZ
- Sectors: cloud databases, SaaS multi-tenant DBs
- What: Feed secondaries from the tail in DRAM/SSD via quorum reads to reduce replication lag without hitting object storage in the common case.
- Tools/products/workflows:
- Log followers that read below last-observed committed LSN (cLSN)
- LSN window tuning to retain a bounded tail for slowish replicas
- Assumptions/dependencies:
- Currently requires client-side handling of gaps (e.g., re-fetch missing LSNs via another node); gossip-based repair not yet in the prototype
- Sizing the LSN window to match replica speed
- Durable, low-latency event logs for single-writer microservices
- Sectors: software (microservices), fintech (per-account or per-ledger partitions), IoT gateways (per-device/partition writer)
- What: Use BtrLog as an append-only commit log per aggregate/partition where a single writer is enforced (event sourcing/CQRS patterns).
- Tools/products/workflows:
- Sidecar library in services; optional gateway layer offering gRPC/HTTP for simpler client integration (with an extra RTT)
- S3-based archival for retention and reprocessing
- Assumptions/dependencies:
- Enforce single writer per stream (application-level ownership)
- AuthN/Z and encryption need to be added (prototype notes CPU headroom for this)
- Disaster recovery and time-travel via object storage archives
- Sectors: finance, healthcare (compliance), SaaS, enterprise IT
- What: Use low-cost S3 archives of log segments for crash recovery, auditing, and point-in-time restore workflows.
- Tools/products/workflows:
- Recovery readers that consolidate multi-variant segments and reconstruct complete logs
- Integration with snapshot managers (e.g., DB snapshots + log replay)
- Assumptions/dependencies:
- Rely on object storage durability and immutability controls (e.g., S3 Object Lock for WORM if required)
- Recovery bandwidth from S3 must be provisioned
- Academic benchmarking and teaching of cloud logging trade-offs
- Sectors: academia (systems courses, research labs)
- What: Use the open-source artifact to teach/benchmark latency vs cost vs availability trade-offs, and compare against EBS/BookKeeper.
- Tools/products/workflows:
- Reproducible AWS/GCP/Azure experiments, per-append cost models
- Assumptions/dependencies:
- Cloud credentials and budgets; access to NVMe-backed instances
Long-Term Applications
These applications require additional engineering, research, or ecosystem integration beyond the current prototype (e.g., multi-writer support, enhanced APIs, or operational maturity).
- Managed, multi-tenant “Logging-as-a-Service” for cloud DBs
- Sectors: cloud providers, DBaaS vendors, enterprise SaaS platforms
- What: Offer BtrLog as a hosted service with SLAs, billing, quotas, isolation, and autoscaling; expose gRPC/HTTP gateway; integrate with IAM/KMS.
- Tools/products/workflows:
- Control plane for tenant onboarding, quotas, and LSN window pricing
- Kubernetes operator, service mesh integration, autoscaling log nodes
- End-to-end encryption and mTLS/DTLS; rotating keys via KMS/HSM
- Assumptions/dependencies:
- Production-grade security, rate limiting, noisy-neighbor mitigation
- Sophisticated scheduling to avoid SSD/DRAM hot spots
- Cross-AZ and cross-region deployments with strong failure resilience
- Sectors: finance (regulatory resilience), healthcare, critical infrastructure
- What: Replicate log tails across multiple AZs/regions, retaining low-latency within-region paths while protecting against datacenter outages.
- Tools/products/workflows:
- Topology-aware quorum policies (e.g., 2/3 within region; geo-quorum variants)
- Asymmetric LSN windows by site; tiered archival (regional/bucket policies)
- Assumptions/dependencies:
- Higher network latency and cross-AZ bandwidth cost; potential need for partial-sync acknowledgments and dynamic hedge strategies
- Careful fencing semantics during regional failover
- Streamed read-replication with gap repair and “replication slot”-style retention
- Sectors: cloud databases, analytics serving layers (HTAP)
- What: Provide first-class streaming reads with backpressure, reader offsets, replica slots, and automatic gap repair (gossip or demand fetch).
- Tools/products/workflows:
- Replica slot management APIs (pin LSN windows per consumer)
- Efficient gossip between log nodes to fix holes and reduce duplicate S3 writes
- Assumptions/dependencies:
- Memory cost per slot/stream; robust admission control and pricing
- More complex server-side read pipelines and monitoring
- NVMe-over-TCP variant for ultra-low latency paths
- Sectors: high-frequency trading, specialized HPC systems, on-prem clouds
- What: Implement a BtrLog variant leveraging NVMe-over-TCP to approach the physical lower bound (network + SSD) for some deployments.
- Tools/products/workflows:
- Client protocol that handles fencing, failover, and load balancing at the NVMe layer
- Assumptions/dependencies:
- Considerable client-side complexity (fencing, backpressure, failover)
- Operational risks similar to RDMA-based systems (robustness and observability)
- Integration into major open-source and commercial DBMSs
- Sectors: software (PostgreSQL/MySQL variants), vendors, cloud-native DBs
- What: Provide pluggable WAL adapters and failover hooks for Postgres, MySQL, and distributed engines (per-partition leaders), transparently replacing disk/EBS WAL.
- Tools/products/workflows:
- Extensions/plugins for WAL backend; fencing integrated with leader election
- Read replica pipelines built atop BtrLog slots
- Assumptions/dependencies:
- Engine-specific integration complexity (fsync semantics, recovery hooks)
- Extensive correctness/perf validation and failover tests
- Offloading consensus-group persistence for Raft/Paxos partitions
- Sectors: distributed databases, stream processors, KV stores
- What: Use BtrLog for durable per-group logs (single leader per group), reducing device IO, improving commit latency for leader-ordered writes.
- Tools/products/workflows:
- Adapter libraries in Raft/Paxos frameworks to map log appends onto BtrLog
- Assumptions/dependencies:
- Careful alignment of commit/flush semantics with consensus timeouts and leader lease/fencing
- Not suitable for multi-writer at a single log address without added ordering logic
- Analytics and observability products on archived WAL segments
- Sectors: data engineering, observability, compliance/audit
- What: Build pipelines that read S3-segmented WAL for auditing, lineage, time-travel debugging, replay testing, and fraud forensics.
- Tools/products/workflows:
- Segment readers that consolidate variants, emit structured streams
- Incremental ETL into data lakes/warehouses; governance catalogs
- Assumptions/dependencies:
- Schema-aware parsing of log records; cost management for S3 scanning
- Privacy/compliance controls over WAL contents
- Policy and procurement guidelines for cost-effective durability
- Sectors: public sector IT, regulated industries, enterprise architecture offices
- What: Incorporate “persist to local SSD with quorum + delayed object-store drain” into resilience blueprints to meet RTO/RPO at lower cost.
- Tools/products/workflows:
- Reference architectures and calculators (per-append, per-GB, cross-AZ cost)
- Compliance mappings (e.g., WORM on object storage, audit trails)
- Assumptions/dependencies:
- Organization acceptance of ephemeral-instance assumptions (local SSD persistence across reboot, not across termination)
- Clear runbooks for failover, cLSN monitoring, and object-store integrity
- Edge and hybrid-cloud transactional logging
- Sectors: IoT, manufacturing, energy (SCADA), telecom
- What: Run small log-node clusters at the edge with periodic archival to central object storage; enable local-low-latency commits with cloud durability.
- Tools/products/workflows:
- Compact deployments (2–3 nodes) on NVMe-equipped edge servers; scheduled backhaul
- Assumptions/dependencies:
- Variable connectivity; careful sizing of LSN windows and SSD capacity
- Security hardening for untrusted environments
Notes on Feasibility, Assumptions, and Dependencies
- Single-writer semantics: Core assumption; many OLTP and partitioned systems already comply (per primary/leader). Multi-writer logs would need additional sequencing.
- Durability model: Acks after fsync to local NVMe on a quorum of nodes; relies on instance storage persistence across reboot and on asynchronous archival to object storage for long-term durability.
- Network and transport: Prototype uses UDP; production needs encryption and authentication (DTLS/mTLS) and may add a gateway (HTTP/gRPC) at the cost of another RTT.
- Memory footprint and multi-tenancy: The in-memory LSN window per stream enables fast reads but incurs memory cost; services must implement quotas, pricing, and backpressure.
- Cloud provider variability: Latency and cost benefits were demonstrated across multiple clouds, but instance types, cross-AZ bandwidth prices, and object-store behaviors vary.
- S3/object-store semantics: Segmenting and conditional writes (If-Match) assumed; immutable archival recommended for compliance (Object Lock).
- Failure handling: Client-side quorum reads mask per-node gaps; optional gossip-based repair and streaming readers with gap repair are future enhancements for production-grade replication.
Glossary
- Apache BookKeeper: An open-source log service designed for single-writer logs, providing client-driven quorum replication and fencing for safe failover. "In contrast to multi-writer log systems, Apache BookKeeper~\cite{sigops/JunqueiraKR13} targets single-writer logs, as in primary/replica database architectures."
- ARIES: A widely used write-ahead logging and recovery protocol that requires durable storage and defines operations like forcing the log at commit. "At commit, the engine calls sync to ensure all records up to the commit LSN are persisted -- the ARIES paper~\cite{tods/MohanHLPS92} refers to this as forcing the log."
- Availability Zone (AZ): A distinct datacenter location within a cloud region used for fault isolation and low-latency deployments. "BtrLog can be deployed within a single data center (Availability Zone or AZ in AWS) for the lowest latency and cost."
- chain replication: A replication strategy that arranges replicas in a chain; writes traverse the chain, adding hops and latency. "Corfu's chain replication requires six network hops; Scalog's primary-backup strategy reduces it to four, which is still well above the single-round-trip latency achievable with single-writer WAL semantics."
- client-driven quorum replication: A replication approach where the client sends data to all replicas and considers the operation complete once a quorum acknowledges. "BookKeeper uses client-driven quorum replication; in the common case, an append completes in a single client round trip."
- committed LSN watermark (cLSN): A marker indicating the highest log sequence number known to be durably committed cluster-wide or per node. "readers may read from a single log node, restricting themselves to LSNs below the committed LSN watermark (cLSN) last observed from that node."
- consensus protocols (leader-based): Fault-tolerant agreement protocols (e.g., Paxos-like, Raft-like) that replicate a log via a leader coordinating followers. "As the left-hand side of \Cref{fig:shared_log_architectures} shows, leader-based consensus protocols~\cite{podc/OkiL88,usenix/OngaroO14,tocs/Lamport98} replicate a log across a set of nodes coordinated by a leader."
- DRAM-based design: A memory-centric architecture that keeps data in RAM on multiple nodes, relying on replication for durability and low latency. "At the other extreme, one can envision a DRAM-based design similar to RAMCloud~\cite{tocs/OusterhoutGGKLM15}, where the client sends data to a quorum of nodes (``log nodes'') that keep data in DRAM and provide durability through replication."
- eager ordering: A design where the system assigns an order (e.g., LSN) only after both ordering and replication complete, increasing latency. "However, these systems still use eager ordering~\cite{sosp/LuoBHAG24}: clients receive an LSN for an append once both ordering and replication are complete (append(rec) LSN)."
- Elastic Block Store (EBS): A network-attached block storage service providing durable volumes to VMs, often with higher latency than local SSDs. "A straightforward approach is remote block storage, such as Amazon EBS, which provides a network-attached volume that can be re-attached to a different VM after failure."
- fencing: A mechanism to prevent old or concurrent writers from accessing a resource after failover, avoiding split-brain. "BookKeeper therefore implements fencing and recovery to prevent split-brain scenarios during failover."
- fsync: An operation forcing buffered data to be flushed to stable storage to ensure durability. "Only after the SSD write and fsync complete, the node acknowledges the append to the client."
- gossip: A peer-to-peer, background protocol for nodes to exchange state or data to repair inconsistencies. "Gossip-based repair would additionally reduce duplicate S3 writes, since log nodes could use it to fill holes in their local segments."
- IOPS: A throughput metric for storage indicating input/output operations per second, often used for provisioning and cost. "For EBS, we assume peak provisioned performance (256{,}000 IOPS for io2 and 80{,}000 IOPS for gp3) and consider only IOPS cost, excluding storage-capacity cost."
- If-Match (S3): An HTTP conditional-write mechanism used with object storage to ensure updates apply only if the object matches an expected version. "BtrLog requires few metadata operations, which can be implemented using conditional writes (If-Match in S3)."
- instance-local (ephemeral) storage: VM-attached storage that may be lost on instance failure or deprovisioning, limiting its use for durability. "In the cloud, instance-local storage is ephemeral and can be lost on instance failure or deprovisioning~\cite{inststoragepersistence}."
- LazyLog: A log system that separates ordering from durability to achieve single round-trip appends by resolving order lazily on reads. "LazyLog~\cite{sosp/LuoBHAG24} reduces append latency by separating ordering from durability."
- Log Sequence Number (LSN): A monotonically increasing identifier assigned to each log record establishing a total order. "Each record is assigned a monotonically increasing log sequence number (LSN)."
- log shipping: The process of transmitting log records to replicas or secondaries to keep them synchronized with the primary. "The BtrLog protocol supports both log shipping and read replication, and the revised \cref{subsec:happy_path} now explicitly mentions read replication as an example scenario."
- LSN window: A bounded range of recent log sequence numbers retained in memory to serve fast reads and acknowledgments without accessing cold storage. "BtrLog's LSN window, now explicitly described in the revised \cref{sec:implementation}, serves a similar purpose: it guarantees that a bounded suffix of the log tail is retained in memory, ensuring that both pipelined write acknowledgments and tail-following readers are served without accessing object storage."
- NVMe SSD: High-performance SSDs using the NVMe protocol, offering low-latency, high-IOPS local storage. "the tail resides in RAM for fast reads and is backed by local NVMe SSDs for recovery after crashes or power loss."
- NVMe-over-TCP: A protocol that transports NVMe commands over TCP, enabling remote access to SSD semantics across the network. "clients could persist data on remote SSDs directly using NVMe-over-TCP~\cite{nvme-over-tcp}, a protocol supported by the kernel and systems such as simplyblock~\cite{simplyblock} and DAOS~\cite{daos}."
- object storage: A cloud storage model that offers high durability and low cost per GB, ideal for large, infrequently accessed objects. "Cloud object storage, such as S3, provides very high availability and durability, low cost per GB, and high throughput for scans."
- OLTP: Online Transaction Processing; workloads characterized by many small, latency-sensitive transactions. "object storage offers excellent durability and low storage cost but is impractical for OLTP due to high latency and per-append cost."
- out-of-place write: A storage technique writing new data to a fresh location rather than overwriting in place, simplifying persistence and recovery. "Upon receiving an append, a node adds the record to an in-memory segment and writes it out-of-place~\cite{lee2026write} to a local NVMe SSD~#1{2}."
- Paxos group: A set of replicas coordinated by a leader to agree on the order of operations for a partition using a Paxos-like protocol. "each replicated partition (Paxos group) elects a leader."
- primary/secondary architecture: A replication topology with a single writer (primary) and one or more readers (secondaries), simplifying ordering. "In primary/secondary architectures such as Amazon Aurora~\cite{sigmod/VerbitskiGSBGMK17}, Microsoft Azure SQL HyperScale~\cite{sigmod/AntonopoulosBDS19}, Alibaba PolarDB~\cite{pvldb/Li19}, and Google AlloyDB~\cite{alloydb}, only the writer instance issues updates and generates the corresponding log records."
- PUT (S3): The operation used to upload an object to S3; incurs per-request cost and latency, motivating batching. "For BtrLog, S3 PUT costs are negligible because appends are batched before issuing PUT requests: assuming 16\,MB batches and 1\,KB requests, the resulting PUT cost is only $\$\,3 \times 10^{-10}$ per append."
- quorum: A majority (or configured subset) of replicas whose acknowledgments are required to consider an operation committed. "each append sends one request per log node and becomes durable once a quorum (e.g., 2 of 3 nodes) has persisted it, limiting the impact of stragglers on commit latency."
- quorum reads: Read operations that consult multiple replicas and require responses from a quorum to ensure consistency. "read replicas can follow the log tail using quorum reads from multiple log nodes."
- read replica: A non-primary node that consumes the log to maintain a read-only copy of state with low replication lag. "read replicas can follow the log tail using quorum reads from multiple log nodes."
- replication slot (PostgreSQL): A mechanism in PostgreSQL that retains WAL for a subscriber, preventing log truncation until consumption. "the PostgreSQL replication slot concept you suggest is a useful analogy."
- S3 Express: A lower-latency variant of Amazon S3 object storage intended to reduce access times compared to standard S3. "However, placing a transactional WAL directly onto object storage (even its lower-latency variant, S3 Express) is impractical due to high latency and high per-append cost."
- sequencer: A component that assigns a total order to client appends across a distributed log, often adding latency. "They therefore include a sequencer that orders writes across clients, linearizing appends."
- single-writer: A log or stream property where exactly one client issues writes at a time, simplifying ordering and enabling lower-latency protocols. "With a single writer, the order of records is implicitly defined by the writer's append sequence."
- split-brain: A failure mode where multiple writers or leaders believe they are active simultaneously, risking divergence. "BookKeeper therefore implements fencing and recovery to prevent split-brain scenarios during failover."
- staging layer: A low-latency buffering tier (e.g., log nodes) where writes are first persisted before archival to object storage. "a cluster of log nodes that form the staging layer"
- stragglers: Slow replicas or operations that lag behind others, potentially increasing tail or commit latency. "reducing sensitivity to stragglers in commit latency."
- tail latency: The high-percentile latency (e.g., p99), important for assessing worst-case delays in distributed systems. "with explicit attention to tail latency and throughput."
- write-ahead logging (WAL): A durability technique where changes are first recorded in an append-only log before being applied to data structures. "Cloud database systems cannot rely on instance-local disks for write-ahead logging (WAL) durability, forcing WAL onto remote storage."
- write path: The system’s performance-critical sequence of operations to persist new data. "The append (write) path is BtrLog's hot path, optimized for single network round-trip commits."
- ZooKeeper: A distributed coordination service often used for metadata management and leader election. "Using S3 or a compatible service for metadata simplifies deployment and avoids a separate coordination system such as ZooKeeper or etcd."
Collections
Sign up for free to add this paper to one or more collections.
