Mutating etcd Towards Edge Suitability

Abstract: In the edge environment servers are no longer being co-located away from clients, instead they are being co-located with clients away from other servers, focusing on reliable and performant operation. Orchestration platforms, such as Kubernetes, are a key system being transitioned to the edge but they remain unsuited to the environment, stemming primarily from their critical key-value stores. In this work we derive requirements from the edge environment showing that, fundamentally, the design of distributed key-value datastores, such as etcd, is unsuited to meet them. Using these requirements, we explore the design space for distributed key-value datastores and implement two successive mutations of etcd for different points: mergeable-etcd and dismerge, trading linearizability for causal consistency based on CRDTs. mergeable-etcd retains the linear revision history but encounters inherent shortcomings, whilst dismerge embraces the causal model. Both stores are local-first, maintaining reliable performance under network partitions and variability, drastically surpassing etcd's performance, whilst maintaining competitive performance in reliable settings.