Assessing Log-Based Coordination Systems for Managed Cloud Environments

Abstract

The distributed systems landscape is undergoing a significant shift toward managed cloud environments, reducing the prevalence of self-hosted coordination services such as ZooKeeper. While ZooKeeper remains a proven and feature-rich solution for coordination tasks, its deployment in cloud environments can introduce component redundancy. This is because the underlying cloud platform already provides internal mechanisms to ensure coordination guarantees. This thesis investigates the design and evaluates the performance of a log-based coordination service library tailored for managed cloud environments. The proposed library removes the ensemble management overhead inherent in ZooKeeper by delegating durability and consistency responsibilities to the cloud provider’s data layer. This architectural simplification enables a modular design, allowing for tailored implementations that exploit the strengths and mitigate the limitations of a system's specified data layer. The library demonstrated feature parity with ZooKeeper for a targeted subset of coordination features, including leader election, membership tracking, and ephemeral state management. The same is noted for migration from an existing ZooKeeper-based application to this work's library, requiring minimal design changes while preserving coordination guarantees. While the results show that this design does not yet match mature coordination services in raw performance, they highlight potential avenues for further research, particularly in optimizing log-based coordination systems for the unique characteristics of cloud-managed data layers. Given the industry’s steady movement toward cloud-native infrastructure, these findings provide a foundation for future exploration into lightweight, platform-integrated coordination solutions.