Sep27

Scylla Summit Sneak Peek: Consensus in Eventually Consistent Databases

Subscribe to Our Blog

In the run-up to Scylla Summit 2018, we’ll be featuring our speakers and providing sneak peeks at their presentations. In the expert track, ScyllaDB Software Engineer Duarte Nunes will host a talk Consensus in Eventually Consistent Databases.

Duarte is based in sunny Lisbon, where he has recently taken up surfing. When not dealing with cranky compilers or inscrutable race conditions, he enjoys good literature and landscape photography. A true epicurean, there are few things he appreciates more than a bowl of Count Chocula.

What was your evolution as a developer? How did you end up working on this project at ScyllaDB?

 

I’ve always been attracted to hard, system-level problems. Concurrent programming, lock-free algorithms in particular, were my gateway topic to distributed systems. I came to ScyllaDB to work on their intersection with high-performance, low-latency software, typically found in the database space. I can’t think of a more fun and challenging project to work on than endowing Scylla with strong consistency and its associated features.

Issue #1359 was opened back in June of 2016. What’s taking so long?

 

That we just started tackling LWT is a consequence of the way we ended up allocating a limited amount of engineering resources. For instance, I was previously working on Materialized Views, which we decided to prioritize. General availability of LWT is bound to still be some months away, as implementing a consensus infrastructure is a non-trivial undertaking. It requires a great deal of thought and design, a meticulous implementation, and very extensive testing. If we learned anything from Materialized Views, it’s that it’s one thing to implement a complex distributed algorithm, and another to ensure it behaves correctly (all corner cases are addressed, as are the interactions with other system properties, like range movements), performs adequately for our standards (use worthwhile tricks and optimizations) and doesn’t negatively affect the cluster (proper back-pressure, load shedding, etc.).

 

There are a lot of distributed coordination and consensus algorithms now. Paxos. Raft. Zookeeper Atomic Broadcast (ZAB). CASPaxos. Which ones are you looking at and why?

 

All of them! These are not radically different algorithms. They occupy the same design space and differ in the trade-offs they choose to make, all of which we need to consider: understandability, number of RTTs, strong vs weak leadership, quorum size, etc.

 

What specifics will you cover in your talk?

 

The talk will go over these tradeoffs and what we decided to implement in Scylla. We will use LWT as a driver for some of these decisions, but we will also cover other use cases, both external — like Materialized View updates — and internal — such as schema changes, membership, and range movements — that benefit from strong consistency, in order to improve their reliability and safety. We will also look further into the future and consider what’s required to support distributed transactions with useful consistency levels.

 

What should prospective attendees already be familiar with to get the most out of your session?

 

No research is required, but it certainly helps if attendees are familiar with distributed systems, eventually consistent databases in particular.

 

If folks haven’t registered for Scylla Summit yet, is there a discount code they can use?

 

Yes! If they want to get 25% off the current admission costs, they can use the code DUARTE25.


Tags: CASPaxos, concurrency, Consensus, eventual consistency, Lightweight Transactions, LWT, Paxos, scylla, Scylla Summit, Scylla Summit 2018, ScyllaDB, Zookeeper, Zookeeper Atomic Broadcast