See all blog posts

Twitter discusses managing stateful systems at ScyllaDB Summit 2017

I was pleased to sit down with Boaz Avital of Twitter to learn more about his upcoming talk at ScyllaDB Summit 2017. At ScyllaDB Summit 2017, participants will join NoSQL developers and users from start-ups and the enterprise for two days of sharing ideas, hearing innovating use cases, and getting real-life tips and tricks from their peers and NoSQL idols. Let’s begin the interview and learn more about what Boaz will be presenting at the Summit.

Please tell us about yourself and what you do at Twitter?

I’m the tech lead of the key value storage team at Twitter. We’ve solved a lot of problems from scratch at a scale that you only see at the largest web companies, so we’ve learned a number of interesting lessons over the years. I primarily work on Manhattan, an in-house distributed key value store that we built to be predictable, operable, multi-tenant, and self-service so that Twitter engineers could easily get storage and build their applications without having to think about infrastructure or waiting to talk to a person.

What will you be talking about at ScyllaDB Summit 2017?

If you’ve ever run a distributed database, you know that managing stateful systems is time consuming and hard. I’ll talk about why that is, the path we took to make Twitter’s Manhattan database easy to run with thousands of nodes and multiple feature sets, and how you should think about operations.

What type of audience will be interested in your talk?

Engineers that have operated or developed any kind of stateful distributed system, especially distributed databases, should see a lot of parallels between their experiences and what Twitter has built.

Can you please tell me more about your talk?

Distributed databases are complex systems. Unlike many other types of services, databases and stateful systems, in general, have many constraints–some obvious and some not so–that must be respected during operations for the system to maintain correctness by not losing or lying about data. An ideal system allows you to operate on it or move data around (e.g. by adding or removing nodes) without sacrificing availability or performance any more than necessary.

Because of these complexities, distributed storage systems are often a huge pain to manage. Manhattan, Twitter’s primary key-value store, operates at a large scale with multiple configurations and capabilities that require careful and subtly different orchestration while managing the cluster, but through a thoughtful and iterative approach to cluster operations, we’ve been able to make it relatively easy and pleasant to run.

This talk will cover some of the reasons why managing stateful systems is hard, including managing availability, data movement and scale, how distributed storage systems generally do it with concrete examples from Twitter’s Manhattan, and the importance of generalized infrastructure for maintaining the sanity of your team members.

Where can we learn more information about your talk?

You can read a bit about the goals and breadth of Manhattan in our original blog post.

How can the people get in touch with you?

Message me on Twitter at @bx!

Thank you very much, Boaz. We can not wait to see your talk in person and learn more. If you want to attend ScyllaDB Summit 2017 and enjoy more talks like this one, please register here.

ScyllaDB Summit is taking place in San Francisco, CA on October 24-25. Check out the current agenda on our web site to learn about the rest of the talks—including technical talks from the ScyllaDB team, the ScyllaDB road map, and a hands-on workshop where you’ll learn how to get the most out of your ScyllaDB cluster.