Podcast: How ScyllaDB got to where it is today and what’s next on the company’s remarkable journey
2023 has been a momentous year for ScyllaDB – from the many amazing customer stories and architectural advancements presented at ScyllaDB Summit, to a constant cadence of elasticity-focused innovation, to a record-breaking P99 CONF, and even a book release. ScyllaDB is fast becoming the de-facto standard for data-intensive apps that require predictable performance at scale – and the tech community is taking note.
As this whirlwind year was winding down, CodeStory podcast host Noah Labhart caught up with ScyllaDB CEO and co-founder Dor Laor to chat about how ScyllaDB got to where it is today and what’s next on the company’s remarkable journey.
Here’s the ScyllaDB podcast episode in its entirety:
And for those of you who prefer to read rather than listen, here’s a transcript of some highlights from the conversation.
ScyllaDB’s Origin Story
Noah Labhart: Tell me about the creation story of ScyllaDB.
Dor Laor: My co-founder Avi Kivity and I met back in 2005 at another startup. After that company was acquired by Red Hat, we wanted to start our own company. We actually talked about a database but figured that there was nothing unique to come up with (this was in 2012). So, we decided to do something else: build a new operating system from scratch to compete with Linux for virtualization workloads. Admittedly, this was a crazy endeavor.
When we wanted to demonstrate how applications ran faster on our OS vs Linux, we tried out some databases as well as general applications. Although most other applications ran faster on our OS, Cassandra did not. We learned that Cassandra’s design and architecture are highly scalable, but its Java-based implementation just wasn’t efficient. Java is a great language for many applications, but not for infrastructure.
We recognized the opportunity here, and we pivoted into the database space following more research. Aware that writing a totally new database is insane, we decided to write a new database with an existing familiar interface (the Cassandra interface). [More backstory here and here]
Rethinking Apache Cassandra
Noah: Let’s dive into the MVP. How did you bring it to life?
Dor: Cassandra is written in Java. We wanted to use C++, especially modern C++ (I think it was C++ 12 or so). Nowadays, there’s C++ 23, with lambda functions and so on as well as newer languages such as Rust.
We used advanced C++ and we reimplemented the best parts of Apache Cassandra within our own shard-per-core design. With this approach, you run one thread on one processor; if you have a server with 20 processors, then you run 20 threads with shared-nothing across the board.
The first step wasn’t to create a full database. We started by creating a library that we call Seastar. That name is a play on Cassandra (C* is shorthand for Cassandra). Seastar is a library for simple distributed apps. The first one we wrote was an HTTP daemon that was very basic, but could handle millions of requests per second on a single machine. The next step towards an MVP was coming up with a database (ScyllaDB) that can do millions of operations per second and still be distributed. This is complicated. We had an investor suggest that our MVP should just run on a single machine, but we felt that was just too minimal even for an MVP.
Building the ScyllaDB Team
Noah: How did you go about building your team?
Dor: The team was very experienced with infrastructure engineering. Many of the team members came to us through the KVM hypervisor project and from Linux. I was searching through my Red Hat network as well as other open source connections. Being rooted in those communities attracted some really great people, and they in turn attracted their own communities – so it was essentially a snowball effect. That’s essentially how we built the core technical team.
Nowadays, we have about 160 employees worldwide – about half of that is engineering. Every team strives to meet that high standard that the core engineering team has set since day 1. Now that we’ve grown, we also have really strong and balanced leadership beyond the technological team.
What’s Next for ScyllaDB
Noah: Okay, this will be fun to ask and hear from the founder’s perspective. What’s the future look like for the product and for your team?
Dor: From the very beginning, we wanted to be the number one NoSQL company, even beyond NoSQL– either to relational or other frameworks. Every week, we reject initiatives that might distract us from this goal. We used to say that ScyllaDB can provide the power of Cassandra at the speed of Redis (Redis is a super fast cache). I think that we achieved that part. Now, we’re trying to go after DynamoDB which leads the industry in terms of usability. It’s the most mature SaaS database and we’re trying to make ScyllaDB as usable as DynamoDB.
Now we’re focused on the infrastructure needed to be very elastic. We can send data across many machines quite rapidly. This becomes really important if you have a workload that’s very dynamic – for example, a social media use case that might have a massive spike at any moment – but you don’t want to pay for all of the infrastructure all of the time. We now have this elasticity in the core product, and we’re going to deliver it next year. I hope that this elasticity will help many users with their workloads.
Learn More About What’s Next for ScyllaDB: At ScyllaDB Summit
Want to hear from Dor and the engineering team on what’s next for ScyllaDB? Access ScyllaDB Summit on demand.
ScyllaDB Summit is where the thousands-strong ScyllaDB community gathers to share best practices and learn what’s next for our monstrously fast and scalable database. This year, you can expect to hear from power users such as Discord, Disney/Hulu, Expedia, ShareChat, and Digital Turbine. We’ll cover the latest in cloud infrastructure, event streaming, and the broader distributed data ecosystem. Plus, you’ll be the first to hear about the latest developments toward the extreme elasticity that Dor referenced above.
