ScyllaDB recently announced ScyllaDB V: the latest evolution of our monstrously fast and scalable NoSQL database.
To explore what’s new and how the latest innovations impact application development teams, DevOps luminary Alan Shimel recently connected with ScyllaDB VP of Product, Tzach Livyatan for a TechStrongTV interview. Alan and Tzach chatted about NoSQL performance and security, the shift to database-as-a-service (DBaaS), the performance impact of Kubernetes, new AWS instances, observability, and more.
Watch the interview in its entirety:
If you prefer to read rather than watch, here’s a transcript (lightly edited for brevity and clarity) of the core conversation.
Intro
Alan: Welcome to another TechStrongTV segment here. Our guest today is Tzach Livyatan. He’s a first-time guest on TechstrongTV, but we’ve covered the company (ScyllaDB) before. Tzach, welcome to TechStrong.
Tzach: Thanks! Thanks for having me, and thanks for pronouncing my name correctly – I appreciate it.
Alan: You know, I work at it. Tzach, a lot of folks out here maybe they’ve heard of ScyllaDB, maybe they haven’t. Maybe they’re not quite sure what you guys do. Why don’t we start with that? Give them a little company background and maybe a little bit of your own personal background.
Tzach: Okay, sure. Let me start with myself. I have a background in Computer Science and have been working in the industry for many, many years – I try not to count anymore. I started as a developer, then moved to product management. I spent many years in the telecom domain in a startup, and later at Oracle’s communication unit. After that, I switched to the NoSQL domain at ScyllaDB.
ScyllaDB’s Sweet Spot in the NoSQL / Distributed Databases World
Tzach: ScyllaDB is a NoSQL database. There are many, many no SQL databases out there. Each of them has a special unique sweet spot. ScyllaDB falls roughly into the wide column databases category. You might have heard of databases like DynamoDB or Apache Cassandra; we address many of the same use cases as they do.
ScyllaDB is focused on high availability. It can stretch from a 3-node cluster to a 100-node cluster – although you don’t need as many nodes as you would with other databases. Let me give you an example of its high availability. One of our customers has a three region cluster in three different regions, and one of their regions actually burned down in a fire – a well-known cloud provider’s data center completely burned down. But, the ScyllaDB deployment just continued to work as is. You have a deployment of 10 nodes in each region, so 20 nodes continue to work without a glitch when one region’s nodes burned down. This is ScyllaDB’s high availability in action.
NoSQL now has a long history, 15 years or more. The first generation NoSQL database was very much focused on high availability…but performance was an afterthought on some of them. I don’t want to trash other databases because all databases made progress. But a lot of them were initially designed as a project and not as a product. Performance, security, and other administrative/operational capabilities were not highly prioritized back then.
You can think about ScyllaDB as a second generation of NoSQL database that was designed for performance, security, and administration from the ground up. So ScyllaDB was the reinvention, if you will, of Apache Cassandra. We wanted to keep the emphasis on high availability, but we re-invented and re-designed everything from scratch in C++ for performance.
Think of performance in terms of two main metrics: throughput, and latency. Each of them can be optimized almost independently. So why is throughput important? If you can, for example, give 10X the throughput of another database, which we do, you can reduce the number of servers by a factor of 10, and you save on administration – it’s much easier to administrate. Of course, it’s also less expensive. If you’re working on AWS and another cloud, just shrinking the number of servers by 10X is a great improvement.
The second parameter is latency. This is a little bit more tricky because it’s harder to improve. While you can always spend more money and add more nodes or servers to handle your throughput problem, you cannot do the same for latency because latency is very much coupled with architecture.
ScyllaDB was designed from scratch with an architectural approach called shard-per-core. I won’t go too deep into the technical parts, but it’s designed from scratch for good performance and low latency. This is how we are able to provide less than one millisecond on average and less than five milliseconds on P99 regardless of the throughput. You can throw up to a million requests per second, which is a lot, on one ScyllaDB node and still get that sub-millisecond latency. This is very significant. If you want, I can explain why it’s significant, but this is a great improvement over the first generation of NoSQL databases that I mentioned earlier.
The Importance of Throughput and Latency at Scale
Alan: Our audience might not be familiar with ScyllaDB, but they’re very technical. I don’t think we need to explain to anyone how important both throughput and latency are. Today, more than ever, it’s about scale. Companies must scale up given the current market conditions. It’s scale up economically or die, right?
Apache Cassandra is an amazing product, an amazing project. Of course, there have been several companies that have tried to monetize it and productize it. And there’s a couple of models, right? There’s the hosted model. There’s the support and training – you know, the usual open source kind of stuff. And then there’s sort of an open core model where you have freemium functionality. ScyllaDB’s special sauce is obviously around reducing latency and increasing throughput. How does ScyllaDB go to market?
ScyllaDB Product Options: DBaaS, Enterprise, Open Source
Tzach: We offer three options. We have ScyllaDB open source. You can go to GitHub, get the code, and compile it yourself. We have many many open source users. That’s definitely an offering that we are supporting and even encouraging. We encourage people in the community to use ScyllaDB and contribute to it.
We also have an enterprise version, which is the closed source version closely following the open source. About 95% of the code is the same. However, it offers some unique features for ScyllaDB Enterprise – a lot of them are around security. For example, let me give you a quick example: encryption at rest. Encryption at rest is a feature that we are offering exclusively in ScyllaDB Enterprise. There are other enterprise features like that, for performance, security and so on.
We also offer ScyllaDB as a service: ScyllaDB Cloud. This is a fully managed ScyllaDB Enterprise version. It’s the same enterprise database, but we manage it for you – either on AWS or GCP. You consume the database, and we take care of all the monitoring, upgrades, security patches, whatever is involved in managing the database. We do it all for you.
In the last year, it’s become more and more popular, and ScyllaDB Cloud database-as-a-service (DBaaS) is now the most popular out of those three options. ScyllaDB Cloud seems to be more and more popular. I’m sure that’s not surprising to you; it’s just following the industry trend that everyone wants to move to the cloud.
Alan: Look, we live in a “SaaS-y” world, right? If you offer it as a SaaS, people like to consume it like that. They pay as they go, there’s not that big upfront cost, and they don’t have to deal with upgrades and patching and all that good stuff.
What’s New in ScyllaDB’s Latest Release
Alan: All right. I think we’ve done a good job laying all this out now Tzach. Tell us what’s new.
Tzach: We just released ScyllaDB Open Source 5.0, which is our yearly major release and the first milestone in our ScyllaDB V series of releases. We’re very excited about it.
If I can squeeze the history of ScyllaDB (which is almost eight years now) into three phases, the first phase was about performance and the architecture for performance, which I mentioned earlier; the second was about complete functionality, and we are now completely compatible with both Apache Cassandra and Amazon DynamoDB; and the third phase, which we are starting now, is about innovation and going beyond compatibility, introducing new features.
Let me talk about two of the most exciting features.
The first is that we’re introducing a new consensus algorithm with Raft. ScyllaDB basically is an eventual consistency database. But there are two parts of consensus or consistency for any database. One is for the data itself, and one is for the metadata. What is metadata? The user, the node, the node in the cluster, the schema — everything which is not the data itself. And so far, this data was propagated in the cluster through Gossip, which is eventual consistency. That was fine a few years back, but today it’s not fast enough or not consistent enough.
With ScyllaDB 5.0 we’re introducing a new consensus algorithm for that. From the user perspective, this means that you can safely update both the schema and the topology of the cluster in a safe way, whenever you want. What do I mean by “change topology?” For example, add nodes. Let’s say that you want to scale your cluster from 3 nodes to 10 nodes. You can now do it in a safe way, and very fast. This serves not only ScyllaDB Open Source and Enterprise, but also our own cloud offering (ScyllaDB Cloud), which can now scale much faster and even shrink faster if you want to.
All of the new ScyllaDB V features, including the performance gain, also apply to Kubernetes, by the way. We have a strong offering on Kubernetes. Everyone is running on Kubernetes these days; it’s not new. We made a lot of effort to make sure that ScyllaDB’s performance on Kubernetes is very close to how it performs running on bare metal and VM. This is not trivial. But, I’m happy to say that we achieved it and we are very happy with that.
Alan: That’s huge
Tzach: The second aspect is performance. Although we are already the fastest NoSQL out there, we keep pushing the limits of performance. I’m happy to say that we achieved a lot of improvement in this release. In particular, we worked with AWS to optimize ScyllaDB for the new I4i instances. I4i is the successor of the i3 and i2 families, which are instances that AWS offers for databases. We got a particularly surprising result on the I4i instances: more than 2X performance compared to the i3. So, for the same number of cores, you get more than 2.5X the throughput and even lower latency. That was a combination of the great hardware that AWS offers (both the storage and network and the CPU) and ScyllaDB being designed from scratch to work with these types of high-end instances.
To conclude, we improved safety and elasticity with Raft and continue to push on performance, especially with the I4i. We introduce new features first on open source, let it stabilize a little bit then we backport to the enterprise version, then backport that to ScyllaDB Cloud. We are now in the first phase of it; in a few weeks, it will be introduced to ScyllaDB Enterprise and ScyllaDB Cloud.
Editor’s note: This Enterprise release has occurred since the interview – see what’s new in ScyllaDB Enterprise. To hear more, including the discussions on Kubernetes and observability, watch the complete interview above.
