Introducing Project Circe: Making the Best NoSQL Database Better

Learn how we’ll transform Scylla into an even more monstrously powerful NoSQL database

In Greek mythology Scylla didn’t start out as a monster at all. Only when she reached adulthood was she transformed (by Circe) into the most powerful beast in the sea. With Project Circe, we look to transform our now mature database into an even more powerful entity.

Scylla is already the most powerful NoSQL database. It’s fast, cost-effective, supports two widely adopted protocols (CQL and the DynamoDB API), has indexes, views and many deployment options. It supports tens of millions of users and an impressive number of mission-critical services that we all rely on.

However, we are not satisfied until Scylla is widely considered the best NoSQL database.

Project Circe is a one-year effort to improve NoSQL performance across important aspects of Scylla in order to bring greater consistency, elasticity and ease of use to our NoSQL database. It includes twelve distinct milestones – one per month (or, one for each of Scylla’s powerful tentacles).

Key NoSQL Database Improvements from Project Circe

Raft

Raft Maintainability Elasticity Performance Stability Deployment

The Raft Consensus Protocol

Consensus in a distributed system is complicated but important. Scylla gained Lightweight Transactions (LWT) through Paxos but these transactions require three roundtrips. Raft allows us to execute consistent transactions without a performance penalty. Unlike Paxos, which we use only for LWT, we plan to integrate Raft with most aspects of Scylla, making a leap forward in manageability and consistency.

Transactional Schema Changes — Our first user-visible value eliminates schema conflicts and allows full automation of DDL changes under any condition.
Transactional Topology Changes — Our next user-visible change will permit adding or removing any number of nodes simultaneously. Currently, Scylla and Cassandra can only scale one node at a time. This means it can take long hours to double or triple the whole cluster’s capacity. Obviously, this is not the elasticity you’d expect if you have bursty intraday traffic.
Tablets — Once range ownership becomes transactional, it will allow many levels of freedom. We plan to improve more aspects of range movements, towards tablets and dynamic range splitting for load balancing.
Strong Consistency — Beyond crucial operational advantages, end users will be able to use Raft, gaining strong transaction consistency at the price of a regular operation!
Dynamic Tablets — Static Tablets allow us to scale multiple nodes at the same time. Reusing the tablet concept allows us to split ranges dynamically in order to load balance shards and to support unbalanced data models
More to Come — We have many more plans for Raft: Materialized views can be made consistent with their base under all scenarios, repairs can be faster and cheaper and so forth.

Raft will become the default option for data manipulation. We will preserve regular operations as well as the Paxos protocol. As of December 2020, the core Raft protocol is implemented in Scylla.

Maintainability

Scylla supports smooth maintenance operations with its built-in IO and CPU schedulers and automatic tuning. But we can do more on our journey to provide the best NoSQL database.

Seedless NoSQL — This year we’d like to simplify your deployments by removing seed nodes, which break the homogeneity of nodes concept, allowing you to restart long-running operations like repair, backup, and even scale-out.
Off-Strategy Compaction — Scylla’s current compaction backlog controllers automatically set the dynamic priority for the compaction queue and the user doesn’t need to do anything but choose the compaction algorithm (such as Scylla’s unique Incremental Compaction Strategy). Our new off-strategy compaction will automatically reshape your data to when it is not aligned with the compaction strategy – for example, after migration or streaming. Off strategy compaction reduces the write amplification and makes it easy to have large data manipulations. In addition, we will expand the ability to load SSTables on any node, regardless of the key range ownership. The node will chauffeur the data to the proper replica set.
Improved Repair — More changes are coming to repair, starting from an IO bandwidth limiter for poor IO environments, to a new IO scheduler and maintenance window.
Repair-based Node Operations (RBNO) — Adding or decommissioning a Scylla node, which typically stores up to 60TB of data, can take several hours. Before RBNO if there was an AWS node failure, the whole process needs to be restarted from scratch. When RBNO is enabled, the restart is immediate and the operation will resume from the last position. Since we already use repair for streaming, the process is also simpler and saves the additional repair execution time.

Elasticity

Scylla was modeled after Cassandra, which was designed before the cloud-native era. When cluster manipulation is complex or problematic, database admins refrain from changing the cluster size often. Yet with more workloads running as-a-service (-aaS), and when spiky Internet-scale traffic can hit your site at any time, a modern distributed database needs to be as elastic as possible.

Topology Changes — One of the basic elasticity improvements we’re now finalizing is the ability to add, remove and replace nodes with full consistency and greater simplicity. This is important for all deployments, and especially for a database-as-a-service (NoSQL DBaaS) such as Scylla Cloud.
Faster Scale-out — By adopting Raft, we’re making key range changes transactional. This allows us to detach the range movement from the actual data movement, which will be asynchronous. So you will be able to add or remove multiple nodes at once. This is a significant improvement over Cassandra/Scylla today as only a single node at a time can be added. Furthermore, key ranges will be represented as tablets, which will automatically get split or united as a function of the load of the shard.
Reduced Stalls — Scylla’s basic execution unit is 500µsec. We define a stall when the computation takes more than this amount. No stalls exist during regular operation, however during administrative operations, stalls can be observed when we repair or stream. We’re about to eliminate all stalls to allow smooth, low-latency execution under any condition.

Performance

NoSQL database performance, especially price/performance, is one of the fundamental reasons we created and continue to build Scylla. This year our performance optimizations will emphasize several key improvements:

Uninterrupted Performance — Under any administration operation, from elasticity to availability. Using our stall detector, our scheduler can locate stalls as small as 500µsec. We’re eliminating them for smoother continuous operations.
Large Partition Performance Improvements — Our goal is to make large partitions perform close to small partitions, with almost no overhead for the clustering key scanning. Better NoSQL performance frees developers from working around their data model and provides consistent performance without any hiccups. We recently implemented a binary search through the SSTable index file, which boosts the speed of reading rows within a large partition. The final step is to cache the index file itself, so we wouldn’t go to read it from the disk.
New IO Scheduler — Controlled IO is one of the key principles of Scylla’s Seastar engine. Until now, all shards in the system divided the global capacity of available storage and scheduled shard IO within their budget. This design worked well for very fast disks (e.g., NVMe SSDs), but slower disks or workloads with extreme latency requirements mixed with large IO suffered – LWT for example. The new IO scheduler has a new algorithm with a global view and a credit-based system, which should improve latency across the board, especially with slower storage options such as HDDs or shared cloud block storage.
Coroutines — Coroutines are a C++ programming abstraction, recently introduced to Scylla with C++20 and the CLANG compiler. Without diving deeply, Coroutines allow Scylla to replace some of the future-promise asynchronous code which extensively allocates memory with larger bulks when it’s appropriate.

Stability

Many Project Circe improvements can be considered stability improvements. For example, making schema changes transactional eliminates a whole family of schema disagreement issues. Same for the elimination of seed nodes. There are, however, many directly related stability improvements.

Memory Tracking — We’re improving the tracking of memory used by queries in order to block/slow-down new queries when memory is limited. Drafts for cancellable IO were published. Thus, when a node becomes suddenly overloaded, we can cancel pending IO and provide an immediate cure.
Tombstones — There is ongoing work around tombstones and range tombstones.
Safe Mode — A new mode of operation that reinforces better defaults and reduces the chance for errors.
Workload Shedding — These are a family of algorithms we developed to prevent overloaded situations, especially when parallelism is unbounded. They’re useful when existing mechanisms to slow down clients via backpressure are not effective.

Deployment Options

Our goal is to make Scylla easily available and deployable wherever and however our users wish.

Scylla Cloud — Scylla Cloud is already available on AWS and AWS Outposts. Starting in January, Scylla Cloud will also run on GCP and soon thereafter on Microsoft Azure. Expect us to support many more instances and many more features.
Machine Images — We will also add standard machine images for GCP and Azure and add marketplace options for our Scylla Cloud DBaaS, Scylla Enterprise and Scylla Open Source deployment options.
Ansible/Terraform — An open source Ansible project exists and we’re adding provisioning options to it.
Kubernetes Operator — Kubernetes (K8s) has become the de facto method for managing systems in a cloud-native environment. In January 2021 we’re announcing our Kubernetes Operator as a fully supported operations management tool. Expect it to work well on all K8s environments, starting with Google Kubernetes Engine (GKE).

But Wait…There’s More to Creating the Best NoSQL Database!

There are many more features and improvements we’re excited about. Change Data Capture (CDC) is going GA together with supportive libraries. We’re going to improve observability. We will implement additional protocols, traceability and much more. Look for announcements at our Scylla Summit and updates to this project page.