Worry-free ingestion – flow-control of writes in ScyllaDB

When ingesting large amounts of data into a ScyllaDB cluster, we would like the ingestion to proceed as quickly as possible, but not quicker. We explain how over-eager ingestion could result in a buildup of queues of background writes, possibly to the point of depleting available memory. We then explain how ScyllaDB avoids this risk by automatically slowing down well-behaving applications to the best possible ingestion rate (“flow control”). For applications which cannot be slowed down, ScyllaDB still achieves the highest possible throughput by quickly rejecting excess requests (“admission control”). In this talk, we investigate the different causes of queue buildup during writes, including consistency-level lower than “ALL” and materialized views, and review the mechanisms which ScyllaDB uses to automatically solve this problem.