Nadav Har'El, Software Engineer, ScyllaDB
When ingesting large amounts of data into a Scylla cluster, we would like the ingestion to proceed as quickly as possible, but not quicker. We explain how over-eager ingestion could result in a buildup of queues of background writes, possibly to the point of depleting available memory. We then explain how Scylla avoids this risk by automatically slowing down well-behaving applications to the best possible ingestion rate (“flow control”). For applications which cannot be slowed down, Scylla still achieves the highest possible throughput by quickly rejecting excess requests (“admission control”). In this talk, we investigate the different causes of queue buildup during writes, including consistency-level lower than “ALL” and materialized views, and review the mechanisms which Scylla uses to automatically solve this problem.
Glauber Costa, VP Field Engineering, ScyllaDB
Jesse Haber-Kucharsky, Software Engineer, ScyllaDB
Mark Smith, Director of Engineering, Discord
Getting started takes only a few minutes. Scylla has an installer for every major platform. If you get stuck, we’re here to help.