Top 5 ScyllaDB Blog Posts from 2016
What does a bug hunt have to do with and I/O Scheduler? Both were top content on the ScyllaDB blog in 2016. Before 2016 fades into the distant past, we’d like to share our most read and shared posts from last year.
Welcome to the asynchronous world
In the networking world, asynchronous processing is so common that it is practically taken for granted. An application sends a message to a remote server, and goes off to do something else. When the response arrives, it is picked up and the application continues from where it left off. There are many application frameworks and even languages that make this easy. In the storage world, the situation is different.
In a datastore like Scylla, there are many actors competing for disk I/O. Examples of such actors are data writers (in Scylla’s parlance they can be either memtable or commitlog writers), and a disk reader fetching the data to serve a cache miss. To illustrate the role that competition plays, if we are just issuing disk I/O without resorting to any fairness or balancing consideration, a reader, for instance, could find itself behind a storm of writes.
In this post we will walk through an investigation of latency spikes observed in the Scylla 1.0.x version family, which are linked to the scylla-jmx service running alongside the Scylla server on the same system. We will show how the scheduler can be tuned to reduce latency spikes. Scylla 1.2, which is at the release candidate stage this week, will have all relevant fixes. Users of that version won’t have to do anything.
ScyllaDB strives to offer its users predictable low latencies. However, in real life, things do not always go according to plan, and sometimes predictable low latencies become unpredictable big latencies. When that happens, it’s time to go into detective mode and go figure out what’s going on.
What is the best request rate I should throw at my cluster? What disk bandwidth should I make available for compactions? How many reader or writer threads should I have? What are the best size for my memtables?
Everybody who has deployed a NoSQL solution has asked themselves these questions. At ScyllaDB, we don’t want to help you answer those questions — we want to get rid of the questions altogether.