This is the second and last part of this article. If you haven’t read the first part, you can do it here. In this part, we will look at the design of the Seastar I/O Scheduler that Scylla uses to manage its disk I/O and discuss how it can be used to not only provide predictable latencies as we saw in our previous installment, but to guarantee fairness and proper balancing among different actors.
In a datastore like Scylla, there are many actors competing for disk I/O. Examples of such actors are data writers (in Scylla’s parlance they can be either memtable or commitlog writers), and a disk reader fetching the data to serve a cache miss. To illustrate the role that competition plays, if we are just issuing disk I/O without resorting to any fairness or balancing consideration, a reader, for instance, could find itself behind a storm of writes. By the time it has the opportunity to run, all that wait would have translated into increased latency for the read.
This is the fifth part in our blog series on Scylla testing. Part 1 covers Apache Cassandra compatibilty testing, Part 2 covers Jepsen tests, Part 3 covers the CharybdeFS fault-injecting filesystem, and Part 4 covers distributed testing.
This is the fourth part in our blog series on Scylla testing. Part 1 covers Apache Cassandra compatibility testing, Part 2 covers Jepsen tests, and Part 3 covers the CharybdeFS fault-injecting filesystem.
Full disclosure: I’m a casual user of Ansible, Python and Bash. I’m not an expert on any of them. The notes below are my personal impression after playing around with Ansible for a few months, mostly to run Scylla and Cassandra clusters on EC2.
This is the third part in our blog series on Scylla testing. Part 1 covers Apache Cassandra compatibility testing, and Part 2 covers Jepsen tests.
Distributed systems are hard to test, which means that creating a solid set of tests for a distributed database is a substantial software project in itself. Software developer Kyle Kingsbury has invented a full-featured tool for testing distributed systems, called Jepsen. (It’s named after Carly Rae Jepsen.) Jepsen is a flexible tool that can be set up to test a variety of distributed systems, including Apache Cassandra. Jepsen and Cassandra have both made a lot of progress since the original set of tests. Joel Knighton, a developer working at DataStax, has updated and enhanced the Jepsen tests for Cassandra and […]
Welcome to the asynchronous world
This series of blog posts will cover how we test Scylla—first of all for stability and correctness, and also to ensure that Scylla keeps Apache Cassandra compatibility constantly as the project proceeds. Software testing is critical, and especially so for a project that needs to achieve compatibility with some previously existing software. This blog series will cover the tests that we put Scylla through.
You can visualize Scylla performance with Tessera and riemann-dash, thanks to collectd support. Scylla can be configured to send data using the collectd protocol, once per second, which makes it easy to keep an eye on the performance of a whole cluster.