Scylla Testing

On a 24×7 basis we hit Scylla with a battery of tests. Many tests were adopted from Apache Cassandra and third parties. Many of the tests were developed exclusively for clearing all possible issues, from correctness, to data-loss to distributed behavior. Here are some highlights from our Scylla testing blog series:

Apache Cassandra compatibility testing

Scylla is an Apache Cassandra compatible database. Thus, besides Scylla’s own tests, the project must pass the relevant Apache Cassandra test suite. This ensures that Scylla will handle the CQL query language exactly as Apache Cassandra does. Learn about Scylla/Apache Cassandra compatibility testing

Jepsen

The latest hit on the distributed software testing scene is Kyle Kingsbury’s Jepsen. A flexible tool that causes hard-to-handle network outages, Jepsen partitions and delays for a variety of distributed systems, including Apache Cassandra, and now Scylla. Learn more

CharybdeFS

Testing for proper handling of filesystem errors is critical. We have developed a new emulated filesystem that can inject any error for any operation, under the control of a test script. CharybdeFS makes rare error conditions into repeatable test cases. Learn how it works

Distributed tests

Scylla distributed tests are an extended version of the dtest project originally developed for Apache Cassandra. Apache Cassandra dtests are functional black boxes that test Apache Cassandra cluster operation. By providing a test suite covering how a Apache Cassandra server should operate, dtest allows us to validate that Scylla is compatible in functionality with Apache Cassandra. Learn more

Longevity testing

Longevity tests create a Scylla cluster in AWS and exercise it by running cassandra-stress and other clients specified in the test scripts. This testing finds problems in the operation of long-running clusters, as is the case with existing Scylla deployments. The system uses Boto for controlling Amazon AWS from Python. Learn more