This series of blog posts will cover how we test ScyllaDB—first of all for stability and correctness, and also to ensure that ScyllaDB keeps Apache Cassandra compatibility constantly as the project proceeds. Software testing is critical, and especially so for a project that needs to achieve compatibility with some previously existing software. This blog series will cover the tests that we put ScyllaDB through.
ScyllaDB must pass three categories of tests: ScyllaDB native tests, Apache Cassandra tests, and third-party tests. First, of course are the project’s own built-in tests.
- ScyllaDB project unit tests.
- Seastar framework unit tests.
- Functional tests for ScyllaDB software packages, using the Avocado framework. These tests cover the final artifacts: the ScyllaDB RPM and deb packages, and the ScyllaDB AMI.
- Longevity testing on a long-running ScyllaDB cluster on Amazon Web Services
But, of course, that’s not all. ScyllaDB must be compatible with Apache Cassandra. So, besides ScyllaDB’s own tests, the project must pass the Apache Cassandra test suite, including both unit tests and distributed tests. Tests for the CQL query language are part of the Apache Cassandra unit tests.
Example of a unit test
|public class TimeuuidTest extends CQLTester|
|* Migrated from cql_tests.py:TestCQL.timeuuid_test()|
|public void testTimeuuid() throws Throwable|
|createTable(“CREATE TABLE %s (k int, t timeuuid, PRIMARY KEY(k, t))“);|
|assertInvalidSyntaxMessage(null, “INSERT INTO %s (k, t) VALUES (0, 2012-11-07 18:18:22-0800)“);|
|for (int i = 0; i < 4; i++)|
|execute(“INSERT INTO %s (k, t) VALUES (0, now())“);|
|Object rows = getRows(execute(“SELECT * FROM %s“));|
|assertRowCount(execute(“SELECT * FROM %s WHERE k = 0 AND t >= ?“, rows), 4);|
|assertEmpty(execute(“SELECT * FROM %s WHERE k = 0 AND t < ?“, rows));|
|assertRowCount(execute(“SELECT * FROM %s WHERE k = 0 AND t > ? AND t <= ?“, rows, rows), 2);|
|assertRowCount(execute(“SELECT * FROM %s WHERE k = 0 AND t = ?“, rows), 1);|
|assertInvalid(“SELECT dateOf(k) FROM %s WHERE k = 0 AND t = ?“, rows);|
|for (int i = 0; i < 4; i++)|
|long timestamp = UUIDs.unixTimestamp((UUID) rows[i]);|
|assertRows(execute(“SELECT dateOf(t), unixTimestampOf(t) FROM %s WHERE k = 0 AND t = ?“, rows[i]),|
|row(new Date(timestamp), timestamp));|
|assertEmpty(execute(“SELECT t FROM %s WHERE k = 0 AND t > maxTimeuuid(1234567) AND t < minTimeuuid(‘2012-11-07 18:18:22-0800’)“));|
|* Test for 5386,|
|* migrated from cql_tests.py:TestCQL.function_and_reverse_type_test()|
|public void testDescClusteringOnTimeuuid() throws Throwable|
|createTable(“CREATE TABLE %s (k int, c timeuuid, v int, PRIMARY KEY (k, c)) WITH CLUSTERING ORDER BY (c DESC)“);|
|execute(“INSERT INTO %s (k, c, v) VALUES (0, now(), 0)“);|
The above code is an example of an Cassandra unit test (source). It performs the following steps.
- Create a table with two columns:
- Insert n (n=4) values in it with the now() function, which is guaranteed to return unique results.
- Verify if there are indeed 4 results in the table. If now() returns a unique UUID, there will be.
- Perform queries that verify that the timeuuid values can be compared appropriately.
- Verify that the functions dateOf and unixTimestampOf are returning correct values by comparison with what java.utils.Date and java.util.UUID.unixTimestamp return for the same strings.
Apache Cassandra developers have written hundreds of such tests, which means that the ScyllaDB project is starting with a solid foundation for QA. However, the Apache Cassandra tests require quite a bit of setup. The cassandra distributed tests, or
dtest suite, is more of a functional cluster test than actual unit testing, it just so happens to be written using python unittest classes. It’s a common trend in Python testing that most types of test classes are derived from the unittest base class.
In order to get the Apache unit tests running on ScyllaDB, you need to start a ScyllaDB cluster with
ccm, the Cassandra Cluster Manager.
ccm create scylla-3 --scylla --vnodes -n 3 --install-dir=[your-install-dir] ccm start
The ScyllaDB project has a fork of ccm that enables starting ScyllaDB. We’re in the process of contributing this extra functionality to the ccm project, and will post to the scylladb-dev mailing list and the upstream project list. We’ll be ready to share this repo with the rest of the world soon, when we have worked out some remaining issues with Git history. We’re still working out if we want to generate a single clean patch to submit, or keep some messy history.
The Cassandra unit tests are a mix of two categories: some depend on the correct behavior of the software, and some that are tied to the internals of the original implementation. Anything that directly depends on the original Java implementation is not useful for ScyllaDB testing purposes. The tests that don’t depend on internals, including CQL unit tests, are in ScyllaDB’s
cassandra-unit-tests repository. So far, it’s been good for us to show how compliant and solid our support for CQL is.
When you have cloned the test repo, execute the tests by doing:
cd cassandra-unit-tests mvn test
What’s the test failure?
mvn test command above will show one failed test. What’s the bug?
The current development version of ScyllaDB does not always return a unique timestamp. This is a known ScyllaDB bug and we’re working on it right now.
ScyllaDB has a Jenkins setup with lots of jobs that run on a daily or per merge basis. We execute all the tests covered above using Jenkins, and as we develop more features, everything is continually tested. Constantly running and expanding the test suite is a top priority. If you’re interested in joining us, and high software quality is your thing, check out the QA automation developer listing on our jobs page.
Next: Jepsen tests
In the next article in this series, we’ll cover the Jepsen distributed testing tool, and how we’re extending Jepsen’s Cassandra testing support to cover ScyllaDB. Follow @ScyllaDB on Twitter or subscribe to this site’s RSS feed for more testing info, release announcements, and other project news.
Apache®, Apache Cassandra®, are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.