See all blog posts

Mutant Monitoring Systems (MMS) Day 6 – Multi-datacenter Replication

IMPORTANT: Since the first publication of the Mutant Monitoring System we have made a number of updates to the concepts and code presented in the blog series. You can find the latest version now in ScyllaDB University. Click here to be directed to the new version.


This is part 6 of a series of blog posts that provides a story arc for ScyllaDB Training

In the previous post, we learned how to visualize the data in the Mutant Monitoring System with Apache Zeppelin by viewing the table data and generating graphs. Now, at Division 3 we must prepare for disaster readiness by expanding our ScyllaDB cluster across geographic regions in a multi-datacenter configuration. In this post, we will set up a new ScyllaDB cluster in another datacenter and learn how to convert our existing keyspaces to be stored in both datacenters. Before we get started, we will first need to understand what a multi-datacenter ScyllaDB cluster actually is.

In a multi-datacenter cluster, each ScyllaDB cluster communicates with one another over a public or private network and the data will is copied asynchronously to the other cluster depending on how the keyspaces were defined. For example, let’s say we have one keyspace with a replication DC1:3. This means that the data will only be stored in the first datacenter on three different nodes. If we change the replication factor to “DC1:3, DC2:3”, ScyllaDB will store three copies of the data in each datacenter. Now that we understand the basics of multi-dc, let’s spin up our ScyllaDB cluster and get started.

Starting the ScyllaDB Cluster

The ScyllaDB Cluster should be up and running with the data imported from the previous MMS blog posts. The MMS Git repository has been updated to provide the ability to automatically import the keyspaces and data. If you have the Git repository cloned already, you can simply do a “git pull” in the scylla-code-samples directory.

git clone
cd scylla-code-samples/mms

Modify docker-compose.yml and add the following line under the environment: section of scylla-node1: :


Now the container can be built and run:

docker-compose build
docker-compose up -d

After roughly 60 seconds, the existing MMS data will be automatically imported.

Bringing up the Second Datacenter

The second datacenter will be referred to as DC2 throughout this post. To bring up the second datacenter, simply run the docker-compose utility and reference the docker-compose-dc2.yml file:

docker-compose -f docker-compose-dc2.yml up -d

After about 60 seconds, you should be able to see DC1 and DC2 when running the “nodetool status” command:

docker exec -it mms_scylla-node1_1 nodetool status

Configuring the keyspaces for Multi-DC

Now that both clusters are up and running, we can begin to convert the existing keyspaces to exist on both data centers so if a mutant attacks the first datacenter at Division 3, we will not lose data. The cqlsh utility will allow us to make these changes with the “ALTER KEYSPACE” argument. Let’s get started by logging in the first node and converting our existing keyspaces named catalog and tracking.

docker exec -it mms_scylla-node1_1 cqlsh

ALTER KEYSPACE "catalog" WITH REPLICATION = {'class': 'NetworkTopologyStrategy', 'DC1':3, 'DC2':3};

ALTER KEYSPACE "tracking" WITH REPLICATION = {'class': 'NetworkTopologyStrategy', 'DC1':3, 'DC2':3};

The keyspaces are now ready for multi-dc, but are we done yet? The answer is no because we need to make sure the data is consistent and in sync in each datacenter. We can accomplish data consistency by running the “nodetool rebuild” command on each node in the second datacenter as shown below.

docker exec -it mms_scylla-node4_1 nodetool rebuild -- DC1
docker exec -it mms_scylla-node5_1 nodetool rebuild -- DC1
docker exec -it mms_scylla-node6_1 nodetool rebuild -- DC1

After that command is run, the data from the first datacenter will be streamed to the second and no output should be displayed from the terminal. Let’s make sure that our keyspaces are accessible from the second datacenter with the following commands:
docker exec -it mms_scylla-node4_1 cqlsh

Great, we can see both keyspaces present. We should also query the data to make sure that it is there:

Now that the data exists in both datacenters, we can begin to do failure testing.

Disaster time

In this scenario, we will destroy our first datacenter and pretend that it was attacked by rogue mutants. This can be acheived by using the docker-compose utility to pause the first datacenter with the following command:

docker-compose pause

After bout 10-20 seconds, we can verify that the first datacenter is down by connecting to the second datacenter and running “nodetool status”:

docker exec -it mms_scylla-node4_1 nodetool status

In the output above, we can see that the first datacenter is in a Down/Normal (DN) status and the second datacenter is in an Up/Normal (UN) status. To ensure that the Mutant Monitoring System data still exists, we should verify that the data can be accessed:

docker exec -it mms_scylla-node4_1 cqlsh

In the about above, we can still see the data and feel safe. Going forward, let’s assume that the first datacenter was brought back up and ensure that we can still access the data from it:

docker-compose unpause
docker exec -it mms_scylla-node1_1 nodetool status

We can see in the output above that all of the nodes are online. Let’s verify the data with the following commands:

docker exec -it mms_scylla-node1_1 cqlsh

Looks like we are back to normal!. An important concept to consider when using ScyllaDB with multi-datacenters is consistency levels. If we use a consistency level of ALL, a quorum of nodes in each data center must be available for read or write requests. If the second datacenter failed with a consistency level of ALL, we would not have access to the data. In an entire site failure scenario, consistency levels of LOCAL_ONE and LOCAL_QUORUM will work because quorum is not needed across sites.


In this post, we learned what a multi-datacenter configuration is and expanded the initial Mutant Monitoring ScyllaDB Cluster from one to two datacenters. By having multiple datacenters, Division 3 was able to protect their data from an entire physical site failure. In future posts, we will learn more on this topic when we discuss the different consistency levels available for multi-datacenter ScyllaDB clusters.

Next Steps

  • Learn more about ScyllaDB from our product page.
  • See what our users are saying about ScyllaDB.
  • Download ScyllaDB. Check out our download page to run ScyllaDB on AWS, install it locally in a Virtual Machine, or run it in Docker.