See all blog posts

What’s New In ScyllaDB Monitoring Stack 2.4

ScyllaDB Monitoring Stack Release Notes

We just released ScyllaDB Monitoring Stack version 2.4. The new version comes with dashboards to support the coming ScyllaDB Open Source 3.1 release.

New Ways To Look at Shards Information

ScyllaDB uses a thread-per-core architecture. This means that each thread-core (called a shard) acts as an individual processing unit.

Almost all metrics are reported at the shard level. When looking at ScyllaDB’s dashboards you can choose between Cluster, Data Center, Instance or Shard view.

Consider the requests served panel found on the overview-dashboard. When looking at it in the instance view all the information from the shards on each of the instances are summed, like in this example:

ScyllaDB Monitor: all shards visualization

The behavior when switching to shard view is now changed, starting in version 2.4. When switching to shard view you would see the information from all shards from all instances. For example:

ScyllaDB Monitoring Stack: per shard view

As you can this can become quite crowded as the number of lines in a panel is the number of shards multiplied by the number of instances.

So you usually would like to choose a specific instance (or a few instances).

To help you choose the instance, an additional enhancement allows you to choose the aggregation function.
Think about a situation when you look at a graph at a shard granularity, you can see that from all the shards only one or two are very high or very low.

Switch back to instance view and change the aggregation function to something else (i.e. per minute), this will help you to see what instance the misbehaving shard belongs to.

New Dashboard

ScyllaDB Errors

In the lifetime of the system, there are situations where things are not working as expected. Examples are system overload, network problem or disk errors.

To help our support and field engineers get a view of what is not working as expected we added the error dashboard, which simply put, shows different metrics indicating that some error occurred in the system.

ScyllaDB Monitoring Stack 2.4: new dashboard

Changes in existing Dashboards

Replica vs Coordinator

ScyllaDB’s data model uses replication for availability and redundancy. This means that when a node gets a request (e.g. insert) it would send that request to multiple instances.

The node that gets the original request is called a Coordinator because it coordinates the work between the other nodes.

The nodes that hold the replicated data and perform the actual action are called Replicas, as you probably guessed.

It’s sometimes confusing to understand if a metric comes from a coordinator or from a replica and the difference is important, for example, assume we are using consistency level 3 when we’ll do a write, for every insert-operation at the coordinator level we are going to see 3 inserts on the Replication level. To make it clearer all panels are now labeled if they come from the replicas or from the coordinator.

The Nodes Table

We added a new table in the overview dashboard for all the nodes, for each node we have the datacenter it belongs to, its IP, its current version and its status.

The node status information is based on a metric that is only available in ScyllaDB 3.1 and later and ScyllaDB Enterprise 3.1 and later.

ScyllaDB Monitoring Stack 2.4: nodes table

Next steps

Now you’ve seen the changes that were made in ScyllaDB Monitoring Stack 2.4 to make it even clearer. The next step is yours! Download ScyllaDB Monitoring Stack 2.4 directly from Github. It’s free and open source. If you try it, we’d love to hear your feedback, either by contacting us privately or sharing your experience with your fellow users on our Slack channel.

About Amnon Heiman

Amnon has 15 years of experience in software development of large scale systems. Previously he worked at Convergin, which was acquired by Oracle. Amnon holds a BA and MSc in Computer Science from the Technion-Machon Technologi Le' Israel and an MBA from Tel Aviv University.