Mar22

ScyllaTop: real-time monitoring for Scylla in the terminal

ScyllaTop is a simple diagnostic tool that enables you to log on to a machine running Scylla, and view its current state, including many helpful metrics. ScyllaTop is a small project, just a single text-based program with no dependencies other than packages present in the standard Linux distribution. It makes it possible to easily monitor your Scylla node without having to set up multiple Docker containers for monitoring servers.

Scylla reports these metrics to collectd, a data gathering service, and ScyllaTop then connects to collectd and presents its metrics to the user. Because ScyllaTop is a general-purpose collectd viewer, you can even use it to view non-scylla related metrics from collectd.

ScyllaTop is a complementary tool, not a replacement for a complete monitoring and charting solution such as Scylla Monitoring or Grafana. Both can work together, using collectd as a common server and proxy.

scyllatop screenshot
This screenshot shows scyllatop monitoring memory-related metrics. See the article for the command line used.

Configuring collectd

First, you need a running collectd service. This collectd service must be configured to use the collectd unixsock plugin. In collectd’s configuration file (on Fedora: /etc/collectd.conf) uncomment the LoadPlugin line for unixsock so that it reads:

LoadPlugin unixsock

Later in the file you will find a commented-out configuration for this plugin. Uncomment it so that it looks something like this

<Plugin unixsock>
    SocketFile "/var/run/collectd-unixsock"
    SocketGroup "wheel"
    SocketPerms "0660"
    DeleteSocket false
</Plugin>

Pay attention to the socket group and permissions. Use a group that will let you have access to it when you run ScyllaTop. After this, restart collectd with

sudo systemctl restart collectd.service

to use the new configuration. Collectd is already set up for you in the Scylla RPM, which is pre-installed on the Scylla AMI on Amazon EC2. The configuration is in the file /etc/collectd.d/scylla.conf.

More helpful collectd configuration

Collectd has other plugins that you can enable to watch other important metrics such as network errors and disk I/O. A single collectd process can monitor all levels of the system, and supply data to scyllatop and to more complicated visualization tools.

Ethstat

The ethstat plugin collects information directly from the device driver for your network interface. Add the following configuration, filling in the name of your network interface in the Interface line.

 <Plugin "ethstat">
   Interface "eth0"
   Map "rx_csum_offload_errors" "if_rx_errors" "checksum_offload"
   Map "multicast" "if_multicast"
 </Plugin>

Disk

The disk plugin will let you track the data read from and written to a block device, the number of read and write operations issued, and the time requireed to complete. As with the ethstat plugin, you need to supply the device name in the configuration file.

More monitoring

You can also add the network and write_riemann plugins to connect the same connectd instance to other monitoring tools.

LoadPlugin network
LoadPlugin write_riemann

Interval 1

<Plugin "network">
    Listen "0.0.0.0" "25826"
</Plugin>
<Plugin "write_riemann">
 <Node "node1">
   Host "127.0.0.1"
   Port "5555"
   Protocol UDP
   StoreRates true
   TTLFactor 20
   AlwaysAppendDS false
 </Node>
</Plugin>

With those enabled, you can add much more monitoring functionality. For more info on monitoring Scylla, see Monitoring Scylla with collectd, riemann-dash, and Tessera.

Running ScyllaTop

If you don’t specify which items to display, scyllatop queries collectd, and displays all metrics available from collectd. (ScyllaTop doesn’t really know anything about scylla, it’s just a collectd client. You can run it for any collectd server for any application.)

To show a subset of commands, you can use a glob pattern (like ls and other shell commands use). The screenshot above was taken with

scyllatop ‘*memory*

When using glob patterns, make sure to quote them, as show above – otherwise the shell might expand them into file names, which is probably not what you wanted.

ScyllaTop is typically used in a simple case for monitoring a single node. It must run locally, on the same host as collectd. Typically all three of these will run on the same host:

  • scylla
  • collectd
  • scyllatop

More complex monitoring options for Scylla can include aggregating collectd or JMX data from multiple hosts.

Available metrics

Many metrics are available, and you can run scyllatop --list to list everything. Among the areas covered are:

  • CPU metrics such as system/user/idle time, and interrupt counts
  • Network traffic: packets in, out, and dropped
  • Memory used for cache
  • Disk I/O

ScyllaTop is flexible – whatever you can pass to collectd, it will let you watch. The 0.19 release of Scylla includes ScyllaTop. It’s part of the scylla-server package, and preinstalled on the Scylla AMI. If you have questions about ScyllaTop, please join us on the scylladb-users mailing list.

Follow @ScyllaDB on Twitter or subscribe to this site’s RSS feed to keep up with future releases.

Scylla TeamAbout Scylla Team

Scylla is the world’s fastest column-store database: the functionality of Apache Cassandra with the speed of a light key/value store.

Tags: monitoring