May24

9 Steps for Building a Highly Available Time-Series Solution with Scylla and KairosDB

Introduction

A highly available time-series solution requires an efficient tailored front-end framework and a backend database with a fast ingestion rate. KairosDB provides a simple and reliable way to ingest and retrieve sensors’ information or metrics, while Scylla provides a highly reliable and performant backend database that scales indefinitely, and can store large quantities of time-series data.

This post describes how metrics from any monitored system, in this example Scylla metrics from one Scylla cluster, are stored in a monitoring solution backend database based on KairosDB and Scylla. Collectd will be used to push Scylla metrics into KairosDB via telnet protocol, and KairosDB will store the data on a separate Scylla cluster via Thrift API. Essentially, Scylla will be storing the metrics of another Scylla cluster. Scylla monitoring with KairosDB and Scylla

KairosDB

KairosDB is a fast distributed scalable time-series database. It was initially a rewrite of the original OpenTSDB project, but evolved to a different system for which data management, data processing, and visualization are fully separated. (More on KairosDB and Scylla can be found here.)

Collectd

Collectd gathers metrics from various sources, e.g. the operating system, applications, log-files and external devices, and stores the information or makes it available over the network. The collected metrics are used to monitor systems, find performance bottlenecks and predict future system load (i.e. capacity planning.)

Why favor Scylla and KairosDB over other choices?

KairosDB is a stateless shim layer on top of Scylla that doesn’t touch the ingress. The combination of Scylla and KairosDB allows infinite scale out of the Scylla storage layer and puts an end to small siloed islands of monitoring and the problems of scaling a monitoring solution. Scylla also provides best of-class high availability and top performance. Now without further ado, let’s jump to the setup.

Setup

You will need the following minimal setup for this procedure:

  • Scylla cluster to be monitored
  • KairosDB node
  • Scylla cluster acting as backend database to store KairosDB data (It is recommended to install Scylla in a cluster, to benefit from high availability)

The Procedure

1. Download KairosDB

2. KairosDB requires Java (and JAVA_HOME setting)

See procedure here: http://askubuntu.com/questions/175514/how-to-set-java-home-for-java

3. Extract the KairosDB file on the node

4. Configure KairosDB

In conf/kairosdb.properties file:

  • Change the kairosdb.service.datastore property to the datastore you wish to use:

kairosdb.service.datastore=org.kairosdb.datastore.cassandra.CassandraModule

  • In the Cassandra section:

kairosdb.datastore.cassandra.host_list=[scylla_host_ip1]:9160,[scylla_host_ip2]:9160, .. ,[scylla_host_ipN]:9160 kairosdb.datastore.cassandra.keyspace=kairosdb

5. Start KairosDB

Note: it is not required to start KairosDB with root (sudo) privileges. Change to the bin directory and start KairosDB using one of the following commands:

  • To start KairosDB and run in the foreground type
> ./kairosdb.sh run
  • To run KairosDB as a background process type
> ./kairosdb.sh start
  • To stop KairosDB when running as a background process type
> ./kairosdb.sh stop

6. Verify KairosDB keyspace and tables are available in Scylla

  • Connect to one of the Scylla cluster nodes you configured in step 4
  • Open cqlsh -> run: ‘cqlsh [node_ip]’
  • Check that the keyspace and tables were created

cqlsh> DESCRIBE TABLES -> you should find the keyspace value you used in step 4 with three tables. Keyspace kairosdb

—————–

data_points row_key_index string_index

7. Verify SELinux is disabled on the to-be monitored Scylla cluster (on all nodes)

  • Check SELinux status -> run: ‘sudo sestatus’
  • If ‘enabled’, disable it by doing the following:
  • Edit selinux config file: ‘sudo vi /etc/selinux/config’
  • Modify ‘SELINUX=disabled’ -> save
  • Reboot node

8. Push data from the to-be monitored Scylla cluster to kairosdb using collectd

Perform the following on each of your scylla cluster nodes:

  • sudo yum install git -y
  • git clone https://github.com/kairosdb/collectd-kairosdb.git
  • sudo cp collectd-kairosdb/kairosdb_writer.py /usr/lib64/collectd/.
  • vi collectd-kairosdb/kairosdb.conf -> KairosDBURI “telnet://[KairosDB_IP]:4242”
  • sudo bash
  • cat collectd-kairosdb/kairosdb.conf >> /etc/collectd.d/scylla.conf
  • sudo systemctl restart collectd

(Note: Scylla comes with collectd running by default, so no need to install it, only configure it.)

9. Generate some traffic on the monitored Scylla cluster, for example by using Cassandra-stress:

cassandra-stress write n=10000000 -mode cql3 native -node [scylla_ip1],[scylla_ip2]…

Execute Queries on KairosDB (web UI / REST-API)

Web UI

At this point you can open the KairosDB web UI by pointing your web browser to: http://[KairosDB_IP]:8080 to execute queries on the relevant metrics the and generate graphs.

Here are several examples of metrics from the Scylla nodes in the monitored system. This Scylla cluster sends the Scylla metrics to Kairos and later these metrics are stored on another Scylla instance.

KairosDB CPU load graph

Query on AVG ‘reactor.0/1/2/3.gauge.gauge_load.value’ metric, to demonstrate the average CPU utilization of the Scylla nodes in the monitored system.

KairosDB MAX latency graph

Query on MAX ‘scollectd.0/1/2/3.latency.latency_.value’ metric, to demonstrate the maximum latency of the Scylla nodes in the monitored system.

Kairos Compactions Graph

Query on MAX ‘compaction_manager.0/1/2/3.objects.objects_compactions.value’ metric, to demonstrate ongoing compactions on the Scylla nodes in the monitored system.

REST-API

Another option is to use the KairosDB REST-API to query Scylla metrics. This can be useful for triggering alerts.

Note: json format can be taken from the web UI, once you generate a query: http://[KairosDB_IP]:8080/api/v1/datapoints/query?query={json}

Ready to get started building a highly available time-series solution? Download Scylla and get started today.

Tomer SandlerAbout Tomer Sandler

Tomer Sandler joined ScyllaDB as a solution architect after a 12 year career in SW Quality Engineering, mostly in storage and telecom lawful interception domains. Prior to ScyllaDB, Tomer held various QA management roles at Dell EMC, leading a group of QA engineers and information developers for ScaleIO storage, and at VERINT, as Senior QA engineer working with U.S. and Spanish Law Enforcement Agencies customers engagement experience. In his free time he enjoys hiking, playing the saxophone, spending time with his family and last but not least, engaging in OSINT (Open Source Intelligence) research.

Tags: kairosdb, monitoring, time-series