Join us at P99 CONF, the virtual event on all things performance. Oct 19 & 20. Registration is free

See all blog posts

ScyllaDB Summit Preview: ScyllaDB Got Slow! Using Tools, Talent and Tracing to Find Out Why

ScyllaDB Summit 2018: ScyllaDB got slow! Using Tools, Talent, and Tracting to Bring it up to Speed [banner graphic]

In the run-up to ScyllaDB Summit 2018, we’ll be featuring our speakers and providing sneak peeks at their presentations. The second interview in this series is with ScyllaDB’s own Vlad Zolotarov.

Vladislav “Vlad” Zolotarov is one of our experts at getting the most out of ScyllaDB, having written articles in the past about CQL tracing, tracing slow queries, securing your cluster, and using Hinted Handoffs. He will speak at the ScyllaDB Summit in a talk entitled ScyllaDB Got Slow! Using Tools, Talent, and Tracing to Find Out Why. He took the time to give a sneak peek into his upcoming session.


Outside of work, Vlad enjoys off-road biking and spending time with his family.


“Slow” can mean many things to many people. Latency or throughput, storage I/O, process cycle times like compactions, or a raft of low-level tweaks such as CPU scheduling. What specific aspects will you focus on in your talk?


In the context of this talk “slow” means unsatisfactory ScyllaDB’s performance: either throughput or latency. We will start from that and will try to drill down trying to identify the possible reason for a slowdown. And reasons may vary from suboptimal OS-level configuration (networking, I/O, etc.) to bad ScyllaDB practices.


Speed improvements can be found within the database, but also in the applications and ecosystem the database is connected to. Will you focus solely on inside-the-box analysis, or also on broader systemic troubleshooting?


We will start with analyzing the server’s state first. However tools ScyllaDB provides allow detecting issues that are caused not only by problems on the server side but on the application side as well.


Photo: Vlad Zolotarov on his mountain bike in the rain.
Photo: Vlad Zolotarov jumping his mountain bike off a ramp.

What are the go-to tools you have in your toolbox?


First of all ScyllaDB Monitoring, which is a bunch of Grafana dashboards representing various metrics from ScyllaDB cluster. This is what you always start with. In many cases this is where it ends too.


But if you need to drill down more you’ll eventually get to nodetool (statistics commands), system log, cqlsh and CQL Tracing.


If you had a single, specific tip to give to ScyllaDB database managers to improve performance, what would it be?


Always try to understand what the problem is before trying to fix anything. Remember that the cluster is going to work as fast as the slowest node and in ScyllaDB’s case — as the slowest shard. Therefore always start with identifying where the bottleneck is. There are a few ways of approaching this, and we’ll discuss various methods during the talk.

Do you have any recommendations for ScyllaDB Summit attendees of articles or topics they should review prior to when they sit down to hear you speak?


I expect people to know ScyllaDB fundamentals and have a basic understanding of how a server like ScyllaDB or Cassandra works: what is a cluster, node, keyspace, partition, row, primary/partition/clustering keys, how data is stored, queried. Some familiarity with basic computers-related concepts is an advantage: multi-queue devices, CPU affinity, NUMA, IRQ, etc.


They should also familiarize themselves with ScyllaDB Monitor 2.0, which we just released.


Thanks Vlad! I am sure a lot of people are going to be interested in your session.


You’re welcome. If anyone hasn’t registered yet, feel free to use the code vlad25monster to get 25% off the current price.

About Peter Corless

Peter Corless is the Director of Technical Advocacy at ScyllaDB. He listens to users’ stories and discovers wisdom in each to share with other practitioners in the industry at large, whether through blogs or in technical presentations. He occasionally hosts live industry events from webinars to online conferences.