In this blog post, we’ll start by describing Kubernetes, a system for automating application deployment and monitoring, discuss how some Kubernetes concepts map to those of ScyllaDB, and provide a complete example (hosted on GitHub) of ScyllaDB on Kubernetes that should serve as a good starting point for your own deployment strategy.
What is Kubernetes, and Why Should You Care?
Software systems are often composed of several sub-systems which interact with each other. For example, an online store has a customer-facing website. Information about available products shown on the website is persisted using a database. Financial transactions are handled through a separate payment-processing service.
Each sub-system has its own software requirements, such as configuration files, dynamically linked libraries, and executables. It may also have resource requirements, such as a minimum amount of memory, disk, or processor time. Finally, there can be requirements for certain network interfaces or other operating system resources.
Containerization describes the practice of “packaging up” an application with all of its dependencies and configuration requirements into a unit that can be manipulated as a whole. Operating systems are able to run many containers concurrently, with strict resource-allocation and security policies being enforced.
In production deployments of systems, we often deal with complex requirements. These include the need for scaling resources (to accommodate increasing or diminishing load), the challenges of heterogeneous hardware, the possibility of network and of arbitrary hardware failure, and the need for distributed configuration management.
Kubernetes is a popular open-source system for automating the deployment and management of containerized applications which helps to solve these challenges.
Using the publicly available ScyllaDB Docker images, you can deploy ScyllaDB using Kubernetes to easily test integrating your applications with ScyllaDB.
Important Kubernetes Concepts and Terms
Understanding key concepts of Kubernetes and how they map to objects in the system is important for building a configuration for your application. We’ll briefly describe some objects that are useful for deploying ScyllaDB.
Pods
A Pod is the building-block of Kubernetes. It is a single logical process running on your cluster. For simplicity, we will consider a Pod to be a running instance of a single container (though co-located containers in a single Pod are also possible).
A Pod is independent of a particular host. Any host which meets the Pod’s resource requirements could be running the Pod.
When we want to have many instances of a process in a cluster (for example, many independent caching servers each handling an equal share of requests), we create many copies (“replicas”) of the same Pod.
Services and StatefulSets
A Service is an aggregation of many Pods. A useful feature of a Service is that it can load-balance requests to an arbitrary number of Pods behind a single IP address.
Services treat Pods as interchangeable and stateless. A Pod can be re-deployed at any time to another host, as Pods do not have fixed addresses.
In ScyllaDB, each node has its own state. Data and requests for data are partitioned across nodes based on node identity. ScyllaDB manages its own cluster state on the assumption of static network identifiers for each node.
A StatefulSet is similar to a Service, except that it allows for processes that are stateful. A StatefulSet associates stable persistent storage and a stable and unique network name to a container. This is true even when a Pod is scheduled to a different host.
Kubernetes API Objects
An important property of Kubernetes is that it is declarative. We describe the static characteristics of the API objects that we want to be present in our cluster (Pods, StatefulSets, configuration, storage, etc) in a YAML file, and then instruct Kubernetes to do all the work necessary to realize the state described by this specification.
ScyllaDB and Kubernetes
There is an example on GitHub with instructions and source files for deploying a small ScyllaDB cluster to Kubernetes on Google Compute Engine (GCE). This is a good starting point for experimenting with ScyllaDB on Kubernetes and developing your own deployment strategies.
For completeness, we’ll repeat the key points here.
The main objective of the demo is to start up an initial cluster and then to dynamically add a new node and have it automatically join the cluster.
Docker Image
We instruct Kubernetes to fetch the Docker image for a specific ScyllaDB release from Docker Hub. No additional Docker-specific configuration of the container is necessary.
Aggregating Pods
We create a StatefulSet of n ScyllaDB Pods with names scylla-0, scylla-1, …, scylla-n. DNS records are automatically created in the form scylla-0.scylla.default.svc.cluster.local.
We still use a Service to refer to the ScyllaDB Pods collectively, but we make sure the Service is “headless” so that no load-balancing or single end-point is set-up. This is important because it would interfere with token-aware ScyllaDB drivers.
Persistent Storage
The StatefulSet describes a PersistentVolumeClaim, which is a template for describing persistent storage requirements for each Pod. Volumes are created dynamically with the GCE storage provisioner based on the number of Pods and each volume bound to a particular Pod.
Seeding the Cluster
A new ScyllaDB node joins a cluster by connecting to one of a set of endpoints that the node knows about beforehand (statically). These are called the “seeds”. Once a node connects to a seed, the seed propagates information about the node to the rest of the cluster and about the cluster to the new node.
The IP address of a Pod changes based on the host it is running on at a given moment, so we should not use a Pod’s IP address in the seed list for new nodes. Instead, we use the seed’s hostname and corresponding DNS name to refer to it. scylla-1.scylla.default.svc.cluster.local will always refer to node 1!
Polling for Readiness
Kubernetes has to track the availability and readiness of all running Pods, and it depends on the Pod’s process defining for itself what it means to be ready.
For ScyllaDB nodes, we can use nodetool to query a node’s status. We wrap a call to nodetool in a shell script — ready-probe.sh — and mount this script in the Pod as part of the StatefulSet specification.
Where to Go From Here
This demo is intended to be a starting point for exploring and experimenting with ScyllaDB on Kubernetes. In any organization and production deployment, there are many operational concerns that we did not take into account. Some examples of these are integration with ScyllaDB’s metrics system, integration with ScyllaDB Manager, and node backup.
We encourage you to build on this example and to share your findings with the ScyllaDB community.
What follows are some ideas for next steps to improve the performance characteristics of ScyllaDB on Kubernetes.
CPU Pinning
An important part of the architecture of ScyllaDB is its shared-nothing model with no locking and explicit message-passing. The high performance that is possible with this architecture depends on having exclusive access to each processor core allocated to ScyllaDB
Kubernetes, by default, divides processor resources to different Pods based on time multiplexing. When we specify to Kubernetes the amount of processor resources to allocate to a Pod, we use a unit of measure defined as follows: 1 “core” is the ratio of running time that the Pod will have on any processor core over a fixed duration of time. For example, if we allocate 500 “millicores” to a Pod, then over a period of 100 ms the Pod will be executed on a core for 50 ms in total. The time the Pod spends executing can be — and often is — shared among many physical cores. This can hinder ScyllaDB’s performance.
A new feature of Kubernetes allows us to “pin” a Pod to a specific core so that no sharing takes place, though we haven’t experimented with this feature yet.
Choosing Good Storage Configurations
In this demo, we have not been careful about the characteristics of the storage provisioned for ScyllaDB nodes. Hardware details will vary based on your cloud provider and storage provisioner.
An in-development feature of Kubernetes will also allow for locally-attached storage, and this may also improve I/O performance.
The choice of file-system also matters. This Kubernetes pull-request describes a method for ensuring storage is formatted with the XFS file-system.
Next Steps
- Learn more about ScyllaDB from our product page.
- See what our users are saying about ScyllaDB.
- Download ScyllaDB. Check out our download page to run ScyllaDB on AWS, install it locally in a Virtual Machine, or run it in Docker.
References
- Scalable multi-node Cassandra deployment on Kubernetes Cluster
- Google Cloud Platform Quickstart for Linux
- Running Kubernetes on Google Compute Engine
- Example: Deploying Cassandra with Stateful Sets
- Strategies for Running Stateful Workloads in Kubernetes
