Scylla Manager 2.3 introduces a new mechanism to suspend and resume operations that can be used to implement maintenance windows or an off peak hours strategy. In this blogpost I will demonstrate how you can use this new feature to avoid running repairs during prime time.
When a cluster is in the suspended state, the only Scylla Manager tasks allowed to run are the health check tasks. That is checking if CQL, Alternator and REST services are responding timely. All the other tasks are stopped. Scheduling new tasks within the next 8 hours period or running tasks manually is not allowed.
To put a cluster into suspended state just execute “sctool suspend” command against the cluster. To validate the tasks are stopped, you can run the “sctool task list” command. In the example below cluster “prod” was suspended stopping an ongoing one shot backup task.
In the “
sctool task list” command output the
[SUSPENDED] prefix indicates the task would not run until the cluster is resumed. When resuming the cluster the default behavior is not to resume the stopped tasks automatically. The tasks would run according to the schedule, that is on “
Next run“. If you wish to resume the task add “
To automate the process of suspending and resuming in the context of reducing load during peak hours we would use crontab. In the example below we would put the cluster to suspend state on weekdays from 7PM to 12PM.
- In the Scylla Manager server, save the current crontab content to a file by executing “
crontab -l > ./my-crontab“
- Open ./my-crontab file in editor, and add the following lines
0 19 * * MON-FRI sctool suspend -c prod &>> ~/prod-suspend.log
0 0 * * TUE-SAT sctool resume -c prod &>> ~/prod-resume.log
- Install the rules by executing “crontab ./my-crontab”
Alternatively you can automate that with an external scheduler using Scylla Manager API. The following example shows how to suspend and resume cluster “prod” using curl from the Scylla Manager server host.
Read the release notes for Scylla Manager 2.3, then check out our download center. Remember that Scylla Manager is freely available for Scylla Open Source users for up to five nodes, and unlimited for users of Scylla Enterprise.