The Scylla Manager team is pleased to announce the release of Scylla Manager 2.4, a production-ready version of Scylla Manager for Scylla Enterprise and Scylla Open Source customers. Scylla Manager is a centralized cluster administration and recurrent tasks automation tool.
Manager 2.4 adds Azure Blobs as a backup destination, automates the cluster wide restore procedure, and many improvements to restore, backups and monitoring. More below.
Scylla Enterprise customers are encouraged to upgrade to Scylla Manager 2.4 in coordination with the Scylla support team.
The new release includes upgrades of both Scylla Manager Server, Manager CLI tool (sctool).
- Download Scylla Manager and Scylla Manager Agent for Scylla Enterprise customers
- Download Scylla Manager and Scylla Manager Agent for open source users (up to 5 nodes)
- Scylla Manager Docker instance and example
- Scylla Manager 2.4 documentation
- Upgrade from Scylla Manager 2.3 to Scylla Manager 2.4
- Submit a ticket for questions or issues with Manager 2.4 (Scylla Enterprise users)
Backup To Azure Blobs
Similar to the existing backups to AWS S3 and GCP Storage, Scylla manager can now upload backups to Azure Blobs. As a prerequisite, you will need to set up a Container, grant permissions , and configure Manager Agent with the Container id. Full documentation for enabling backup to Azure is here.
Until this release, while backup was completely automated, the restore operation required a node by node operation.
Scylla Manager 2.4 includes two important improvement which, together, automate the restore from backup procedure:
- A new Scylla Manager Agent functionality to list and download backup files.
- A new Ansible script, automating a cluster wide restore on an entire cluster
New Manager Agent functionality
scylla-manager-agent download-files command, allows you to:
- List clusters and nodes in a backup location. Example:
scylla-manager-agent download-files -L <backup-location> --list-nodes
- List the node’s snapshots with filtering by keyspace / table glob patterns. Example:
scylla-manager-agent download-files -L <backup-location> --list-snapshots -K 'my_ks*'
- Download backup files to Scylla upload directory. Example:
scylla-manager-agent download-files -L <backup-location> -T <snapshot-tag> -d /var/lib/scylla/data/
- Download to table upload directories or keyspace/table directory structure suitable for sstable loader (flag
- Remove existing SSTables prior to download (flag
- Dry run mode (flag
- Validation of disk space and data dir owner prior to download
- Download bandwidth limit (flag –
- Manifest JSON printing to stdout (flag
Cluster wide restore
Cluster clone automation with Ansible. Restore to a different cluster of the same topology, read more about it here.
Improve Backup Retention Handling
Previous Manager release did not always properly clean old backups, for example:
- Backup files of removed or replaces nodes
- Backups files of removed backup tasks
- Backups that failed at the upload stage
All of these backup leaks are now handled automatically by a backup purge stage, running after uploading files.
Clean Orphan Backup Files
In some rare cases, a bug in previous Scylla Manager releases, left orphan backup files behind. An orphan file is a file which is not listed in any of the backup manifests, and as such will never be removed.
Scylla Manager 2.4 adds a new task type and command “
sctool backup validate” that ensures that all backup files are listed. The command can also be used to remove orphans files when executed with
We recommend running backup validate after upgrading to Manager 2.4. Note this command may take some time (a few hours) to run, as is scanning the backup target for unlisted files.
--intensity < 1 in row level repair. In case of
--intensity < 1 the number of token ranges sent to Scylla is a fraction of the number of shards (of the smallest repaired node). This allows you to repair 1 vNode token range at a time. Effectively make the repair slower and consume less bandwidth.
- Labels use cluster ID not name
- New metric for cluster ID to name mapping
- Repair and backup metrics provide per table view
- Internal metric aggregation is removed
You can use Scylla Monitoring Stack 3.8 with -M 2.4 option to get Manager 2.4 dashboards
New Configuration Options
- GCS default path for service_account_file is added (“/
etc/scylla-manager-agent/gcs-service-account.json“), there is no need to modify Scylla Manager Agent config file when configuring GCS service account.
- S3, GCS, add storage_class parameter, example
- New section “rclone” where users can set general rclone options like number of files uploaded in parallel.
For a full configuration documentation see here.
- Automatic port reading prioritizes HTTP over HTTPS, when running sctool on the same machine as the server we read ports from the config file. If HTTP and HTTPS is enabled we prefer HTTPS now it’s plain HTTP.
- Support for running Scylla Manager Agent on a custom port (not 10001), docs are here.
- Sctool prefixing error messages with “
failed to” is removed.
system_schemakeyspace is added to each backup, using text based schema is deprecated.
29 Jun 2021