The ScyllaDB Manager team is pleased to announce the release of ScyllaDB Manager 2.4, a production-ready version of ScyllaDB Manager for ScyllaDB Enterprise and ScyllaDB Open Source customers. ScyllaDB Manager is a centralized cluster administration and recurrent tasks automation tool.
Manager 2.4 adds Azure Blobs as a backup destination, automates the cluster wide restore procedure, and many improvements to restore, backups and monitoring. More below.
ScyllaDB Enterprise customers are encouraged to upgrade to ScyllaDB Manager 2.4 in coordination with the ScyllaDB support team.
The new release includes upgrades of both ScyllaDB Manager Server, Manager CLI tool (sctool).
- Download ScyllaDB Manager and ScyllaDB Manager Agent for ScyllaDB Enterprise customers
- Download ScyllaDB Manager and ScyllaDB Manager Agent for open source users (up to 5 nodes)
- ScyllaDB Manager Docker instance and example
- ScyllaDB Manager 2.4 documentation
- Upgrade from ScyllaDB Manager 2.3 to ScyllaDB Manager 2.4
- Submit a ticket for questions or issues with Manager 2.4 (ScyllaDB Enterprise users)
Backup To Azure Blobs
Similar to the existing backups to AWS S3 and GCP Storage, ScyllaDB manager can now upload backups to Azure Blobs. As a prerequisite, you will need to set up a Container, grant permissions , and configure Manager Agent with the Container id. Full documentation for enabling backup to Azure is here.
Until this release, while backup was completely automated, the restore operation required a node by node operation.
ScyllaDB Manager 2.4 includes two important improvement which, together, automate the restore from backup procedure:
- A new ScyllaDB Manager Agent functionality to list and download backup files.
- A new Ansible script, automating a cluster wide restore on an entire cluster
New Manager Agent functionality
scylla-manager-agent download-files command, allows you to:
- List clusters and nodes in a backup location. Example:
scylla-manager-agent download-files -L <backup-location> --list-nodes
- List the node’s snapshots with filtering by keyspace / table glob patterns. Example:
scylla-manager-agent download-files -L <backup-location> --list-snapshots -K 'my_ks*'
- Download backup files to ScyllaDB upload directory. Example:
scylla-manager-agent download-files -L <backup-location> -T <snapshot-tag> -d /var/lib/scylla/data/
- Download to table upload directories or keyspace/table directory structure suitable for sstable loader (flag
- Remove existing SSTables prior to download (flag
- Dry run mode (flag
- Validation of disk space and data dir owner prior to download
- Download bandwidth limit (flag –
- Manifest JSON printing to stdout (flag
Cluster wide restore
Cluster clone automation with Ansible. Restore to a different cluster of the same topology, read more about it here.
Improve Backup Retention Handling
Previous Manager release did not always properly clean old backups, for example:
- Backup files of removed or replaces nodes
- Backups files of removed backup tasks
- Backups that failed at the upload stage
All of these backup leaks are now handled automatically by a backup purge stage, running after uploading files.
Clean Orphan Backup Files
In some rare cases, a bug in previous ScyllaDB Manager releases, left orphan backup files behind. An orphan file is a file which is not listed in any of the backup manifests, and as such will never be removed.
ScyllaDB Manager 2.4 adds a new task type and command “
sctool backup validate” that ensures that all backup files are listed. The command can also be used to remove orphans files when executed with
We recommend running backup validate after upgrading to Manager 2.4. Note this command may take some time (a few hours) to run, as is scanning the backup target for unlisted files.
--intensity < 1 in row level repair. In case of
--intensity < 1 the number of token ranges sent to ScyllaDB is a fraction of the number of shards (of the smallest repaired node). This allows you to repair 1 vNode token range at a time. Effectively make the repair slower and consume less bandwidth.
- Labels use cluster ID not name
- New metric for cluster ID to name mapping
- Repair and backup metrics provide per table view
- Internal metric aggregation is removed
You can use ScyllaDB Monitoring Stack 3.8 with -M 2.4 option to get Manager 2.4 dashboards
New Configuration Options
- GCS default path for service_account_file is added (“/
etc/scylla-manager-agent/gcs-service-account.json“), there is no need to modify ScyllaDB Manager Agent config file when configuring GCS service account.
- S3, GCS, add storage_class parameter, example
- New section “rclone” where users can set general rclone options like number of files uploaded in parallel.
For a full configuration documentation see here.
- Automatic port reading prioritizes HTTP over HTTPS, when running sctool on the same machine as the server we read ports from the config file. If HTTP and HTTPS is enabled we prefer HTTPS now it’s plain HTTP.
- Support for running ScyllaDB Manager Agent on a custom port (not 10001), docs are here.
- Sctool prefixing error messages with “
failed to” is removed.
system_schemakeyspace is added to each backup, using text based schema is deprecated.
29 Jun 2021