ScyllaDB Manager 2.4

The ScyllaDB Manager team is pleased to announce the release of ScyllaDB Manager 2.4, a production-ready version of ScyllaDB Manager for ScyllaDB Enterprise and ScyllaDB Open Source customers. ScyllaDB Manager is a centralized cluster administration and recurrent tasks automation tool.

Manager 2.4 adds Azure Blobs as a backup destination, automates the cluster wide restore procedure, and many improvements to restore, backups and monitoring. More below.

ScyllaDB Enterprise customers are encouraged to upgrade to ScyllaDB Manager 2.4 in coordination with the ScyllaDB support team.

The new release includes upgrades of both ScyllaDB Manager Server, Manager CLI tool (sctool).

Useful Links

Download ScyllaDB Manager and ScyllaDB Manager Agent for ScyllaDB Enterprise customers
Download ScyllaDB Manager and ScyllaDB Manager Agent for open source users (up to 5 nodes)
ScyllaDB Manager Docker instance and example
ScyllaDB Manager 2.4 documentation
Upgrade from ScyllaDB Manager 2.3 to ScyllaDB Manager 2.4
Submit a ticket for questions or issues with Manager 2.4 (ScyllaDB Enterprise users)

New Features

Backup To Azure Blobs

Similar to the existing backups to AWS S3 and GCP Storage, ScyllaDB manager can now upload backups to Azure Blobs. As a prerequisite, you will need to set up a Container, grant permissions , and configure Manager Agent with the Container id. Full documentation for enabling backup to Azure is here.

Automate Restore

Until this release, while backup was completely automated, the restore operation required a node by node operation.

ScyllaDB Manager 2.4 includes two important improvement which, together, automate the restore from backup procedure:

A new ScyllaDB Manager Agent functionality to list and download backup files.
A new Ansible script, automating a cluster wide restore on an entire cluster

New Manager Agent functionality

A new scylla-manager-agent download-files command, allows you to:

List clusters and nodes in a backup location. Example:
scylla-manager-agent download-files -L <backup-location> --list-nodes
List the node’s snapshots with filtering by keyspace / table glob patterns. Example:
scylla-manager-agent download-files -L <backup-location> --list-snapshots -K 'my_ks*'
Download backup files to ScyllaDB upload directory. Example:
scylla-manager-agent download-files -L <backup-location> -T <snapshot-tag> -d /var/lib/scylla/data/
Download to table upload directories or keyspace/table directory structure suitable for sstable loader (flag --mode)
Remove existing SSTables prior to download (flag --clear-tables)
Dry run mode (flag --dry-run)
Validation of disk space and data dir owner prior to download
Download bandwidth limit (flag –-rate-limit)
Manifest JSON printing to stdout (flag --dump-manifest)

Cluster wide restore

Cluster clone automation with Ansible. Restore to a different cluster of the same topology, read more about it here.

Improve Backup Retention Handling

Previous Manager release did not always properly clean old backups, for example:

Backup files of removed or replaces nodes
Backups files of removed backup tasks
Backups that failed at the upload stage

All of these backup leaks are now handled automatically by a backup purge stage, running after uploading files.

Clean Orphan Backup Files

In some rare cases, a bug in previous ScyllaDB Manager releases, left orphan backup files behind. An orphan file is a file which is not listed in any of the backup manifests, and as such will never be removed.

ScyllaDB Manager 2.4 adds a new task type and command “sctool backup validate” that ensures that all backup files are listed. The command can also be used to remove orphans files when executed with --delete-orphaned-files flag.

We recommend running backup validate after upgrading to Manager 2.4. Note this command may take some time (a few hours) to run, as is scanning the backup target for unlisted files.

Repair Improvements

Support for --intensity < 1 in row level repair. In case of --intensity < 1 the number of token ranges sent to ScyllaDB is a fraction of the number of shards (of the smallest repaired node). This allows you to repair 1 vNode token range at a time. Effectively make the repair slower and consume less bandwidth.

Monitoring

Labels use cluster ID not name
New metric for cluster ID to name mapping
Repair and backup metrics provide per table view
Internal metric aggregation is removed

You can use ScyllaDB Monitoring Stack 3.8 with -M 2.4 option to get Manager 2.4 dashboards

New Configuration Options

GCS default path for service_account_file is added (“/etc/scylla-manager-agent/gcs-service-account.json“), there is no need to modify ScyllaDB Manager Agent config file when configuring GCS service account.
S3, GCS, add storage_class parameter, example

s3:
storage_class: ONEZONE_IA

New section “rclone” where users can set general rclone options like number of files uploaded in parallel.

For a full configuration documentation see here.

More Improvements

Automatic port reading prioritizes HTTP over HTTPS, when running sctool on the same machine as the server we read ports from the config file. If HTTP and HTTPS is enabled we prefer HTTPS now it’s plain HTTP.
Support for running ScyllaDB Manager Agent on a custom port (not 10001), docs are here.
Sctool prefixing error messages with “failed to” is removed.
The system_schema keyspace is added to each backup, using text based schema is deprecated.

29 Jun 2021