fbpx

Scylla Manager 2.4

The Scylla Manager team is pleased to announce the release of Scylla Manager 2.4, a production-ready version of Scylla Manager for Scylla Enterprise and Scylla Open Source customers. Scylla Manager is a centralized cluster administration and recurrent tasks automation tool.

Manager 2.4 adds Azure Blobs as a backup destination, automates the cluster wide restore procedure, and many improvements to restore, backups and monitoring. More below.

Scylla Enterprise customers are encouraged to upgrade to Scylla Manager 2.4 in coordination with the Scylla support team.

The new release includes upgrades of both Scylla Manager Server, Manager CLI tool (sctool).

Useful Links

New Features

Backup To Azure Blobs

Similar to the existing backups to AWS S3 and GCP Storage, Scylla manager can now upload backups to Azure Blobs. As a prerequisite, you will need to set up a Container, grant permissions , and configure Manager Agent with the Container id. Full documentation for enabling backup to Azure is here.

Automate Restore

Until this release, while backup was completely automated, the restore operation required a node by node operation.

Scylla Manager 2.4 includes two important improvement which, together, automate the restore from backup procedure:

  1. A new Scylla Manager Agent functionality to list and download backup files.
  2. A new Ansible script, automating a cluster wide restore on an entire cluster

New Manager Agent functionality

A new scylla-manager-agent download-files command, allows you to:

  • List clusters and nodes in a backup location. Example:
    scylla-manager-agent download-files -L <backup-location> --list-nodes
  • List the node’s snapshots with filtering by keyspace / table glob patterns. Example:
    scylla-manager-agent download-files -L <backup-location> --list-snapshots -K 'my_ks*'
  • Download backup files to Scylla upload directory. Example:
    scylla-manager-agent download-files -L <backup-location> -T <snapshot-tag> -d /var/lib/scylla/data/
  • Download to table upload directories or keyspace/table directory structure suitable for sstable loader (flag --mode)
  • Remove existing SSTables prior to download (flag --clear-tables)
  • Dry run mode (flag --dry-run)
  • Validation of disk space and data dir owner prior to download
  • Download bandwidth limit (flag –-rate-limit)
  • Manifest JSON printing to stdout (flag --dump-manifest)

Cluster wide restore

Cluster clone automation with Ansible. Restore to a different cluster of the same topology, read more about it here.

Improve Backup Retention Handling

Previous Manager release did not always properly clean old backups, for example:

  • Backup files of removed or replaces nodes
  • Backups files of removed backup tasks
  • Backups that failed at the upload stage

All of these backup leaks are now handled automatically by a backup purge stage, running after uploading files.

Clean Orphan Backup Files

In some rare cases, a bug in previous Scylla Manager releases, left orphan backup files behind. An orphan file is a file which is not listed in any of the backup manifests, and as such will never be removed.

Scylla Manager 2.4 adds a new task type and command “sctool backup validate” that ensures that all backup files are listed. The command can also be used to remove orphans files when executed with --delete-orphaned-files flag.

We recommend running backup validate after upgrading to Manager 2.4. Note this command may take some time (a few hours) to run, as is scanning the backup target for unlisted files.

Repair Improvements

Support for --intensity < 1 in row level repair. In case of --intensity < 1 the number of token ranges sent to Scylla is a fraction of the number of shards (of the smallest repaired node). This allows you to repair 1 vNode token range at a time. Effectively make the repair slower and consume less bandwidth.

Monitoring

  1. Labels use cluster ID not name
  2. New metric for cluster ID to name mapping
  3. Repair and backup metrics provide per table view
  4. Internal metric aggregation is removed

You can use Scylla Monitoring Stack 3.8 with -M 2.4 option to get Manager 2.4 dashboards

New Configuration Options

  • GCS default path for service_account_file is added (“/etc/scylla-manager-agent/gcs-service-account.json“), there is no need to modify Scylla Manager Agent config file when configuring GCS service account.
  • S3, GCS, add storage_class parameter, example

s3:
  storage_class: ONEZONE_IA

  • New section “rclone” where users can set general rclone options like number of files uploaded in parallel.

For a full configuration documentation see here.

More Improvements

  • Automatic port reading prioritizes HTTP over HTTPS, when running sctool on the same machine as the server we read ports from the config file. If HTTP and HTTPS is enabled we prefer HTTPS now it’s plain HTTP.
  • Support for running Scylla Manager Agent on a custom port (not 10001), docs are here.
  • Sctool prefixing error messages with “failed to” is removed.
  • The system_schema keyspace is added to each backup, using text based schema is deprecated.

29 Jun 2021