Scylla Open Source 4.0

The Scylla team is pleased to announce the availability of Scylla Open Source 4.0.0, a production-ready release of our open source NoSQL database.

Scylla 4.0 is focused on promoting experiential features Lightweight Transactions (LWT) and our Amazon DynamoDB Compatible API (Alternator) to production level.

In addition we are adding a new Scylla AMI user data format and more (below).

Scylla 4.0 also includes new and improved experimental features, like CDC, UDF and a major update to the underlying infrastructure, making all streaming operations based on existing repair logic (more below)

Scylla is an open source, NoSQL database, with superior performance and consistently low latencies. Find the Scylla Open Source 4.0 repository for your Linux distribution here. A Scylla 4.0 Docker image is also available.

Please note that only the latest two minor releases of Scylla Open Source project are supported. Starting today, only Scylla Open Source 4.0 and Scylla 3.3 will be supported, and Scylla 3.2 will be retired.

Related Links

New features in Scylla 4.0

Lightweight Transactions (LWT) #1359

Lightweight transactions (LWT), also known as Compare and Set (CAS), add support for conditional INSERT and UPDATE CQL commands. Scylla supports both equality and non-equality conditions for lightweight transactions (i.e., you can use <, <=, >, >=, !=,= and IN operators in an IF clause).

LWT was introduced as an experimental feature in Scylla 3.2, and now, with Scylla 4.0, LWT is promoted to production ready.

Selected updates from Scylla 3.3:
Support LIKE operator in conditional expressions #5777

You can learn more on LWT in Scylla and LWT optimizations from the latest LWT Webinar (registration required) and LWT documentation.

Scylla Alternator: The Open Source DynamoDB-compatible API

Project Alternator is an open-source implementation for an Amazon DynamoDB™-compatible API. The goal of this project is to deliver an open source alternative to Amazon’s DynamoDB that can be run anywhere, whether on-premises or on your own favorite cloud provider, or in Scylla Cloud.

Scylla Alternator was introduced as an experimental feature in Scylla 3.2, and it is now, in Scylla 4.0, promoted to production.

Selected updates from Scylla 3.3:

  • Limit huge requests #5832
  • Allow tagging tables #5066
  • Support ConditionExpression #5035
  • Support KeyConditionExpression #5037
  • Support parallel Scan #5059
  • Add support for ScanIndexForward #5153
  • Hook to admission control (like CQL does) #5029

Read more here: Open Source DynamoDB-compatible API Documentation.

Deployment

  • Scylla EC2 AMI user data format has changed. The new format is JSON based, and while it supports all the options it is *not* backward compatible.
  • Completing the new AMI format, we are releasing a Cloud Formation template, for easy launch of a Scylla cluster in one region.

Experimental features in Scylla 4.0

Change Data Capture (CDC)

While functionally complete, we are still testing to validate CDC is production ready, with the plan to move it to GA in a subsequent Scylla 4.x release. No API updates are expected.

Selected updates from Scylla 3.3:

  • Support postimage #4992
  • Support collections in CDC #5669
  • Handle cluster topology changes in CDC #4989
  • Protect CDC Log and CDC Description tables from being modified by the user #4991
  • Add new CDC mertics #6002 (for a full list of new metrics in 4.0 see below)
  • Include CDC in tracing info #5912
  • CDC now logs Lightweight Transactions (LWT) operations correctly. #5869
  • Performance improvement
  • Many more bug fixes

See CDC Documentation

Repair based node operations

Scylla has two use cases for streaming data between nodes:

  • Topology changes, like adding and removing nodes
  • Repair, off line process to compare and sync data between nodes

Up to Scylla 4.0, the two used different underline logic. Starting in this release, the same streaming logic used for repair is also used for topology changes, making it more robust, reliable, with better visibility into streaming progress. In particular, streaming can restart from the same point it stopped, for example in case a node,or the network, restart during streaming. This is a big time saver when adding, or removing large nodes.

In Scylla 4.0.0, repair base node operation is disabled by default, you can enable by adding the following to scylla.yaml:

enable_repair_based_node_ops: true

Related issues #4208 #3003

User Defined Functions (UDFs) #2204

Scylla now has basic Lua based UDF. First released in Scylla 3.3, it is still under development and will stay experimental in 4.0.

Select updates from Scylla 3.3

  • Fix Lua: list and list were broken #5711

More on UDFA and UDA here

Other notable updates and bug fixes in the release

  • Date Tiered Compaction Strategy (DTCS) is deprecated and eventually be removed. If you are still using it, please switch to Time Window Compaction Strategy (TWCS).
  • API: add error injection to REST API (used for testing only)
    A new API allows enable/disable error injections to different places in the code #3295
  • Scylla will now ignore a keyspace that is removed while being repaired, or without tables, rather than failing the repair operation. #5942 #6022
  • Local secondary indexes, as well as materialized views that have the same partition key as their base table, are now updated synchronously. This improves consistency and simplifies concurrency control. #4365. Alternator was updated to take advantage of this.
  • a new REST API for deleting a table from a snapshot #5805
  • CQL: Fix missing aggregate functions for counters
  • The long-deprecated RandomPartitioner and ByteOrderedPartitioner partitioners were removed #5636

07 May 2020