Project Alternator is an open source project for an Amazon DynamoDB™-compatible API. The goal of this project is to deliver an open source alternative to DynamoDB, deployable wherever a user would want: on premises, on other public clouds like Microsoft Azure or Google Cloud Platform, or still on AWS (for users who wish to take advantage of other aspects of Amazon’s market-leading cloud ecosystem, such as the high-density i3en instances). DynamoDB users can keep their same client code unchanged. Alternator is written in C++ and is a part of Scylla.
Dynamo is the NoSQL database that changed the industry in 2007 when the paper about its architecture was published. The Dynamo paper set out a number of principles for a distributed database that have been emulated in a number of projects since:
However, Dynamo remained an internal proprietary project. In 2012 Amazon offered DynamoDB as a commercial cloud-based service on Amazon Web Services (AWS). It is not the same database, but was “built on the principles” of the original Dynamo. It is also proprietary, only available on Amazon’s own AWS.
DynamoDB can be very costly. Companies spend a great deal of time and engineering effort in their attempts to reduce their DynamoDB costs.
This last aspect became the challenge for ScyllaDB — to create a free DynamoDB open source alternative that would allow users to deploy a DynamoDB-compatible database to any cluster of their choosing (whether on-premises, or on any public or private cloud), while eliminating the barrier of high cost of ownership — the on-demand capacity or provisioned capacity mode costs in DynamoDB parlance.
The ScyllaDB team began development in 2014, and Scylla was released as an open source project in 2015. It was written to be highly performant by running close-to-the-metal.
Like DynamoDB, Scylla is also suitable for many key-value and wide-row use cases, while exhibiting the same properties of scalability, flexibility and reliability. Scylla was designed initially to be a drop-in replacement for Apache Cassandra™, a NoSQL system designed at Facebook and eventually open sourced through the Apache Foundation. While there are some significant differences between Dynamo and Cassandra, there are also many similarities. This is due in no small part to them having shared a co-inventor in Avinash Lakshman. In this way, Scylla can be seen as a technological descendant of Dynamo, and thus, a distant cousin to DynamoDB.
Based on ScyllaDB’s extensive experience in making API-compatible databases (e.g., Scylla’s existing Apache Cassandra compatibility), a design principle for the Alternator project is that no DynamoDB API calls should need to be altered from the developer’s perspective. The database system should accept input from the client application in DynamoDB-compatible API format, and transparently translate it into appropriate calls to the underlying database. Resultant data from queries would also need to be returned to users in the DynamoDB-compatible format.
All users would need to do is deploy Scylla, enable Alternator in scylla.yaml, and then point their current DynamoDB client applications to the Scylla cluster. Additionally, an Apache Spark™-based streaming solution to migrate data from existing DynamoDB instances into Scylla is currently available.
Scylla Alternator provides three key benefits to DynamoDB users:
1. Cost: DynamoDB charges for read and write transactions (RRUs and WRUs). A free open source solution eliminates these costs and minimizes other operational expenses. Scylla’s design efficiency allows developers to use significantly fewer resources for the same task or workload. According to our benchmark, users can expect to save 80% – 93% overall to support the same workload (5x-14x less expensive). Because of its highly performant design Scylla would also eliminate the cost of any in-memory cache (such as DAX).
2. Performance: While DynamoDB offers throughput guarantees, its latencies, especially long-tail p99 latencies, suffer in comparison to Scylla. That’s because Scylla is designed around asynchronous communications and a “shared nothing” architecture, allowing it to take full advantage of underlying modern multi-core, multi-CPU NUMA hardware. While both DynamoDB and Scylla have the capability to scale, we believe in giving you the most efficient performance possible, getting the highest utility of the underlying infrastructure you’re running on.
3. Openness: Scylla is open source. Users can review the source code and any known defects, and in return can add their own contributions to the project. Operationally, Scylla can run on any suitable server cluster regardless of location (on-premises, in any private or public cloud, or on our own DBaaS, Scylla Cloud) or deployment method (bare metal, containerized, virtualized, or deployed in pods via Kubernetes). This contributes to the user’s lower TCO by allowing deployment flexibility in line with their existing operations.
Our alternator.md and design doc provide detailed information of what’s supported and not yet supported today. In short, most standard applications will just work. The JSON HTTP API is mostly implemented, indexing works, multi zones are implemented, and many more features will work.
You can run Alternator right now with Scylla Open Source on your own Amazon, Azure and Google Cloud instances, using deployment methods like Docker and Kubernetes.
We have and will continue to harden the code, ensuring its production quality by putting it through our robust quality assurance cycles. We also continue to strive for completeness in implementing any API differences.
Scylla Open Source 4.0 also provides our streaming feature: Change Data Capture (CDC), which is currently experimental. We may add compatibility with DynamoDB streams later on.
For many lean organizations, one of the most attractive features of DynamoDB is utilizing it as a completely managed Database as a Service (DBaaS). So we also released a beta version of Alternator on our Scylla Cloud which you can try today. It currently runs on AWS, and you can utilize your own account to keep your existing AWS terms and discounts. Future updates to Scylla Cloud will run on Azure and GCP as well.
In the future, we will also offer a fully standalone Scylla Enterprise release containing the Alternator software.
We have a beta version of our Kubernetes operator so you can fully deploy and manage a DynamoDB-compatible database wherever you wish.
To make the switch as easy as possible we’ve extended our online migration tools to be relatively simple: just start streaming the changes from DynamoDB plus run a full scan. The Scylla Spark Migrator project has been enhanced to support the DynamoDB-compatible API.
Scylla and the Alternator interface are true open source projects. We encourage those looking to contribute to review the source code and any known defects, and read our guidelines on how to add their own contributions.
ScyllaDB co-founders Avi Kivity, Dor Laor and Distinguished Engineer Nadav Har’el gave an in-depth webinar (including demos) of Scylla Alternator. Click below to watch the recorded webinar.