Virtual Workshops
Twice-monthly interactive sessions with our NoSQL solution architects.
Join Our Next Session >

PROJECT ALTERNATOR - The Open Source Amazon DynamoDB-compatible API

Share on facebook
Share on twitter
Share on print
Share on email

Project Alternator is the open source Amazon DynamoDB-compatible API found in our Scylla NoSQL database. With Alternator you can use Scylla as a drop-in replacement for DynamoDB, supporting the same client SDKs, data modeling and queries. However, the advantage is that you can deploy Scylla wherever you want: on-premises, or on any public cloud. Plus manage it however you want via Docker or Kubernetes. You can read more about the project on Github here.

The Background: Switch to Scylla with No Changes to DynamoDB Client Code

Dynamo is the NoSQL database that changed the industry in 2007 when the paper about its architecture was published. The Dynamo paper set out a number of principles for a distributed database that have been emulated in a number of projects since:

  • key-value NoSQL store that doesn’t need a rigid relational or hierarchical data model
  • horizontally scalable to take advantage of economically efficient commodity hardware
  • highly available and reliable through a distributed, replicated peer-to-peer architecture
  • tolerant of failures through hinted handoffs, anti-entropy mechanisms like Merkle trees, and gossip-based cluster membership and adaptation to topology changes

However, Dynamo remained an internal proprietary project. In 2012 Amazon offered DynamoDB as a commercial cloud-based service on Amazon Web Services (AWS). It is not the same database, but was “built on the principles” of the original Dynamo. It is also proprietary, only available on Amazon’s own AWS.

DynamoDB can be very costly. Companies spend a great deal of time and engineering effort in their attempts to reduce their DynamoDB costs.

This last aspect became the challenge for ScyllaDB — to create a free DynamoDB open source alternative that would allow users to deploy a DynamoDB-compatible database to any cluster of their choosing (whether on-premises, or on any public or private cloud), while eliminating the barrier of high cost of ownership — the on-demand capacity or provisioned capacity mode costs in DynamoDB parlance.

Scylla as a Platform for a DynamoDB-compatible API

The ScyllaDB team began development in 2014, and Scylla was released as an open source project in 2015. It was written to be highly performant by running close-to-the-metal.

Like DynamoDB, Scylla is also suitable for many key-value and wide-row use cases, while exhibiting the same properties of scalability, flexibility and reliability. Scylla was designed initially to be a drop-in replacement for Apache Cassandra™, a NoSQL system designed at Facebook and eventually open sourced through the Apache Foundation. While there are some significant differences between Dynamo and Cassandra, there are also many similarities. This is due in no small part to them having shared a co-inventor in Avinash Lakshman. In this way, Scylla can be seen as a technological descendant of Dynamo, and thus, a distant cousin to DynamoDB.

In testing between the two systems Scylla was proven to outperform DynamoDB in throughput for the same cost, with lower latencies and overall better price-performance. However, due to their different data access methods, switching between the two systems is non-trivial to users. Scylla uses the Cassandra Query Language (CQL), which is syntactically akin to Structured Query Language (SQL), while DynamoDB data types require Javascript Object Notation (JSON) format for queries. Thus, for a user to switch systems, they would be forced to rewrite their applications and re-design their queries.

Based on ScyllaDB’s extensive experience in making API-compatible databases (e.g., Scylla’s existing Apache Cassandra compatibility), a design principle for the Alternator project is that no DynamoDB API calls should need to be altered from the developer’s perspective. The database system should accept input from the client application in DynamoDB-compatible API format, and transparently translate it into appropriate calls to the underlying database. Resultant data from queries would also need to be returned to users in the DynamoDB-compatible format.

All users would need to do is deploy Scylla, enable Alternator in scylla.yaml, and then point their current DynamoDB client applications to the Scylla cluster. Additionally, an Apache Spark™-based streaming solution to migrate data from existing DynamoDB instances into Scylla is currently available.

Why Scylla is a Better Alternative to DynamoDB

Scylla Alternator provides three key benefits to DynamoDB users:

1. Cost: DynamoDB charges for read and write transactions (RRUs and WRUs). A free open source solution eliminates these costs and minimizes other operational expenses. Scylla’s design efficiency allows developers to use significantly fewer resources for the same task or workload. According to our benchmark, users can expect to save 80% – 93% overall to support the same workload (5x-14x less expensive). Because of its highly performant design Scylla would also eliminate the cost of any in-memory cache (such as DAX).

2. Performance: While DynamoDB offers throughput guarantees, its latencies, especially long-tail p99 latencies, suffer in comparison to Scylla. That’s because Scylla is designed around asynchronous communications and a “shared nothing” architecture, allowing it to take full advantage of underlying modern multi-core, multi-CPU NUMA hardware. While both DynamoDB and Scylla have the capability to scale, we believe in giving you the most efficient performance possible, getting the highest utility of the underlying infrastructure you’re running on.

3. Openness: Scylla is open source. Users can review the source code and any known defects, and in return can add their own contributions to the project. Operationally, Scylla can run on any suitable server cluster regardless of location (on-premises, in any private or public cloud, or on our own DBaaS, Scylla Cloud) or deployment method (bare metal, containerized, virtualized, or deployed in pods via Kubernetes). This contributes to the user’s lower TCO by allowing deployment flexibility in line with their existing operations.

Scylla uses popular open source projects for its monitoring stack (Grafana, Prometheus) making it easy to use in or out of AWS, or to integrate to a 3rd party monitoring system like DataDog.

The Current Status of Alternator

Alternator is available as part of Scylla Open Source. Originally made available with Release 3.2 as an experimental feature, it is now tested and ready for production deployments as of Release 4.0.

Our alternator.md and design doc provide detailed information of what’s supported and not yet supported today. In short, most standard applications will just work. The JSON HTTP API is mostly implemented, indexing works, multi zones are implemented, and many more features will work.

Roadmap to Open Source Integration

You can run Alternator right now with Scylla Open Source on your own Amazon, Azure and Google Cloud instances, using deployment methods like Docker and Kubernetes.

We have and will continue to harden the code, ensuring its production quality by putting it through our robust quality assurance cycles. We also continue to strive for completeness in implementing any API differences.

Scylla Open Source 4.0 also supports production-ready Lightweight Transactions based on the Paxos consensus algorithm, allowing Alternator to support the equivalent of DynamoDB conditional updates.

Scylla Open Source 4.0 also provides our streaming feature: Change Data Capture (CDC), which is currently experimental. We may add compatibility with DynamoDB streams later on.

Alternator on Scylla Cloud — Our Managed Cloud Service

For many lean organizations, one of the most attractive features of DynamoDB is utilizing it as a completely managed Database as a Service (DBaaS). So we also released a beta version of Alternator on our Scylla Cloud which you can try today. It currently runs on AWS, and you can utilize your own account to keep your existing AWS terms and discounts. Future updates to Scylla Cloud will run on Azure and GCP as well.

Enterprise-Grade DynamoDB Alternative

In the future, we will also offer a fully standalone Scylla Enterprise release containing the Alternator software.

Kubernetes Operator

We have a beta version of our Kubernetes operator so you can fully deploy and manage a DynamoDB-compatible database wherever you wish.

Migrating from DynamoDB to Scylla is Easy!

To make the switch as easy as possible we’ve extended our online migration tools to be relatively simple: just start streaming the changes from DynamoDB plus run a full scan. The Scylla Spark Migrator project has been enhanced to support the DynamoDB-compatible API.

Contributing to Project Alternator

Scylla and the Alternator interface are true open source projects. We encourage those looking to contribute to review the source code and any known defects, and read our guidelines on how to add their own contributions.

Developers looking to become open source contributors to the project are further encouraged to join the scylladb-dev mailing list, or to visit ScyllaDB’s Slack.

Resources

Alternator Webinar

ScyllaDB co-founders Avi Kivity, Dor Laor and Distinguished Engineer Nadav Har’el gave an in-depth webinar (including demos) of Scylla Alternator. Click below to watch the recorded webinar.