Building a New Rust Driver for ScyllaDB’s DynamoDB API – with 58% More Throughput

How our new Rust driver load-balances DynamoDB-style requests across a ScyllaDB cluster, and how we extended Latte to measure its performance

For years, Rust developers using ScyllaDB’s DynamoDB-compatible API (Alternator) had one option: use the standard AWS DynamoDB SDK. This interface was designed for a managed service behind a single endpoint – so every request lands on one node while the rest of the cluster sits idle. However, ScyllaDB is designed to put every node and shard to work, which means that this approach constrains its performance.

So, we built a dedicated Rust driver for ScyllaDB Alternator, a driver that’s optimized for how ScyllaDB actually works. It achieves ~58% higher throughput than the AWS SDK driver on a 3-node cluster, with steady low P99 latencies under load. Measuring the driver’s performance required a second project, extending ScyllaDB’s Rust benchmarking tool (Latte) to work with the DynamoDB API as well as CQL.

In this blog post, we describe the development of a new Rust driver for ScyllaDB Alternator and its integration into Latte. This work was performed as part of the Student Team Programming Project (ZPP) at the University of Warsaw in collaboration with ScyllaDB. It’s being further developed by ScyllaDB.

The New Rust Driver

Until now, Rust developers using ScyllaDB’s DynamoDB API (Alternator) interface have relied on the standard DynamoDB driver. It serves its purpose (allowing the developers to communicate with ScyllaDB), but it cannot fully utilize ScyllaDB’s highly optimized architecture.

Building a dedicated driver isn’t uncharted territory. Alternator drivers for Java, Python, and Go already do it, all the same way. They keep the vendor SDK’s API, learn the cluster’s topology underneath, and route each request to a node that can serve it. Rust was next in line, which made it an ideal student project. The plan was to match the AWS SDK’s API so existing applications compile unchanged, fit the routing layer into the SDK’s async architecture, and then profile until the throughput justified the switch. That last step took the most work and led to the three optimizations below.

Internally, the new driver functions as a wrapper built on top of the official aws-sdk-dynamodb crate, which ensures compatibility with the official AWS DynamoDB API and simplifies migration. However, it leverages the SDK’s Interceptor framework to inject optimization logic tailored specifically for ScyllaDB’s architecture and Alternator-specific behaviors. Below we describe the optimizations that the driver currently supports.

Load Balancing

DynamoDB is a managed service: you get one endpoint, every request goes to it, and AWS does the routing itself.

If the same approach were taken in the Alternator client, every request would hit the same node – and that node would become both overloaded and a single point of failure.

The new driver fixes that. It learns the live nodes of the user-configured routing scope (cluster / datacenter / rack) and picks one per request. Currently, two strategies are supported, following the other Alternator drivers:

• Round-robin (the default) rotates through the discovered nodes on each request so load is spread evenly across the cluster.
• Key affinity deterministically maps each partition key to a single coordinator node instead, so it’s compatible with other Alternator clients. This mode is especially recommended for workloads that rely heavily on Lightweight Transactions (LWTs). Here, consistently routing the same key to the same coordinator significantly reduces coordination overhead and contention between concurrent requests.

Header Stripping

The standard AWS SDK adds several metadata headers to the transmitted HTTP requests – headers that are used by the AWS environment, but redundant for Alternator. We ensured they are stripped from outgoing requests, and that reduced the volume of sent headers by 45%.

Request Compression

Also, the new driver allows developers to enable HTTP request compression:

• Gzip and Zlib are currently supported
• Developers can set a body size threshold for compression
• It’s disabled by default

It’s worth mentioning that the new driver is designed to maintain high API compatibility with the official AWS DynamoDB driver. Users can transition by simply importing a different crate and updating the configuration. The following example demonstrates the driver in use, with comments highlighting the few lines that differ from standard DynamoDB usage. Note that you usually don’t replace all imports of the existing driver entirely, as our driver reuses its structs. If you wish, you can always use both drivers in your application. Think of it as an unobtrusive extension.

async fn main() {
    // Import our crate along with the standard AWS SDK
    use alternator_driver::*; // <----
    use aws_sdk_dynamodb::config::*;
    use aws_sdk_dynamodb::types::*;

    // Use AlternatorBuilder instead of Builder
    let config = AlternatorBuilder::new() // <----
        .endpoint_url("http://localhost:8000")
        .behavior_version(BehaviorVersion::latest())
        .allow_no_auth()
        // Optionally configure features like request compression
        .request_compression(RequestCompression::default()) // <----
        .build();

    // Initialize the AlternatorClient using the generated config
    let client = AlternatorClient::from_conf(config); // <----

    // The rest of the API remains identical to the AWS SDK
    client
        .put_item()
        .table_name("ExampleTable")
        .item("ExampleKey", AttributeValue::S("ExampleItemKey".into()))
        .item("ExampleAttribute", AttributeValue::S("ExampleItem".into()))
        .send()
        .await
        .unwrap();
}

The driver also supports overriding global Alternator-specific settings on a per-request basis. By using the .customize() method, users can call .alternator_config_override() with an AlternatorBuilder. This serves as the Alternator-specific equivalent to the AWS SDK’s .config_override() method, which utilizes a Builder to apply local configuration changes.

async fn main() {
    // ...
    client
        .put_item()
        .table_name("ExampleTable")
        .item("ExampleKey", AttributeValue::S("ExampleItemKey".into()))
        .item("ExampleAttribute", AttributeValue::S("ExampleItem".into()))
        .customize()

        // Replaces the standard .config_override(Builder) pattern
        .alternator_config_override(
            AlternatorBuilder::new()
                .request_compression(RequestCompression::default())
        )
        .send()
        .await
        .unwrap();
}

Note that .alternator_config_override() is implemented by the AlternatorCustomizableOperation trait and therefore must be imported either explicitly or with use alternator_driver::*.

Modifying Latte for Alternator Support

The first application of the new Rust driver was adding Alternator support for Latte. With that move, we immediately got practical testing of the driver and enabled benchmarking – which we discuss in a moment. Because we based our new driver on the official AWS DynamoDB driver, we could work on the driver and the Latte modifications in parallel. We then switched to our new driver in Latte when it was finished.

Latte is a Rust-based high-performance, versatile benchmarking tool used for testing ScyllaDB. It is also incredibly flexible thanks to its use of Rune, an embedded scripting language used for writing complex, custom test workloads. However, Latte was originally built with only Cassandra in mind. While most of the code was database-agnostic, certain core fragments and the entire context provided to Rune were CQL-specific. Consequently, supporting Alternator required implementing new logic and refactoring existing code.

So, why did we choose to adapt Latte instead of relying solely on existing DynamoDB testing tools like YCSB? Since Latte is written in Rust, modifying it allowed us to integrate and rigorously benchmark our new optimized driver under realistic, high-throughput workloads. Furthermore, Latte is already the standard tool for testing ScyllaDB, and its performance is unparalleled. Combining the lack of Java overhead with the flexibility of Rune scripting made it extremely worthwhile to bring these advantages to the Alternator ecosystem.

To make Latte support the DynamoDB API, we opted for conditional compilation. We chose this approach to keep our changes as non-invasive as possible. Using Rust traits, by contrast, would have forced us to refactor nearly every file in the Latte repository. We removed the dependency on CQL-specific features from core Latte modules and isolated the database-specific logic into separate cql and alternator submodules. By using Cargo features, we could select which submodule we wanted to compile with and generate separate binaries for latte and latte-alternator.

Using the Alternator API in Rune

To allow for writing Latte workloads efficiently, the DynamoDB interface had to be exposed to Rune. While the native DynamoDB Rust SDK is highly verbose and strictly typed, Latte users can now write short, expressive code to interface with the database. Our API abstracts the native driver functions so they can be easily called directly from Rune scripts.

Here is a short example demonstrating table creation and basic operations in a Rune workload:

// Single primary key with default String type
ctx.create_table("table_1", "userId").await?

// Complex keys
ctx.create_table("table_5", #{
    primary_key: "pk",
    sort_key: #{name: "sk", type: "B"}
}).await?

// Simple puts and gets
ctx.put("table_1", #{userId: "1", name: "test"}).await?
ctx.get("table_1", #{userId: "1"}, ()).await?

You can also take advantage of full support for DynamoDB Sets, manual and automatic pagination for batch operations (batch_get_item, batch_write_item), and request retry handling. For more details and examples of these features, please refer to the ALTERNATOR.md documentation in the Latte repository.

Furthermore, with the integration of the new driver, Latte now exposes Alternator-specific features. Those let you set and tweak the various configuration options of the new driver (such as load balancing strategies, request compression, and header stripping) by passing command line options.

Benchmarks

To verify our implementation, we conducted two benchmark groups: one comparing YCSB with Latte using the default AWS SDK driver, and another comparing our optimized Alternator driver with the AWS SDK.

Latte vs YCSB

In this test, we benchmarked the benchmarking tools to compare our implementation’s performance to that of YCSB. We compared latte-alternator with YCSB running in a c5.4xlarge AWS instance against ScyllaDB running on an i3.2xlarge instance with –alternator-write-isolation=only_rmw_uses_lwt. Both instances were on the same VPC/AZ with ~0.1 ms baseline RTT. The test setup was selected to highlight the importance of benchmarking tool performance, while keeping the costs low. We tested a mixed workload of 50% reads and 50% updates. The Latte workload was closely replicating the YCSB workload. We ran two experiments, the first aiming for maximum throughput and the second comparing rate-limited latency.

Maximum throughput

In this setup, by observing just the YCSB test, one could conclude that the measured throughput was bounded by the ScyllaDB node. After all, it was utilized to almost 90%, while the loader instance seemed to be underutilized. But in the same conditions, Latte successfully saturated the ScyllaDB instance while using roughly half as much of the CPU as YCSB. It turns out that YCSB failed to saturate the ScyllaDB instance, reaching only ~78% of this ScyllaDB instance’s throughput, bottlenecked by the loader’s CPU. This shows that Latte is way more efficient and requires fewer resources to push the database to its limits.

Rate-limited latency

We’ve seen that Latte uses resources better than YCSB in the throughput experiment, but we also had to test how it affects latency measurements. When running with request rate limited to 5000 ops/s, Latte sustained the 5000 ops/s rate, while YCSB reported an average of only 4900 ops/s (even though YCSB had enough resources to sustain the target throughput). The lower reported YCSB throughput is mainly caused by inaccuracy in YCSB reporting and garbage collection overhead. This already makes the measurements less trustworthy than Latte’s by itself. We measured two types of latency: request and cycle latency. Request latency is service time, the time between sending the request and the response arriving. Cycle latency is the time between the intended request start time and the response arrival, which also includes client-side queuing and rate-limiter delay.

Because request latency represents service time, both tools should report similar results under comparable conditions. While their average latencies were close, Latte reported a slightly lower mean and a substantially tighter P99. YCSB’s elevated tail also appears to inflate its average, suggesting that Latte more accurately reflects backend request latency and is more reliable.

For cycle latency, which accounts for coordinated omission, YCSB’s mean latency is over 6 times higher than Latte’s, with its P99 being even 160 times worse. When measurements include the client overhead, the performance and consistency issue is obvious. This shows that Latte has significantly better scheduling and lower tool overhead than YCSB.

Driver performance gains

Multiple benchmarks were run to evaluate the performance gains from our optimizations. Here we compared maximum throughput, latency mitigation by load balancing, and also the impact of hot partitions. For the driver benchmarks, we used a 3-node cluster of ScyllaDB instances running on i3.xlarge instances. Compared to the Latte vs YCSB tests, we downgraded the ScyllaDB instance type so the loader instance could saturate the 3-node cluster. The tests were conducted on a development build that allowed for building with a chosen driver (this is normally not possible in Latte). We compared the DynamoDB SDK and our Alternator driver with both load balancing options described in the Load Balancing section above.

Maximum throughput

The most obvious gain of load balancing is increasing the throughput. While writes eventually involve all nodes in the cluster (for RF=3), reads can be handled directly by the coordinator. Communicating with a single node leaves the remaining nodes heavily underutilized. Our optimized driver successfully saturated the entire cluster for both load balancing strategies. It reached ~58% higher throughput than the AWS SDK driver.

Latency limiting

A less intuitive result of load balancing is its influence on latency. To show this, we ran a test limiting the throughput to 30000 ops/s. That was high enough for the DynamoDB driver to overload the single node, but didn’t push the entire cluster to its limits.

In this test, the latency for the AWS SDK driver greatly increased, while our optimizations managed to keep the latency low – and especially the P99 latency. In reality, even relatively short increases of traffic can affect the latency and load balancing greatly mitigates this problem.

LWT hot partitions

LWT operations require consensus among replicas, so using the same coordinator for a given partition key reduces coordination overhead. This is exactly what key affinity provides (see the Load Balancing section). For these tests, 99% of requests affected 10% of the keys from a small pool of partitions. We configured the ScyllaDB instances with –alternator-write-isolation=always_use_lwt.

In the hot partition scenarios with lightweight transactions, round-robin performs poorly compared to key-based affinity and even the AWS SDK driver, since it increases concurrent contention on hot keys/shards. This showcases the benefit of key-based affinity routing. It performs well under hot partition scenarios and has the load balancing benefits shown in the previous tests.

Impact

The driver has been extensively tested and optimized for performance, proving that it can handle demanding Alternator workloads. That performance work also unlocked a broader outcome: it made it possible to bring Alternator support to ScyllaDB Latte, giving a practical way to benchmark Alternator at scale, track its performance over time, and keep pushing it further.

If you use Alternator, consider measuring this for yourself. Point Latte at your cluster, describe a workload that looks like your real traffic, and see what the numbers say. If your application falls well short of what Latte can pull from the same cluster, start looking at the client. Generic AWS SDK clients talk to a single endpoint and know nothing about your cluster’s topology, so they leave throughput on the table no matter how well the server is tuned. A dedicated Alternator driver is usually the cheapest performance win available, and the Rust driver (the one behind the Latte integration described here) is expected to be officially released soon. When it lands, switching is a small change that could yield a large payoff.