How we accidentally wrote our own FFI framework when developing a driver – a wild ride into modernizing a decade-old legacy codebase while dodging undefined behaviour and chasing maximum throughput.
In our previous post, we already explained how we forged our own FFI framework in the fires of inter-language alchemy to connect the managed world of C# with the highly performant Rust ecosystem. This blog post focuses on why we needed that framework in the first place. It was actually a big part of the ScyllaDB One Driver To Rule Them All initiative. The C# driver, originally developed over 10 years ago by DataStax, was next in line for a long-awaited ScyllaDB update. Our goal was not only to bring the correctness and performance improvements that the ScyllaDB Rust driver provides, but also to keep the API (mostly) unchanged from the officially supported driver.
We’ll cover how we utilized the C#<->Rust FFI framework to modernize the C# driver, make it easier to maintain, and – most importantly – maximize performance.
How to Connect Two Languages?
Connecting two vastly different languages – managed, garbage-collected C# and performance-oriented Rust – naturally seems like it would involve considerable effort and overhead. While the first part is true (as we explained in our previous post), it turns out that raw P/Invoke and “reverse” P/Invoke function calls are very cheap despite crossing the Foreign Function Interface (FFI). By utilizing improved interoperability capabilities introduced in .NET 5 (e.g., the UnmanagedCallersOnly attribute or unmanaged function pointers), we managed to take advantage of the speed of the Rust driver directly from C#…with minimal additional costs.
Keeping our Rust library in sync with the C# code was a different kind of challenge. For a loooong time, we had to manually build and link the native library before every test run. This led to some “fun” debugging sessions. Nothing was actually wrong with the code, but the Rust library was out of date. We wasted so much time that we can officially call it a hobby. In the end, we automated the building and linking process directly in C# .csproj files. Now MSBuild handles the logic, and we can stop worrying about forgetting simple things.
Starting Building Blocks
We started out with a simple proof of concept that contained FFI abstractions from the ScyllaDB CPP RS Driver. For example, it had BridgedPtr – a pointer to Rust-owned data (adapted from CassPtr) parametrized by a lifetime, pointee’s type and mutability (exclusive vs shared). C# objects are passed as a raw IntPtr and opaque for Rust; they are never dereferenced in Rust and are used only to be passed back to C#. Structs originating from C# must have an identical corresponding Rust struct; they can be safely stored in Rust only if they are released from the GC’s reign of terror.
We also took an experimental approach to bridging the asynchronous runtimes of .NET and Tokio. A naive approach would be to block the Rust futures and return everything synchronously back to C#. This approach would be a good idea if we wanted to support only the sync subset of the driver’s API. However, our ambitious goal was to bridge the async API of the driver, maximizing the performance. We needed a way for C# to await a Rust future without any thread sitting idle (see the Sharp Edges of Asynchronicity section in our previous post).
There are many more rules for safe FFI that were included in the PoC. Some of them were refined later and are described in detail in the other blog post. But just to mention a few here:
- All raw pointers should be packed in a thin wrapper to enforce strong typing and lifetime checks.
- Panics never unwind across FFI.
- Pointers to C# objects are never dereferenced in Rust.
- All passed structs must have a matching layout in both languages and be marked with proper attributes (C#:
[StructLayout(LayoutKind.Sequential)], Rust:#[repr(C)]).
FFI Philosophy: Callback Hell Heaven
We’ve already covered passing simple types through FFI. However, sometimes we need to pass complex values from one language to another. When it comes to complex types and data structures in Rust and C#, everything’s organized differently. Converting a complex type, especially a recursive one (!), poses a significant challenge.
We wanted to avoid allocating any temporary intermediary data; the goal was to create end C# objects directly from Rust and vice versa. This led us to the tight coupling model, where it’s common for one language to call the other and the other then immediately calls back.
An LLM explained the idea quite well:
It’s an inversion of the usual “caller hands data to callee” principle: the side that owns the buffer (Rust) drives, and the side that owns the data (C#) fills on request. It’s a shape we reused elsewhere in the bridge — whenever one language can’t afford allocating the other’s objects, it hands construction back across the boundary instead of reaching over.
Differing Driver Architectures
Both Rust and C# drivers offer their own set of APIs that do not always match. One of the major differences in architecture between C# and Rust was the entry point of database interaction. In the DataStax C# driver, the user had to create a Cluster object that serves as a manager for used sessions and handles control connection to the server. This allowed users to query for metadata before a session was even created. On the other hand, the Rust driver does not provide a similar abstraction, and the user has to create and manage a session manually. To maintain compatibility with the previous driver, we decided to keep the cluster, but the control connection is handled by each session separately.
DataStax C# Driver Architecture
Rust Driver Architecture
Our C# Rust Hybrid Driver Architecture
Serialization
Serialization plays a key role every time you send a request to a ScyllaDB node. It translates the query parameters’ values (supplied as C# objects) into a binary protocol that’s understood by the database.
The High-Level Idea: C# initializes a Rust-managed contiguous buffer once, then streams serialized parameter bytes directly into it.
How it works:
- Initialization: C# calls Rust to set up a request builder and allocate a contiguous memory buffer.
- Control Transfer: Rust returns control to C#.
- Serialization: C# iterates through its values, serializing each and calling Rust to copy the raw bytes into the buffer.
Our first decision was choosing where to perform serialization: C# or Rust. Reusing the existing Rust driver machinery seemed ideal at first, but it quickly proved to be an uphill battle with no real upside.
Because the C# driver supplies query parameters as generic Object types – ranging from simple integers to nested maps – we would have to pass type metadata across FFI into Rust. That’s why we decided to keep serialization on the C# side to reuse the driver’s existing serializers.
It is a natural assumption that the serialized bytes should live on the heap. We considered two ways to approach this:
The first one, in line with the Rust driver’s design, was an incremental buildup of the Rust’s internal buffer through its native builder. The C# side would serialize the values into bytes, allocate them, and call into Rust to copy them and append into its native buffer.
An alternative could be to allocate a byte buffer large enough to store all the concatenated serialized values at once. As tempting as this looked, it would require invasive surgery on the Rust driver’s internals, complicating maintainability with any future changes.
In contrast, we also played around with allocating directly on the stack using C#’s stackalloc. While tempting, pure stack allocation was too risky. Unbounded types like maps or sets can produce byte arrays of possibly unbounded size, which would be a recipe for disaster when attempted to stackalloc.
We even attempted a hybrid approach: stack-allocating small primitives while falling back to the heap for larger structures. That way, almost all of the users’ performance-critical queries would require zero intermediate mallocs and memcpys.
We decided to move forward with the first approach, allocating each value individually on the heap, prioritizing providing full functionality first. If the performance wasn’t good enough, we could shift to one of the other proposed solutions (preferably the last one).
We also wanted to avoid pointer chasing. Stashing the Rust builder on the heap and passing its pointer back to C# would force Rust to dereference that pointer on every single value. Instead, Rust allocates the builder right on its own stack during the initial setup call, then hands a pointer to that builder over to C#. As C# loops through and serializes each value, it simply passes that stack pointer – along with the raw byte buffer – back to Rust.
Connecting to the Database
We decided that each C# session will hold a reference to a Rust session and delegate queries there. This approach allowed handling most of the session’s complicated logic by the Rust driver (like query execution and connection pooling), but resulted in a particularly nasty bug: a TOCTOU (Time-of-Check to Time-of-Use) race condition when closing a session. If a user thread checked whether a session was open and proceeded to execute a query, another thread could close the session in the milliseconds between the check and the actual execution, resulting in a UAF (use-after-free) in the native layer.
Our first instinct was to put a lock around the session lifecycle on the C# side. For a query to execute an FFI call to Rust, it would have to first acquire the lock and later release it in the finally block. This solution had a major flaw, as we quickly realized. We overlooked the fact that returning a Task is not the same as completing it! The moment the FFI call was initiated and C# generated the Task object, the execution flow exited the try block and hit the finally block, releasing the lock. Meanwhile, the actual native operations were still running in the background. The session could be disposed of and freed while the native database thread was still actively using it (Hello segfault, my old friend…).
To resolve this, we moved the synchronization logic entirely into the Rust layer and introduced a RWLock (Read-Write Lock) on the native session. When a thread wants to run a query, it calls try_read() on the lock. This is completely non-blocking. If try_read() fails to acquire the lock immediately, we can reject the query on the spot. When a session shutdown is called, a thread calls write() blockingly. This requests exclusive access, ensuring that the teardown sequence waits until every single ongoing query finishes its read operation.
After acquiring the lock, the second step is to check the Option value in the session field. If it is set to None, the session was already shut down, and the current thread ceases execution. Otherwise, with the lock held and the session active, we can safely run the query or initiate shutdown.
Transmuting Errors into Exceptions
Since C# and Rust have different ways of reporting errors, we needed to find a way to react to any error that the underlying Rust driver throws at us with a corresponding C# exception. Early on in the development process, we realized that always throwing a RustException on any Rust error is not desirable, and it became necessary to map different errors onto different Exceptions.
To achieve this, we designed a Rust ErrorToException trait.
This trait proved to be very convenient to use. Whenever a new error needs to be mapped, we just add an implementation of a simple to_exception() function.
The ExceptionConstructors is a simple table with pointers. These point to UnmanagedCallersOnly functions that can create specific exceptions. They are then wrapped in a FFIGCHandle and passed back to Rust. When the call stack returns to C#, they can be safely thrown to inform of an error. For maximum speed, this table is allocated once when the .NET runtime starts and is then passed by reference to any fallible Rust function.
Another thing to pay attention to (to avoid shooting ourselves in the foot) was to avoid unwinding the stack across the FFI layer. The rule we came up with: No C# exception can be thrown if Rust is present anywhere on the stack. It is easy to forget to include a try-catch block around all UnmanagedCallersOnly functions.
Likewise, Rust panics default to unwinding the stack. Catching these unwinds would be tedious and easy to forget – that’s why we decided to change the behaviour to instant abort.
Topology Metadata
GetReplicas identifies which hosts store a record for a given Partition Key using a function already handled by Rust Driver (by ReplicaLocator). Our C# implementation required two main architectural decisions.
First, we didn’t want GetReplicas to instantiate new Host objects on every call – our goal was to maintain object identity. We built a global cache of Host objects. C# wraps Rust’s ClusterState as BridgedClusterState, which lets us query the native ReplicaLocator to find replicas and return a list of (Host UUID, Shard) pairs. To map the ids to its own Host objects, C# needs to have HostRegistryies – read-only dictionaries, keyed by UUIDs – and return the Hosts associated with them. For efficiency, the latest BridgedClusterState needs to be globally cached, as well as its HostRegistry. But since the cache could be replaced at any time, the simplest way to avoid race conditions and costly locking was to bundle each BridgedClusterState and its associated HostRegistry into a single ClusterSnapshot, then swap the cached instance atomically.
Second, we needed to ensure eager, deterministic disposal of native ClusterState memory. Relying only on the .NET Garbage Collector wasn’t really a viable option here. From the GC’s perspective, the snapshot was just a tiny managed wrapper, unaware of the actual size of the memory held by the ClusterState. To mitigate that, we decided to implement manual reference counting mimicking Rust’s Arc, which required balancing two conflicting SafeHandle behaviors:
Dispose()decrements the ref count and disarms the finalizer, but its idempotency silently swallows duplicate calls,DangerousRelease()strictly decrements the count, but leaves finalizers armed.
The fix was to split the roles. Exactly one primary reference disposes through SafeHandle.Dispose() (closing the handle and disarming its finalizer, once), while every clone releases through DangerousRelease (a pure decrement). That way, each CloneByRef increment is matched by exactly one decrement, and the count reaches zero precisely when the last reference drops.
Schema Metadata
We decided to redesign access to C# schema metadata. In the original driver, the implementation was inconsistent across different endpoints. Some data was cached during the initial fetching of the object, while other data was lazily retrieved when accessed. That meant that when a user modified a keyspace, some information was updated while some remained stale.
Because of this inconsistency and the severe concurrency synchronization issues caused by maintaining a live, mutating view across FFI, we chose to treat schema metadata as a snapshot at the moment it was accessed. For endpoints returning keyspace/table names, it was straightforward. The same goes for TableMetadata, since it was a simple object with only properties and no methods. However, an issue arose when we wanted to bridge KeyspaceMetadata, which exposed methods to access its tables, views, UDTs, etc., by fetching fresh metadata on demand.
Eager bridging of all information required by this object would be slow and inefficient since the user rarely requires all the data contained by the keyspace. Another solution would be to delegate each call to the Metadata class to lazily retrieve them. However, that would ruin the snapshot approach that we were trying to achieve (since data acquired that way would always be fresh).
In the end, we decided that the KeyspaceMetadata object would hold a reference to the BridgedClusterSnapshot from the moment it was created, and use it to fetch the data required by the user if necessary.
Logging
Relegating work to the Rust driver exposed a new challenge: vital logging information became trapped in Rust’s stdout, completely isolated from the C# logging mechanism. For a long time, we relied on Rust environmental variables and stdout to debug whatever issue we came across…and it was time-consuming and cumbersome. To fix this, we built a custom forwarding layer for the Rust tracing crate that bridges the two.
Whenever a Rust logging event is triggered, it is automatically forwarded to the C# logger. Now, users of our driver can simultaneously control both the log verbosity and storage location for both Rust and C# logs – either directly within their application or via environment variables.
If we just want to have the logs output to console, we can run the program with:
We even added coloring so it is a pleasure to look at.
Benchmarks
This all looks great on paper: fast inter-language communication, minimal additional memory allocation, close to zero overhead on the “happy path.” We really expected to see some performance gains. We did.
We measured the performance of our driver compared to ScyllaDB’s fork of the DataStax C# driver and to the underlying Rust Driver. In all cases, our driver is at least as fast as the original DataStax driver fork. In some cases, especially heavily concurrent workloads, our driver is almost twice as fast as the DataStax C# driver fork.
Environment and methodology
All benchmarks were performed on a dedicated, high-performance machine named POTWOREK to ensure consistent results and minimize external variability. We utilized a locally hosted, three-node ScyllaDB cluster managed by CCM (Cassandra Cluster Manager).
To ensure stability, each benchmark was executed using the ScyllaDB driver benchmarker with the following configuration:
- Repetitions: 3 runs per scenario to ensure stability
- Scalability: 4 steps with multiplicative number of queries
- Concurrency: Concurrent scenarios used 100 workers performing operations in parallel, while sequential benchmarks focused on single-task execution.
Benchmarking suites
We categorized our benchmarks into three suites to evaluate the driver’s performance:
Write performance
These benchmarks measure the driver’s write throughput by performing a total of N prepared statements that insert simple (uuid, int) rows into a single table.
Read performance
The read benchmarks consist of N full-table selects of a 10-row table with the same schema as the write benchmarks.
Serialization/Deserialization
These benchmarks test the cost of handling complex types (the exact schema of the table is uuid, int, timeuuid, inet, date, time, tuple<text, int>, udt, set<int>, duration) during database requests.
The results demonstrate that our new Rust-backed C# driver is a strong candidate for replacing ScyllaDB’s fork of the DataStax driver, performing at least as fast as the baseline in all tested scenarios. Most notably, in high concurrency workloads the new driver shows much better performance, with results being not that far off the baseline set by the Rust driver.
Can I use the driver?
The driver is available on GitHub: https://github.com/scylladb/csharp-rs-driver.
Its development is temporarily paused, but the plans are to continue it once we finish work on the Node.js RS Driver.
We welcome all contributions, opinions, and issues about missing features!
