How we got tokio and .NET’s async runtime talking to each other, over the C ABI
There’s just one ScyllaDB, but there are plenty of ScyllaDB Drivers… For every language you want to write your application in, you need a driver to talk to ScyllaDB. Maintaining and developing the whole herd of drivers has been a tough task for us, the Driver Team.
One day we came up with an intriguing idea, which we called One Driver To Rule Them All: to only develop our favourite Rust Driver and build all other drivers on top of it, as thin language binding layers! It all started with an internal hackathon in 2021, which produced the first prototype of CPP RS Driver (which is already 1.x!). See this talk for more details on the original idea.
On our way to make One Driver To Rule Them All truly happen, we decided it’s time for the C# RS Driver. We already had a ScyllaDB-customized (shard-aware, tablet-aware) DataStax C# Driver fork, but we ultimately wanted to rewrite its API on top of the Rust Driver, as we have been doing with other drivers. After successfully rewriting the CPP Driver and Node.js Driver on top of Rust, we hoped that doing the same for C# would be a similar success story.
The first step was researching existing solutions to bridge Rust and C#. However, we found that, apparently, there’s little available when it comes to Rust <-> C# interop!
Uncharted Territory
What We Found And Why It Wasn’t Enough
Our research led us to two options:
- uniffi-rs-based uniffy-bindgen-cs.
- csbindgen.
Both are just bindgens, meaning that they ensure basic compatibility and reduce some hand-written boilerplate. As a trade-off, they also hide some FFI complexity, which might cause hard-to-debug errors.
We had already had a painful experience with napi-rs: immense inefficiencies, hidden hard-to-debug errors deep in the library, lack of comments, cryptic library code, incomprehensible autogenerated binding glue. This wasn’t quite what we had expected from a well-known cross-language framework like napi-rs.
Because of that, and because no bindgen tools supported async interop between Rust and C#, we decided to come up with something self-baked, drawing on our past experience with CPP RS Driver and Node.js RS Driver.
Merciful .NET FFI Support
We started by exploring what the language and the runtime themselves offer when it comes to FFI. It quickly showed that while all reasonably aged .NET versions provide some native interop capabilities, recent versions introduced important convenience improvements. As a notable example, the UnmanagedCallersOnly attribute, central for our taken approach, was introduced only in .NET 5. That’s why we decided that we would drop support for older runtimes, with .NET 8 being the lowest supported. With such convenient constraint, we got access to plenty of primitives on the C# side that we could build upon.
The following sections cover know-how that we also described in the driver’s Issue #2 – read if you’re interested in more details or different phrasing.
Cross-language calls
The backbone of any FFI is being able to call from one language to another, in both directions.
Both Rust and C# fortunately allow interop with C (the lingua franca of programming languages), so that’s the common ground we met them both at.
While Rust’s extern “C” makes it perfectly compatible with C with no additional limitations, C# – being a managed language with a sophisticated .NET runtime involved (including garbage collector!) – has somewhat more complex requirements and distinct mechanisms.
C# -> Rust
The first mechanism, used in .NET itself to call native libraries, is P/Invoke (Procedure Invoke). Quoting .NET documentation, P/Invoke is a technology that allows you to access structs, callbacks, and functions in unmanaged libraries from your managed code. In our case, it will be used to call Rust from C#.
On the Rust side, we define a pub extern “C” function with name mangling off.
On the C# side, we declare the unsafe static extern function, translating its signature from Rust via C to a corresponding C# representation. So: u8 becomes byte, a pointer becomes IntPtr. We add a DllImport attribute, declaring the calling convention used and providing the name of the native library to link the function’s symbol against. NativeLibrary.CSharpWrapper is just "csharp_wrapper", since the driver wrapper lives in libcsharp_wrapper.so.
This tells C#, “look up row_set_type_info_get_code function in the "csharp_wrapper" native library, and use Cdecl convention when calling it”.
So far so good. Now, the trickier part: Rust calls C#.
Rust -> C#
Let’s first understand why this is harder than the other direction. When calling a Rust function from C#, the function resides as a compiled machine code in a shared library, is properly marked as a symbol in the ELF symbol section, and has a known static offset. It’s thus really simple to call it! Just load the library into memory, find the function in the symbol’s table, then compute the routine’s first instruction’s address (library’s offset in program’s memory + routine’s offset in library’s contents), fill in arguments, and jump.
It’s not that simple at all when it comes to C#.
The first problem arises due to possible lack of the managed context: when calling a C# function from Rust, the executing thread may not be a .NET thread but rather a plain Rust one (for example, a tokio executor). Therefore, C# must not assume .NET’s presence during the function’s call.
The second problem arises from the fact that C# is not fully a compiled language. Even if you make a function a static method of a static class, it does not mean that it’s precompiled and built in an ELF with a known address as it was the case with Rust. What it actually means is that the runtime will JIT-compile the function during the program’s run time, and only then will the function get a stable address.
.NET 5+ introduced a proper solution to the first problem: the UnmanagedCallersOnly attribute. Quoting its documentation: Any method marked with UnmanagedCallersOnlyAttribute can be directly called from native code. The function token can be loaded to a local variable using the address-of operator in C# and passed as a callback to a native method.
The second quoted sentence also hints at how we’re going to solve the second problem: by passing pointers to functions as callbacks. In reality, each C# function that is going to be called from Rust needs to have its pointer passed to Rust first. To be 100% sure that the pointer stays valid for the whole program’s lifetime, we create a static delegate to the unmanaged function.
C# side:
And the Rust part: 
The whole mechanism is (at least informally) called Reverse P/Invoke.
Cost
According to .NET docs, PInvoke has an overhead of between 10 and 30 x86 instructions per call. Wow! Tiny overhead! That’s suitable for many language crossings. Let’s continue reading:
In addition to this fixed cost, marshaling creates additional overhead. There is no marshaling cost between blittable types that have the same representation in managed and unmanaged code. For example, there is no cost to translate between int and Int32.
We’ll get to marshaling soon. For now, it’s important that blittable types are passed between Rust and C# at no additional cost.
Good, that’s for sync function calls. What about async calls? After all, both Rust Driver’s API and a major part of C# Driver’s API are asynchronous.
Sharp Edges Of Asynchrony
Reconciling Async Executors
Rust with tokio and modern C# have a surprisingly similar approach to asynchronous programming. Both languages use async/await keywords and both can spawn async tasks in the background. In both cases, there’s a pool of executor threads that keep pushing tasks until completion, retreating from a task and moving to another if a blocking point is faced.
Drawing upon this similarity, we engineered the following framework to plug tokio tasks into C# Tasks completion mechanisms, allowing for bridged async execution.
Cryptic abbreviations
- TCS (
TaskCompletionSource) – built in C# to allow manual control over async tasks. - TCB (
Tcb, Task Control Block) – a package to manage a task and control its completion from Rust.
The high-level idea of the async bridge is surprisingly simple:
- C# creates a TaskCompletionSource<IntPtr> – the standard .NET class for manually completing a
Task– which perfectly fits our needs. - The TCS is wrapped in a GCHandle (described later in this post) so the garbage collector won’t move or collect it while Rust holds a pointer to it.
- A TaskControlBlock (TCB) is created – our own C-compatible struct containing the pointer to the GCHandle’d TCS and two static function pointers: one for success, one for failure.
- C# calls into Rust via P/Invoke, passing the TCB. This call returns immediately after enqueuing work onto Tokio.
- Rust spawns the actual work on the Tokio runtime. When future completes, it calls back into C# through the function pointers in the TCB (reverse P/Invoke).
- The C# callback recreates the TCS from the GCHandle pointer and calls
SetResultorSetException, which completes the Task
From the caller’s perspective, it’s as simple as await session.ExecuteAsync(statement); the interaction between the two runtimes is completely invisible.
Famine
Note that it’s tokio executor threads that complete the C# Task via TCB. This means that non-.NET threads execute C# code! This is benign if all the work done is setting the continuation of the Task as ready to run. However, .NET optimizes async/await for performance by reducing context switches. To achieve that, by default it immediately runs the continuation synchronously on the same thread. What does that mean for us? Tokio workers can get starved!
While in C# executors can be spawned on demand if the runtime detects deadlock, tokio trusts in the benevolence of its tasks – meaning that they won’t block the executor. Therefore, tokio never spawns more executor threads than a predefined fixed number. Here, we identified a key incompatibility of the models.
Fortunately, there’s a solution in .NET: the TaskCompletionSource has a flag (RunContinuationsAsynchronously) that forces continuations onto the .NET thread pool, preventing tokio executor starvation. We set the flag in RunAsyncWithIncrement, which is our abstraction for running async tasks that borrow an object (in practice, async methods). Remember: without this flag, when Rust calls the completion callback, .NET would run all continuations synchronously on the Tokio worker thread! For more information, see issue #2.
We can now call Rust <-> C# synchronously, and C# -> Rust asynchronously. That’s all we need for our driver use case. Now let’s try to actually pass some data.
Passing Data Back And Forth
Data Marshaling
Type conversion is basically free for blittable types: primitives like integers (int / i32, long / i64), floats (float / f32), and raw pointers (IntPtr / *const ()). These can be passed across the boundary instantly; a 64-bit integer in C# is laid out the same way as a 64-bit integer in Rust.
.NET Marshaling
A problem occurs for more complicated types (like strings, arrays, or structs), which have different memory structures in each language and require conversion.
In traditional .NET Interop (P/Invoke), those differences are handled by a built-in Marshaling Engine. When you pass a non-blittable type across FFI, .NET uses attributes like [MarshalAs(UnmanagedType...)] to trigger the transformation. Unfortunately, it is incompatible with features like [UnmanagedCallersOnly], which require 100% blittable types at run time.
To make passing those types simple and safe, we created abstractions implemented as custom layout-compatible structs on both sides of the boundary. To unify the way those structs are represented, we use the C ABI (C Application Binary Interface) as an intermediary – enforced with the help of #[repr(C)] in Rust and [StructLayout(LayoutKind.Sequential)] in C#. As a result, both definitions occupy the same size in memory with identical padding and can be safely used in the FFI context. Important: we still must make sure the fields on both sides are of the same type and in the same order.
FFISlice
In Rust, a slice &[u8], being a “fat pointer” (purely a Rust concept), cannot be directly passed across FFI, but its raw components can be represented as a pair consisting of a pointer and length. We mirror this exact layout in C# by creating an explicit FFISlice struct.
 |
 |
This allows us to project unmanaged Rust buffers directly into C# as ReadOnlySpan<byte> with zero allocations and zero copies.
FFIString
Builds upon FFISlice, with a type-level knowledge that it points at an unmanaged UTF8-encoded Rust string. Used to pass strings from Rust to C#; C# has a method to convert it to its managed UTF16-encoded string.
FFIBool
At first glance, bool seems like a very simple type with no distinct representation. Both System.Boolean in C# and native bool in Rust take up 1 byte of memory. The issue was that by default, .NET’s P/Invoke mechanism assumes it is communicating with the classic C ABI. In old C/C++, there was no native bool type, and booleans were just integers. Therefore, the default .NET marshaller expands a 1-byte C# bool into a 4-byte integer when passing it across the FFI boundary. Because of that, Rust was passed four bytes when it only expected one, which caused undefined behavior.
Our first solution was to use built-in marshaling and rely on [MarshalAs(UnmanagedType.U1)]bool to correctly handle conversion. This seemed to work fine… until we needed to use a bool in a callback from Rust to C#. That resulted in a Non-blittable parameter types are invalid for UnmanagedCallersOnly methods error. After that, we decided that to avoid any inconsistencies with bool’s representation, we would simply use a byte / u8 wrapped in an FFIBool struct to describe true or false values, with explicit from and into implementations.
 |
 |
FFI Type Safety
When developing CPP RS Driver, we learned that strong typing is the key to preventing nasty bugs in the FFI layer. A fruit of that work was argconv.rs: a set of useful Rust abstractions over weakly-typed FFI, making it strongly-typed and hardening against misuse.
We adapted argconv.rs from CPP RS Driver into ffi.rs in C# RS Driver, and extended it with new abstractions specific for interop with a managed language with GC like C#.
For example, one part of this hardening was to ensure that every pointer is strongly typed, so there’s no risk of mistaking one pointer for another. Some pointers are specific for opaque Rust data exposed to C#, and some for opaque C# data exposed to Rust.
We used static assertions on the Rust side to make sure size of type representation on both sides match.
Cross-language Memory Management
C# Manages Rust Resources
After encountering a lot of memory safety challenges, we realized we needed a reliable way to manage Rust resources from C#. In our case, C# will create Rust resources, use them, and at some point dispose them. The key is to ensure that whenever the GC collects a C# object that manages a Rust resource, that resource is freed. Otherwise, we would have a memory leak!
Writing correct code that handles both automatic finalization (upon GC collection) and manual disposal (a user calling Dispose()) happens to be incredibly hard.
Fortunately, .NET bundles SafeHandle, a primitive designed to solve our exact problem. SafeHandle is the intended way to manage native resources that handles any concurrency edge cases correctly, allowing for deterministic disposal of native resources. It relies on atomic reference counting to prevent premature collection and UAF conditions. All it requires is inheriting from it and implementing IsInvalid() and ReleaseHandle() methods.
RustResource
Let’s use the Session struct as a showcase of how it works. When the Rust driver creates a Session, it is passed to C# along with its corresponding destructor function. We use this pair to create a RustResource object, which inherits from SafeHandle. To prevent use-after-free errors, the RustResource class exposes two functions (one for sync usage – RunWithIncrement, one for async usage – RunAsyncWithIncrement) in which the reference counter is temporarily incremented while the raw pointer is accessed.
When the Garbage Collector determines that RustResource is no longer referenced, the destructor is called, and that ensures memory is not leaked. This orchestration is great because of its simplicity and reasonably low overhead.
Rust Accesses C# Resources
Pinning
The biggest challenge in exposing C# data to Rust was preventing the Garbage Collector from moving or freeing them while they’re still in use. C# objects are heap-allocated. In languages such as C/C++ or Rust, heap-allocated objects usually keep a stable address in memory (unless manually reallocated, think of vector resizing for example). C#, being equipped with an advanced GC, does not provide this guarantee: objects can be moved on the heap at any time! The GC can fire at arbitrary moments and mess up with the pointers. If staying in safe C#, this is not the problem because the GC switches all the managed pointers from the old location to the new one. The problem arises when using unmanaged pointers, which is the only way to pass pointers across FFI. The GC does not trace them, so it does not update them! Now, the usual way to pin memory in C# and prevent moving is to create a GCHandle. It has the following features we like:
- It prevents the pointed memory from being freed,
- (optionally) it prevents the pointed memory from being moved – but this comes with more cost and is generally discouraged,
- it must be manually freed, so GC won’t break our assumptions,
- it has a stable memory address (it’s immovable itself), so we can use unmanaged pointers to it safely.
GCHandle is thus a perfect solution for cases when the lifetime is dynamic and not known up front. Most notably, this is the case of asynchronous calls. However, we noticed it’s overkill for simpler cases when the lifetime is static and known up front – the synchronous calls case.
Stack pinning for sync calls
The key observation we made: if we are inside a Rust function that was called from a C# call stack, we are guaranteed that the whole C# call stack is alive! Well, this is the foundational assumption of automatic, i.e. stack based, memory management. If we add another observation that the stack is immovable (i.e. GC cannot move items inside stack frames), we get a foundational idea of our solution – stack pinning.
Wait, you said stack. But you also said that C# objects live on the heap. How to reconcile these two?
Well, objects do live on the heap, but C# operates on objects via references, which – if kept as variables – do live on the stack*. So: objects can be moved on the heap freely whenever the GC desires, but references to those objects sit on the stack still. OK, great! Then those references are our Holy Grail. We pass pointers to those references to Rust, and whenever we dereference those pointers in the synchronous call, they will still point to our references. The validity of those references, being managed pointers, is guaranteed by the GC itself – they are updated whenever the pointee is moved in memory.
Technically, we do it using Unsafe.AsPointer combined with the ref keyword. If we passed the reference to Unsafe.AsPointer without using ref, we would end up converting the reference to the heap object into an unmanaged pointer to a heap object – leaving us with a potentially dangling pointer (if GC fires and moves the object). With ref, we convert a reference to the stack reference to the heap object into an unmanaged pointer, which is exactly what we wanted.
There’s one catch though: the compiler technically could decide that once we created unmanaged pointers to stack-living references, those references are no longer needed (because they are untracked). It could, for instance, reuse their memory in the stack frame by putting some other data there. We need to prevent it, and for that we use GC.KeepAlive(). This is purely information for the compiler, For stack values, this has zero runtime overhead. It just prevents the compiler from reusing the stack frame slot prematurely.
*Actually, they can also live in CPU registers if the optimizer kicks in, but we can easily prevent it by taking a pointer to the variable. Since register variables cannot be pointed to, the optimizer must leave them on the stack.
Stack pinning demonstrated:
Heap pinning for async calls
For asynchronous scenarios, we use a custom FFIGCHandle. This acts as a wrapper around a standard GCHandle and a destructor function. By implementing Rust’s Drop trait for this FFIGCHandle, we ensure that the handle is automatically freed the moment Rust is done with it. That removes the need for manual cleanup, which is particularly a pain when returning early with an error.
The Framework in Action
We implemented this framework mainly to help us develop ScyllaDB’s new C# RS Driver. A separate blog post describes how it worked and presents the performance evaluation. Spoiler: it’s impressively fast!
