Bringing QUIC to Seastar

TL;DR For a University of Warsaw student project in collaboration with ScyllaDB, we built a working QUIC transport for Seastar (the engine that powers ScyllaDB). We implemented it on top of the sans-I/O ngtcp2 library and adapted Seastar’s RPC layer to run over it. The benchmarks show a bounded, predictable cost on a lossless loopback, and a clear advantage once the network starts dropping packets.

Since 2019, ScyllaDB engineering has been collaborating with University of Warsaw students – mentoring them as they work on the ever-growing queue of optimizations. This year, we took on a transport-layer project: integrating QUIC into Seastar, the asynchronous C++ framework that ScyllaDB is built on. We decided to pick this up for two main reasons. First, the depth of the project, which required us to strengthen our networking skills and also deeply understand the environment of the project. Second, we were intrigued by creating a feature that could possibly improve an existing implementation.

While working on the project, we encountered quite a few difficulties, from figuring out the best approach for RPC implementations to wrestling with modern C++ and its many wonders. This post shares a detailed walkthrough of our project.

Background: why even bother with QUIC?

Seastar nodes use TCP protocol for communication. After all, it’s considered “old reliable” for a reason. It’s well documented and has been proven to be the best solution in the majority of cases worldwide. Given that, there are still a few architectural “issues” which may slow down the exchange.

• Head-of-line blocking: TCP delivers a single, strictly ordered byte stream. That means that if one packet is lost, everything behind it waits (including data from completely unrelated requests), until the retransmission arrives. When one connection requires many independent operations between two nodes, a single dropped packet stalls all of them.
• Handshake latency: new TCP connection additionally requires at least 1 round trip before connection establishment to enable TLS encryption (older version TLS 1.2 requires 2 RTT).

QUIC, the protocol underneath HTTP/3, in theory avoids these problems. It runs over UDP and moves most of its features out of the kernel and into user space. Given that there is freedom of implementation. However, most libraries still follow the standards of RFC 9000, which include:

• Independent streams: one QUIC connection carries many logical streams that are delivered independently. A packet that’s lost on one stream no longer blocks delivery on the others – so QUIC removes transport-level head-of-line blocking between streams.
• Encryption built in: TLS 1.3 is part of the QUIC handshake rather than a separate layer on top, which cuts the round trips needed to get a secure connection running. The protocol also supports 0-RTT resumption, although its usage is situational and our project hasn’t covered aspects of it.

QUIC, besides being a fairly new technology, already carries a large share of internet traffic as the transport under HTTP/3. It’s still being actively improved as companies including Google, Meta, Microsoft, and Apple run it in production.

These properties make it an interesting alternative to TCP and proving that bringing it to the Seastar library is worth a shot.

Why Seastar makes integration troublesome

Seastar, as an advanced framework for high-performance applications, has certain architectural properties that cannot be bent in order for it to work as intended. The most obvious one is its shared nothing concept: each thread has assigned a CPU core for its event loop. Threads do not share any memory, so costly synchronization mechanisms such as locks and atomic variables can be avoided. Given that, it’s obvious that a potential QUIC library CANNOT reintroduce the locking mechanism or context switching that framework exists to avoid. Besides, Seastar also heavily relies on its asynchronous model when it comes to e.g. I/O operations. We cannot afford to starve the core waiting for prolonged I/O tasks to finish so our QUIC library also has to be sans-I/O (networking operations such as opening sockets, handling sending/receiving data up to the user).

Choosing the best engine for our needs

Before we went searching for an engine that would meet our rather specialized needs, we made a checklist. The detailed criteria are described in this deck; the most critical criteria were:

• No hidden I/O
• No hidden threads
• Must conform to the IETF QUIC standard and implemented QUIC features.
• Must be an actively maintained project

This helped us rule out most candidates fast. MsQuic by Microsoft runs its own worker threads and owns its sockets. Google’s quiche had no clean sans-I/O design and conflicted with our build system. Another group (Cloudflare’s quiche, neqo by Mozilla, TQUIC by Tencent) is written in Rust, which would have meant a foreign-function-interface boundary and a second TLS stack inside a pure C++ framework.

The closest miss was lsquic, which offered fast, sans-I/O and no threads. The problem was that its packet-output callback expects the application to report synchronously – from inside the callback – how many packets were sent. In Seastar, sending a packet is an asynchronous operation owned by the reactor; it must never block inside a callback waiting for it. We could have buffered packets internally and sent them later, but that would have required us to build a fragile layer on top of assumptions about how lsquic retries unsent packets. Also, it preferred BoringSSL while Seastar uses GnuTLS. So we ruled that out too.

The options narrowed down to ngtcp2, which was an almost perfect match. It’s a low-level, sans-I/O state machine written in C (without threads, locks, or networking). It also ships without built-in cryptography. Instead, it exposes callbacks for plugging in a TLS library. Since Seastar already links GnuTLS, we could reuse the cryptographic stack that the framework already depends on. It also provides decent documentation and is actively developed by the well-known and respected Tatsuhiro Tsujikawa. In short, ngtcp2 gave us the protocol primitives and stayed out of the way of everything else.

Getting it into the build was a small project of its own. We introduced ngtcp2 as a git submodule and wrote a CMake module so the ngtcp2 target resolves automatically for anyone building Seastar. Along the way, we hit a naming collision. ngtcp2’s CMake options were too vague and may have introduced collisions with Seastar CMake variables. We fixed this issue by prefixing them with the NGTCP2 keyword.

Making a state machine work with Seastar

A lot of the project time went into dealing with a mismatch between two execution models. ngtcp2 is synchronous and callback-driven: you feed it bytes, it mutates state, and then calls back into your code. Seastar is asynchronous and coroutine-driven. That means every operation is a future, driven by the reactor and owned by a single core. A lot of the engineering went into bridging the two together.

Some examples…

One actor owns the protocol state. The public API never touches the ngtcp2 object directly. Each operation (send these bytes, open a stream, return receive credit, close the connection…) becomes a command on a queue. A single per-connection actor is the only code that drives the state machine, reads packets, and flushes UDP datagrams. Other coroutines just enqueue intent and await a future. Since ngtcp2 has strict ordering requirements, funnelling everything through one owner keeps it correct by construction.

Streams look like ordinary Seastar sockets. A QUIC stream is exposed through Seastar’s standard input and output streams, and we wrote an adapter that turns a bidirectional stream into a connected_socket, which is the same type the rest of Seastar’s networking already uses. That single adapter is what later let us run the entire RPC stack over QUIC with almost no changes:

// Client side: open a QUIC connection, take one bidirectional stream,
// and present it as a type the rest of Seastar already understands.
connection session = co_await client.connect(std::move(cfg));
stream s = co_await session.open_stream({ .type = stream_type::bidirectional });
connected_socket socket = to_connected_socket(std::move(s));

auto out = socket.output();
co_await out.write(sstring("request"));
co_await out.flush();

Routing happens in user space. TCP gives you one kernel socket per connection. QUIC multiplexes many connections over a single UDP socket. Before decryption, our server parses each datagram’s Destination Connection ID from the header and looks it up in a map of live connections. If the ID is known, the datagram goes to that connection’s actor. If it’s unknown and the packet is not an Initial packet, then the datagram is dropped and there is no key to decrypt it with anyway.

Reconciling two flow-control systems. QUIC has its own credit-based flow control and Seastar has its own memory-bounded backpressure. The two had to work together so that 1) a fast producer is slowed when the consumer falls behind, and 2) queued memory stays bounded, without deadlocking when a single write is larger than the whole budget. That’s a case we hit and had to correct.

Dealing with this involved things like:

• Streams half-closed in one direction but live in the other
• Peer signals such as STOP_SENDING that are normal lifecycle events rather than failures
• stream-ID exhaustion that must be retried rather than thrown
• A rule that a secure-randomness failure fails visibly instead of silently falling back to a weaker RNG.

Ultimately, we learned that a QUIC transport for Seastar cannot be a thin wrapper over ngtcp2. It needs a real adaptation layer. The value of that layer is in keeping the complexity contained, so the code above it never has to see it.

Putting RPC on top: two approaches

Seastar’s RPC subsystem, which lets ScyllaDB nodes communicate, was written assuming a TCP socket underneath. We adapted it twice (on purpose) to compare two approaches.

The one-to-one approach is straightforward: leave RPC as it is and replace the TCP socket with a single QUIC stream presented as a socket. Thanks to the connected_socket adapter, the RPC logic (request framing, reply matching, compression, timeouts, cancellation, streaming RPC) never learns that the transport changed. This validates compatibility. However, since the whole RPC session still rides one stream, it does not fully expose QUIC’s inter-stream head-of-line-blocking benefit.

The QUIC-Aware approach uses QUIC the way it is meant to be used. One connection is shared between two peers, a single control stream handles negotiation once, and then every RPC call gets its own fresh bidirectional stream. This is a design that can actually remove head-of-line blocking: each call rides an independent stream, so a packet lost on one call cannot stall the others.

The one-to-one adapter reuses the existing socket-oriented RPC machinery, including streaming RPC coverage. The QUIC-Aware approach validates ordinary calls, no-wait calls, concurrency, ordering, compression, timeouts, cancellation, connection lifecycle, control-stream negotiation, and stress cases over per-request QUIC streams.

So what about performance?

Benchmarking performance

We benchmarked QUIC at a disadvantage: a single machine, a lossless in-order loopback link, and a single Seastar shard. This setup removes every advantage QUIC was designed for. There’s no network latency to amortize a handshake over, and no packet loss for stream independence to help with. All we’re really measuring is raw per-packet cost.

We compared three transports running the same RPC stack: TLS/TCP (the baseline), one-to-one QUIC, and QUIC-Aware.

All main benchmark numbers below come from a single-shard release build (GCC 15.2.1, -O3 -DNDEBUG) on one Ryzen 9 5950X host over IPv6 loopback, built against ngtcp2 commit c131c76d and GnuTLS 3.8.13. Each row ran for 10 seconds and was repeated 30 times. TLS/TCP and the one-to-one QUIC adapter were measured from branch rpc/quic-v1 (commit 76c4cb7), and QUIC-Aware RPC from branch feature/rpc-on-quic (commit e40b7404). Packet-loss experiments used tc netem with five repetitions per loss level.

For round-trip latency on small messages, results are additive. Replacing TCP with the QUIC adapter adds a fixed ~130 µs per call. That offset is the same for a zero-byte ping and a 1 KiB echo. The overhead depends on the number of calls vs their size: a per-call cost, not a per-byte one. QUIC-Aware doesn’t really add anything more at small sizes, and opening a fresh stream per call is cheap for small payloads.

(median round-trip latency, single connection, no pipelining)

For throughput on request/response (echo) traffic, the QUIC variants stay within a bounded factor of the heavily optimized TCP stack: at worst around 3.6x across the echo matrix. The clear outlier is fire-and-forget (discard) traffic. Here, TCP can push 1 KiB requests as fast as the byte stream accepts them while QUIC pays per-packet encryption framing on every datagram. The one-to-one adapter is about 10x below TLS/TCP there, while QUIC-Aware falls much further (to about 3.1 MB/s versus 454.8 MB/s for TLS/TCP) because it opens a fresh QUIC stream for every fire-and-forget call. The per-call-stream model also degrades throughput at large payloads: at 64 KiB, opening a fresh stream whose flow-control window has to ramp up from its initial value pushes QUIC-Aware latency to roughly 1.7x the baseline.

On a perfect loopback link, then, QUIC-Aware incurs the overhead but has no chance to show its main advantage. So, we added packet loss.

We re-ran one configuration over a loopback impaired with tc netem a fixed 0.5 ms delay and packet loss at 0%, 1%, and 5%. TLS/TCP is fastest with no loss and slowest at 5%.

(throughput, echo, 1 connection x 4 in-flight calls, 4 KiB)

TCP falls to a quarter of its throughput at 5% loss: one dropped segment stalls the whole byte stream. The single-stream QUIC baseline degrades more gradually, which is consistent with QUIC’s userspace loss recovery reacting better in this setup. QUIC-Aware degrades least of all. It holds 76% of its throughput, overtaking TCP at 1% loss and the QUIC baseline at 2%. This is the head-of-line-blocking avoidance the lossless loopback cannot demonstrate. With each call on its own stream, a lost packet stalls one call while the others continue.

The per-call-stream design therefore pays an overhead on a perfect link and earns it back once packet loss enters the picture in this controlled setup.

Current status and what’s next

We set out to see whether QUIC could live inside Seastar without breaking the model that makes ScyllaDB fast. It can. The standalone transport exposes Seastar-style streams, can be adapted into a connected_socket shape, and keeps protocol complexity behind an asynchronous API. The RPC work shows both a compatibility path and a QUIC-Aware path that opens independent request streams on one connection. Across the controlled echo benchmarks, the cost is bounded and predictable rather than open-ended (although the discard result shows that the per-request-stream model is not free).

The QUIC transport API has been prepared as a contribution to the upstream Seastar codebase. It is a working, validated integration, but it is not yet a production transport. Please treat what’s presented here as controlled implementation results. They were obtained with one host, one shard, IPv6 loopback, synthetic workloads, and shared self-signed TLS material.

Some future work:

• Measure it over real network links where QUIC’s advantages can actually show
• Scale it across all of Seastar’s cores instead of just one shard
• Tune its transport parameters per workload
• Ultimately, drive ScyllaDB’s node-to-node traffic over QUIC-Aware RPC to see what it does for a real database under real load.

That last step is the one we are most curious about.

***

This project was built over the academic year by four Computer Science students at the University of Warsaw: Kamil Dalidowicz, Stanisław Kalewski, Adam Karaczewski, and Piotr Korcz, under the supervision of dr hab. Jakub Pawlewicz, as part of the University’s ongoing collaboration with ScyllaDB. Thanks to the ScyllaDB engineers who reviewed our designs and talked us out of more than one wrong turn: Mikita Hradovich and Paweł Pery.