Our Aussie host’s take on P99 CONF around the clock and (now) anytime on demand
P99 CONF… you’ve got to love this conference, which just completed its 6th edition. I’ve been lucky to help host the last 4 of them and I reckon the format is bang on. As a speaker, you get to pre-record and then listen to yourself “live” while the audience asks you questions in chat. You can even “heckle yourself,” as one speaker put it. And as an attendee, you get extraordinary access to the experts (all questions were answered) plus fun/smart/snarky banter with your peers throughout.
Although we didn’t mess with the format, we did try something different with the timing this time. We hosted a dedicated European/India/Asia-Pacific track for those people who have been joining at all hours of the night for the past 6 years. Fair call, we owed you that.
And while the conference continued its focus on tail latency, it was – like practically everything else these days – increasingly about AI. Gunnar Morling built a Parquet parser from scratch with AI. Likewise, Dor Laor built an alternative LRU algorithm for ScyllaDB, also with AI. I think this goes to show that AI is an incredibly useful tool when chasing P99; it doesn’t have to be the “slop” fest many have been forced to endure. I reckon a dozen or so of the 60+ talks touched AI in some way. Some of the most interesting ones to me were about agents doing the performance work in production…more on that in a bit.
It’s not all about AI though. We also saw our usual spread of topics (which our community helps vote for btw) with talks on io_uring everywhere, database internals, eBPF and tracing, networking, sandboxes, lots of Rust, and a little Zig even. Here’s my guide – you can pick your own path on demand!
Let’s talk about IO. Avi Kivity’s keynote on “The Difficult Life of an IO Request” is not some coming-of-age story but the history of asynchronous IO from the 1980s all the way to modern day implementations using io_uring. From there, a whole stack of talks also dived deep on how far you can push IO.
Georg Kreuzmayr’s keynote from TigerBeetle shared a similar journey, this time building an object storage client from scratch in Zig with all the accoutrements: custom DNS, TLS, HTTP, single-thread, zero-copy, io_uring … And in true TigerBeetle style, results were 10x faster than AWS and GCP SDKs. Love that!
Marcin Szopa, Jakub Czyszczoń and Witold Formański showed how Seastar’s new asymmetric io_uring backend hands off IO to dedicated network cores and gives application cores a bit of a breather. Pere Diaz Bou from Turso also broke new ground with SQLite’s single writer bottleneck and row-level MVCC via an io_uring async IO stack. Evan Chan from Conviva walked through a real 14GB/sec pipeline on… you guessed it… io_uring with a dose of Tokio, Arrow, and Rust. And we also heard from Tyson Brown at Thoughtworks, who argued that well-designed write pipelines are more likely CPU bound not disk bound when using modern NVMe with Celeriant pushing durable/replicated writes well past 500K/sec.
Don’t forget Piotr Gankiewicz’s talk on ultra-performant message streaming in Apache Iggy and Jared Hulbert’s talk on Memory is Slow, Disk is Fast if you want to rattle your previously held assumptions.
Agents did a lot of coding this year, and coincidentally, we spared you from a lot of AI-generated talk submissions (thanks Cynthia)! I already mentioned Gunnar Morling and Dor Laor’s keynotes, and like I mentioned earlier, I found the production stories using agents were quite interesting. Rajat Shah from Netflix described an autonomous performance agent hunting inefficiencies across live services, then tracing them to source and proposing fixes along the way. This was a great conversation starter in chat. Agent autonomy was continued with Filipe Oliveira from Redis who gave an agent a whole multi-core cluster plus performance profiles over MCP (and budget) and then basically said, “Run the performance lab, thanks mate!” There were some other AI-driven optimization nuggets in there from Disney. Tomás Senart from Perfloop also made the case for closed-loop performance engineering, and Hayden Stainsby from HERE showed us AI-assisted profiling for Rust services in prod. The philosophical talk was Dominik Tornow’s “The Prompt is the Platform” which is a must-listen to complement the practical ones.
It would be remiss of me not to mention the database talks, which are a mainstay of P99 CONF. ScyllaDB turned up in numbers with Michael Litvak and Felipe Cardeneti Mendes delivering an excellent talk on logstor – a new storage engine for ScyllaDB that skips LSM traversal entirely. If you’re driving super wide columns AND key value workloads, then that presentation is a must-see. Pavel Emelyanov’s talk explained why Seastar’s flat scheduler falls apart as the number of groups grows and how hierarchical scheduling fixes that. Marcin Maliszkiewicz showed us how to survive connection and request storms, which I think every seasoned SRE is curious about. Piotr Korcz and Kamil Dalidowicz also got QUIC running inside Seastar’s RPC and found the sweet spot for packet loss thresholds with multiplexed streams.
The database talks extended far beyond ScyllaDB. Madelyn Olson closed the conference with a keynote on how Valkey gets over vertical scaling walls via slot migration/rebalancing. Raluca Constantin showed us how Aurora’s bets on transaction isolation, optimistic concurrency and time sync can do it all at the commit stage. Almog Gavra asked whether everything can be an LSM and introduced SlateDB for building on object storage. Roman Khavronenko shared many great details, like string interning, caching, and other goroutine tricks that keep VictoriaMetrics humming along.
And even if you’re not building/running databases and just like to nerd out on data structures, Ragnar Groot Koerkamp gave us a great talk on the SimdQuickHeap priority queue, Daniel Lemire and Francisco Geiman Thiesen teamed up on a billion query per second lookup and benchANT tested how far you can push analytics with PostgreSQL. One longtime favorite presenter for me was Benjamin Cane, delivering another entertaining talk about “correctness” at American Express.
We hosted a mini vector conference too. turbopuffer shared how they search 100B vectors with over 200TB of object storage at 1K QPS. Szymon Wasik from ScyllaDB discussed what engineers often neglect: what happens to your vector indexes in real life with an ever-changing index of 100M vectors at the edge of recall/QPS/RAM frontiers? And more importantly, what happens to your P99 when you start to run concurrent search with inserts and deletes? Karol Nowacki continued this with a separate benchmark on native full-text search on ScyllaDB against OpenSearch. And Sam Rose from ngrok also made a compelling case for more fearless quantization.
In contrast, Jubin Soni from Yahoo shared his adventures in vectorless RAG. And related to all this searching were feature stores. DoorDash talked about escaping Redis gossip on RocksDB and Zipline covered low-latency feature serving with high fanout scenarios.
So what else makes a P99 CONF complete? Observability, of course, and eBPF held pole position again. Tanel Poder was back with some very interesting production pitfalls of eBPF itself; Peter Zaitsev showed us how to tell if the network is at fault using eBPF and Ershaad Ahamed Basheer was using eBPF for metrics collection on HPC systems. On tracing, Ivan Goncharov from Azul shared tracegrams, which sparked a lot of interest in chat, and Salesforce casually aggregated 12B spans/hour into something you can actually query. Cristian Velazquez from Uber unlocked Go’s execution tracer and Amos Wenger (fasterthanlime) gave us a great keynote on holistic profiling, which was a great listen.
Networking topics, of course, always have a presence, and Elvis Chidera from Delivery Hero compared HTTP2/3 in production. CDN77 chased nginx throughput problems all the way through to a Linux kernel bug (with fix). Cong Wang covered transparent TCP splicing to kill the localhost tax for colo services. Kishor Yadav Kommanaboina hunted 8ms worth of tail latency through a Redis proxy, and Vahab Jabrayilov got bare-metal tail latency inside cloud VMs with Machnet.
Sandboxes seemed to be a new vibe for P99. Lovable spun up sandboxes for every prompt in under a second. Felipe Huici from Unikraft got sub-millisecond cold starts on a million-scale VM density. Steve Karam from Oxide gave us a healthy reminder about performance having layers; be sure to check that out, and JuiceFS covered how they manage approximately a billion-plus files for AI workloads.
And finally, Rust is still popular at P99 CONF. Great talk from Dylan Martin at PostHog, who rewrote their feature flag service from Python to Rust and Carl Lerche this time brought Rust to web apps with Topcoat. Not only Rust though, Den Odell kept the browser’s main JS thread free to keep a lid on latency and improve the overall experience and IBM captured lightning in a bottle with Java.
So that is another huge P99 CONF, and there is a ton of on-demand content here for you to watch if you couldn’t solve our concurrent stage trilemma.
I absolutely love being part of this conference, so a huge thank you to the team running the whole show – Natalie for the brilliant coordination, Cynthia for finding the talent and answering questions in chat before I’ve had the chance to read them, Felipe and Gui (the lounge crew) for pleasing the demo gods, and most importantly, all the presenters and audience for showing such deep interest and willingness to share your experiences.
Good on ya!
