Hello readers, or should I say G’day?
I’m excited to share the behind-the-scenes journey, the incredible numbers, and the impact of this year’s conference P99 CONF from the perspective of one of your hosts.
Hosting the conference has been nothing short of a whirlwind adventure for me. From helping to select and review all the content through to the excitement, challenges, and triumphs of hosting a rather large virtual event with a wonderful team – it’s been so much fun.
Firstly, I’d like to comment on the high-quality content provided by so many individuals from around the world. There were over 160 submissions, from which the organizing team had to distill down to around 60 recorded sessions over 2 half days. Those numbers alone speak to what strong an interest there is in the world of low-latency, high-performance distributed computing challenges. Why does this matter so much to me? I’ve been a long-term advocate for performance engineering from different roles, so it’s stunning to see such keen interest in this space. A big thank you to all those who submitted an idea – hopefully the team can keep expanding the conference as a whole, and I’d encourage you to consider submitting again next year.
For the 60 presentations that made it through, what a smorgasbord of information there was! So another big thank you for all the preparation, practice, and performance of your own individual presentations. I’m sure that amounts to days, if not weeks of combined effort, so well done.
The irony of this being a virtual event, with live in-person hosting by myself and Wayne Ariola, was not lost on me. However, secretly, I always love an opportunity to travel, so I really appreciated flying over from Australia to San Francisco to help host live from the production studio. As I mentioned in my opening, I trust that you all found the conference to be bigger, better and way less sweaty than an in-person conference. It was great to see the lively discussions being held in each stage’s chat. To be honest, I think this is the real beauty of this event format. The stress of preparing and delivering content by the presenters has already been overcome, which frees them up to reflect, observe, and answer attendees questions live. Way better than standing around in a line hoping to bend someone’s ear at the end of a talk in a noisy hall!
Watch P99 CONF Talks on Demand
Ok, so let’s recap some stand-out presentations for me. The conference opened up with puppies – err – “Quantifying the Impact of Shard-per-core Architecture” by Dor Laor, Co-Founder of ScyllaDB. What a great start to the conference – this really set the stage for what was going to be a highly technical yet very informative event. I walked away from that presentation feeling guilty about all the times I had hit terraform apply to scale out with more infrastructure, not having thought about the option to simply scale up. ScyllaDB takes advantage of both directions in scaling, with impressive numbers presented and some interesting insights into what’s next on their roadmap, particularly distributing data among shards dynamically with a feature called Tablets. Excited to hear more about that.
Next was Jon Haddad’s presentation “Distributed System Performance Troubleshooting Like You’ve Been Doing it for Twenty Years.” As the title suggested, this was literally 20 years of experience wrapped up in 20 minutes. If you’re new to the world of performance, then treat this like a blueprint for observing performance. If you consider yourself a veteran, then USE it as a checklist to see if there’s anything you missed 😉
Another hidden gem in the instant access area was Kerry Osbourne’s presentation on “How to Improve Your Ability to Solve Complex Performance Problems.” This was more like 40 years of experience in 40 minutes. Some great thinking in this one (both fast and slow) about problems and how we resolve those problems in a deeply technical context. Great to see so many of our cognitive biases mentioned here. Definitely worth adding to your blueprint for success in this space.
Mark Gritter gave an excellent presentation on “The Latency Stack: Discovering Surprising Sources of Latency” from his trials and tribulations solving performance issues over the years. The list was long, and had the usual suspect, DNS (hands up if you’ve been caught by this before) among many other culprits including hypervisors, hardware, and queues to name a few. I truly enjoyed the story telling element to this presentation.
I have grown fond of coding in Rust, so it was great to hear Carl Lerche explore Rust’s potential benefits to application backend development in his presentation “Expanding Horizons: A Case for Rust Higher Up the Stack”. Glauber Costa delved into the challenges of massive data replication to the edge with his presentation “Writing Low Latency Database Applications Even If Your Code Sucks.” That really set the scene for my favorite part of day 1: a live discussion panel with Glauber, Carl and Jarred Sumner (creator of Bun.js). What a trio – I fell away to the sidelines to listen in with awe as these three hashed out the differences, and also what features languages like Zig and Rust might borrow or benefit from each other. To share some behind the scenes, getting three great personalities online at the same time is no easy task. A big thanks to Glauber and the organizing team for helping make this happen over some (cold) pizza and lobby refreshments the night before!
There were many other favorite presentations tucked away in the day (it was a blur to me) but the most memorable was Liz Rice’s presentation on “eBPF vs Sidecars” showing some novel ways to approach the problem of application instrumentation with less fuss. Henrietta Dombrovskaya also gave a presentation on “ORM is Bad, But is There an Alternative?” and it was wonderful to see such strong advocacy between the boundaries of development and databases – something I’ve found difficult to negotiate in the past. I would love to learn more about her use of contracts within the No-ORM Framework.
More great content in the afternoon from Steven Rostedt with “Using Libtracecmd to Analyze Your Latency and Performance Troubles.” This was a deep dive into what happens at the kernel level – not just the scheduler, but also how long locks are held and where and when the interrupts are coming from. Surfacing from the deep was an excellent presentation “Beyond Availability: The Seven Dimensions for Data Product SLOs” by Emily Gorcenski which really cemented my understanding of the hierarchy of all the SL* including indicators, objectives, and agreements.
To wrap up the day, we had a closing keynote from Paul McKenney on “How to Avoid Learning the Linux-Kernel Memory Model.” Paul seems to have a knack for explaining deeply technical concepts at a level I can understand. I only wish I had someone like Paul as my professor at University. Be sure to watch this presentation to get a glimpse at finger measurements of the speed of light at the CPU core level. Fascinating stuff!
There were no major bloopers from the live hosting side that day. Apart from the normal nerves of presenting to an invisible audience of close to 18K peers, it was an incredible experience within the green screen studio. I definitely slept well after day 1. Stay tuned for an update on day 2
Catch the Sessions You Missed, On-Demand
