What’s next for data-intensive applications? Join us at ScyllaDB Summit 2024 on Feb 14-15. Free + Virtual. Register now >

Google | Building Recoverable (and optionally Async) Spark Pipelines

Holden Karau, Developer Advocate, Google23:58December 18, 2018

Have you ever had a Spark job fail in it’s second to last stage after a “trivial” update or been part of the way through debugging a pipeline to wish you could look at it’s data or had an “exploratory” notebook turn into something less exploratory? Come join me for a surprisingly simple adventure into how to build recoverable pipelines and have more debuggable pipelines. Then join me on the adventure where in we find out our “simple” solution has a bunch of hidden flaws, how to work around them, and end on the reminder of how important it is to test your code.

Share this

Video Slides