VIEW VIDEO

Google | Building Recoverable (and optionally Async) Spark Pipelines

Holden Karau, Developer Advocate, Google

23:58December 18, 2018

Have you ever had a Spark job fail in it’s second to last stage after a “trivial” update or been part of the way through debugging a pipeline to wish you could look at it’s data or had an “exploratory” notebook turn into something less exploratory? Come join me for a surprisingly simple adventure into how to build recoverable pipelines and have more debuggable pipelines. Then join me on the adventure where in we find out our “simple” solution has a bunch of hidden flaws, how to work around them, and end on the reminder of how important it is to test your code.

Share this
Share on facebook
Share on twitter
Share on print
Share on email
Share on linkedin

VIEW SLIDES

Let’s do this

Getting started takes only a few minutes. Scylla has an installer for every major platform and is well documented. If you get stuck, we’re here to help.