Numberly on Joining Billions of Rows in Seconds with One Database Instead of Two: Replacing MongoDB and Hive with ScyllaDB

Many organizations struggle to balance traditional big data infrastructure with NoSQL databases. Other organizations do the smart thing and consolidate the two. This presentation explores Numberly’s experience migrating an intensive and join hungry production workload from MongoDB and Hive to ScyllaDB. Using ScyllaDB, we were able to accommodate a join of billions of rows in seconds, while also dramatically reducing operational and development complexity by using a single database for our hybrid analytical use case. As a bonus, we’ll cover benchmarks for Dask (a flexible parallel computing library for analytic computing) and Spark, highlighting their differences and lessons learned along the way.