Stream Processing with ScyllaDB – No Message Queue Involved!

At Palo Alto Networks, we process terabytes of events each day. One of our many challenges is to understand which of those events (which might come from various different sensors) actually describe the same story but from many different viewpoints. Traditionally, such a system would need some sort of a database to store the events, and a message queue to notify consumers about new events that arrived into the system. We wanted to mitigate the cost and operational overhead of deploying yet another stateful component to our system, and designed a solution that uses ScyllaDB as the database for the events *and* as a message queue that allows our consumers to consume the correct events each time.