fbpx
See all blog posts

Introducing the Kafka ScyllaDB Connector

Apache Kafka is an increasingly foundational component of enterprise Big Data architectures. Data streaming and event-driven systems have rapidly supplanted batch-driven processes since Kafka was first invented by engineers at LinkedIn in 2011. It was then brought into the Apache Software Foundation (ASF) as open-source Apache Kafka as well as made into a commercial product by the good folks over at Confluent. Over the ensuing decade, the drive for real-time stream processing has seen Kafka spread to tens of thousands of companies, including the majority of the Fortune 100.

Apache Kafka is capable of delivering reliable, scalable, high-throughput data streams across a myriad of data sources and sinks. A great number of our open source users and enterprise customers — like IBM, General Electric, Nauto, Grab, and Lookout — use ScyllaDB and Kafka together. For those looking for more information, you can read about using Kafka and ScyllaDB together to, for example, create a scalable backend for an IoT service.

New Shard-Aware Kafka Connector for ScyllaDB

Because ScyllaDB is an API-compatible implementation of Apache Cassandra, to date users who wished to connect ScyllaDB to Kafka have been able to use the Kafka Cassandra Connector (also known as the Confluent Cassandra Sink Connector). Users could simply swap out their Cassandra database with ScyllaDB transparently with no code changes.

However the disadvantage of that approach was that Kafka performance was not as speedy as could be, since it was not fully taking advantage of ScyllaDB’s underlying shard-per-core, shared-nothing architecture.

To remedy this, we are now introducing a new shard-aware Kafka sink connector for ScyllaDB. This will allow you to use Apache Kafka and the Confluent platform to their fullest advantages.

Using ScyllaDB and Apache Kafka Together

ScyllaDB’s high performance NoSQL database is a natural fit with Apache Kafka. Both support the massive scalability and high throughput required in modern event streaming data architectures.

Users are guaranteed flexibility, either via open source NoSQL database, enterprise-grade or cloud-hosted fully managed solutions for both technologies. For example, consider the following combinations:

  • ScyllaDB Open Source and open source Apache Kafka
  • ScyllaDB Enterprise and the Confluent Platform
  • ScyllaDB Cloud (NoSQL DBaaS) and Confluent Cloud managed cloud services

With the new Kafka ScyllaDB Connector you now have the final missing piece with all the flexibility and performance you need to build the next generation of Big Data applications.

Learn More in Our Webinar

Learn more about the advantages of using ScyllaDB together with the Confluent Cloud platform in our webinar Building a Real-time Streaming ETL Framework Using ksqlDB and NoSQL.

Further Reading

DOWNLOAD THE KAFKA SCYLLA CONNECTOR NOW!

About Peter Corless

Peter Corless is the Director of Technical Advocacy at ScyllaDB. He listens to users’ stories and discovers wisdom in each to share with other practitioners in the industry at large, whether through blogs or in technical presentations. He occasionally hosts live industry events from webinars to online conferences.