Key-Key-Value Store: Generic NoSQL Datastore with Tombstone Reduction and Automatic Partition Splitting

Discord’s Senior Software Engineer, Stephen Ma, shares how Discord built a developer-friendly key-value service atop Scylla that hides many database-specific complexities from developers. Discord wanted to shield developers from having to be involved in database specific schema design and schema migrations while still providing powerful datastore primitives that support a broad set of developer needs.

Their solution was a key-value store service that supports storing and querying entities stored as protobufs and identified by composite primary keys. The composite key consists of a parent ID and an entity ID and the service supports range scans of entity IDs within one parent ID. Their “Key-Key-Value Store” also safeguards developers against ScyllaDB performance complexities through a tombstone reduction mechanism and automatic splitting of large partitions.