Monster Scale Summit

Summary: Todd Coleman introduces Yieldmo’s migration journey from DynamoDB to ScyllaDB to cut costs and enable multi-cloud flexibility. Yieldmo’s ad platform relies on sub-10 ms, high-throughput database lookups to power real-time auctions across billions of daily transactions. After evaluating DynamoDB, Aerospike, and caching, Yieldmo selects ScyllaDB for its DynamoDB-compatible API, controlled server requirements, and cross-cloud deployability. A phased migration using Spark and Kafka ensures data consistency, and production clusters deliver half the cost and lower latencies

Topics discussed:

• How Yieldmo cuts database costs and cloud dependencies by migrating from DynamoDB to ScyllaDB
• What database requirements drive the design: ultra-fast, single-digit-millisecond reads, high-throughput writes, scalability, and reliability
• Why DynamoDB’s geographic latency and rising costs limit true multi-cloud deployment
• What alternative architectures were considered: caching layers, Aerospike’s in-memory indexing trade-offs, and code change overhead
• How ScyllaDB’s DynamoDB-compatible API and modest server footprint enable minimal-code migration
• How Spark and Kafka pipelines orchestrate bulk data transfer and real-time write synchronization
• What load-distribution challenges emerged, from DNS round-robin workarounds to upcoming built-in node balancing
• Why instance family selection (I3en vs. I4i) and TTL management shape cluster scaling and performance

Takeaways:

• Prioritize API compatibility to reduce migration effort: adopting a database with a DynamoDB-compatible interface cut Yieldmo’s code changes to days rather than weeks
• Leverage existing message queues for migration: Kafka offsets snapshotting and replay provide continuous data consistency during bulk transfers
• Evaluate network latency across regions before designing caching strategies; geographic proximity to the database can invalidate cache benefits on cache misses
• Plan for scaling in atomic groups: ScyllaDB requires three-node increments and careful TTL governance to prevent storage hot spots and ensure smooth growth
• Select compute instances optimized for local SSDs (e.g., AWS I4i) to maintain sub-10 ms read latencies under heavy load

Top takeaway: Choosing a DynamoDB-compatible, multi-cloud database like ScyllaDB delivers sub-10 ms latency at scale, halves operational costs, and lets you deploy flexibly across clouds.