Monster Scale Summit

Summary: Avi Kivity reviews scalability in ScyllaDB. He weighs horizontal against vertical growth, pointing to 192‑vCPU / 120 TB nodes that keep clusters near 100 machines. He then covers Seastar’s per‑core design, a 50‑50 memtable‑cache split, global materialized‑view indexes, a small pool of heavyweight vector‑search nodes, and one CDC stream per shard to prevent hotspots.

Topics discussed

• Why horizontal scaling ensures geo‑distribution and fault tolerance while vertical scaling shrinks operational overhead
• How 192‑vCPU Amazon/AMD storage instances enable single‑node capacity of 120 TB, reducing cluster count
• How Seastar’s thread‑per‑core model keeps every core busy, trading some efficiency for utilization
• Why a 50/50 memtable‑to‑cache split balances LSM write amplification with read speed on big nodes
• How global materialized‑view indexes avert anti‑scaling seen in local or storage‑attached designs
• How vertical‑scale vector search with a few huge, possibly GPU‑backed nodes beats per‑tablet partitioning
• How one CDC stream per shard maintains throughput without hotspot contention

Takeaways

• Large nodes (>192 vCPU, >100 TB) paired with roughly 100‑node clusters cut failure frequency, paging load, and upgrade pain while still meeting geo‑distribution needs.
• Seastar’s per‑core sharding shows that modest efficiency sacrifices can yield full‑core utilization, which is essential once servers cross 100 cores. Adapt similar per‑core designs in latency‑critical services.
• On big LSM nodes, split RAM evenly between memtables and cache; this keeps write amplification low without starving reads and is insensitive to small tuning errors.
• Favor global secondary indexes and a small pool of heavyweight vector‑search nodes; per‑node or per‑SSTable indexes anti‑scale as clusters or tablet counts rise.

Top takeaway: Running ScyllaDB on a manageable set of very large nodes—each with hundreds of vCPUs and 100 TB‑class storage—delivers near‑linear throughput without blowing up latency.