CASE STUDY

Building Data Store to Support 60 Million Online Shoppers

About Mogujie

Mogujie is a large fashion retail site, with more than 3 billion yuan (approximately US $430 million) in turnover, serving over 60 million online users shopping for clothes, shoes, bags, accessories, and beauty products. With the amount of traffic produced by the online shoppers, a resilient, low-latency database is critical.

The Need

The requirements for the database solution are straightforward, sustainable ~50,000k transactions per second (TPS), smooth scaling, and high availability.

Apache Cassandra or HBase? Meet ScyllaDB

Our use case required good performance for mixed read-and-write workloads while providing architecture with no single point of failure and high-scalability features. HBase was, at that time, missing those features.

Apache Cassandra was the most logical choice but when we tested a 6-node cluster of 24 cores each, we got the following results:

  • TPS maintained at 30k ~ 40k. When increased to a higher rate, we experienced dropped mutations
  • After 2 to 3 days of continuous operation, Apache Cassandra would crash

“Thanks to Scylla’s superior performance over Cassandra we saved operational and capital costs. More than a year in production, Scylla is serving 600,000 requests per second.”

– FengLin (MengYe Shen), Mogujie

We found ScyllaDB by chance while trying to resolve the above issues. One of our team members discovered an open-source C++ implementation of Apache Cassandra, called Scylla. We were excited! At that time, we had a limited number of adequate servers to meet the requirements. We only had six machines, but after deploying Scylla on those machines our performance SLA requirements were met! Scylla delivered 100,000 TPS. Instantly our resources and efficiency issues were solved!

Hitting hardware limitations and solving them

After the trial with Scylla, we found its performance has exceeded our expectations, but in the process we use, we found that when we increase query rates up to 300,000 TPS, they will yield a time-out. Our initial suspect was the network card, but once we start using netdata monitoring, we discovered that in fact, our time-outs were the result of disk input/output operations per second (OPS).

When the rate of arrival 300,000 TPS, disk utilization has reached 100%, which leads to Scylla latencies. We followed ScyllaDB team’s recommendation and moved from spinning disk to SSD. Trying to delay the move to SSD a little longer, we decided to optimize our data modeling, updating the schema from

create table sentry2.datapoints (
key text,
time timestamp,
data FROZEN ,
PRIMARY KEY (key, time));
to

CREATE TABLE sentrydata.datapoints (
metric bigint,
tags bigint,
wholehour bigint,
data map <int, frozen double, bigint, bigint, double, double>>>,
PRIMARY KEY (metric, tags, wholehour));

inserting 360 data points at a time, allowing 100 million data points per minute. With this update, the bottleneck moved from disk to the network card.

Summary

Once we had Scylla up and running, we took several steps to remove bottlenecks and improve performance. Scylla enables us to provide a low-latency, cost-effective solution without compromising on high availability and no single point of failure. Our nodes volume spans 10TBs per node, which is unparalleled in the industry. We have been running Scylla in production for more than a year and we’re extremely satisfied.