Founded in 2008, AdGear has been at the forefront of digital advertising technology for quite some time. The Canadian company’s online ad serving platform compiles vast amounts of user data, which it uses to bid on the ad placements that online exchanges auction off in real time.
In the ad serving business, time is of the essence. As David Haguenauer, Software Engineer at AdGear, explains, “We have to be able to decide quickly and communicate with the exchange if we want to participate in a given bid. Auctions typically complete within 100 milliseconds.”
AdGear had been using Cassandra as the back-end for its bidding gateway service, which performs about 1,000,000 bids per second. They had 31 Cassandra nodes across 3 data centers serving regional traffic for the western United States, eastern United States and western European Community.
From a database perspective, the most critical factor is read latency. The cluster must sustain traffic while keeping a read latency consistently in the low milliseconds. Haguenauer explains that “we have only 10 – 15 milliseconds to wait for the database to retrieve user profiles. If we have to wait longer, it’s too late — we’ve already lost the opportunity to bid.”
AdGear liked Cassandra but they were experiencing too many read latency spikes, either due to higher loads or for reasons unknown. “We hit a wall with Cassandra. We weren’t able to tune it to behave in a more reliable way,” said Haguenauer. “If it was too slow that meant lost business to us.”
“We realized we could just put it where Cassandra was and everything would keep working.”
– David Haguenauer, Software Engineer, AdGear
“Scylla was the natural choice for us since it’s fully compatible with Cassandra,” said Haguenauer. “We realized we could just put it where Cassandra was and everything would keep working. We started our transition to Scylla without rewriting any of our stack.’
Since this smooth transition, AdGear has seen significant performance improvements.
With Scylla, AdGear is able to serve the same amount of requests with about half the hardware. Each Scylla node serves a bit more than twice as many queries as each Cassandra node did. At peak traffic the effective read latency fell from 21 milliseconds on 31 Cassandra nodes to 13 ms on 13 Scylla nodes. More importantly, Scylla latencies are much more stable over the course of a day.
However, the measurement that hits closest to home for AdGear is time-outs, when a database read is so slow that it must be discarded. At AdGear, a time-out means they failed to participate in a bid. “With Cassandra we were unable to respond to 1 in every 7 bid requests. With Scylla in place we’re able to bid 95% of the time — and when we’re not able to bid it’s not because of the database.”
As an added bonus, moving to Scylla also improved the bulk insert process for AdGear, who are now able to load user data directly into the database without a time-consuming intermediate step and without causing spikes in read latency. This has also enabled them to simplify their client-side code.
“Scylla was perfect for our use case,” said Haguenauer. “We save money on hardware and we get much lower and more predictable read latencies.”