As we prepare for ScyllaDB Summit 2019, we are producing a series of blogs highlighting this year’s featured presenters. And a reminder, if you’re not yet registered for ScyllaDB Summit, please take the time to register now!
Today we are speaking with David Blythe, Principal Software Developer at SAS. His presentation at ScyllaDB Summit 2019 is entitled Changing All Four Tires while Driving an Ad Tech Engine at Full Speed.
SAS’s implementation of NoSQL is somewhat unique. Rather than expose the raw CQL interface common to Cassandra, DataStax Enterprise, and ScyllaDB, you created a C++ API to abstract the underlying database. What was the strategy behind such a decision?
The abstraction is just a good object-oriented practice, shielding the application level code from the particulars of the implementation. It afforded us the ability to create other implementations we use in unit and local testing, with no need for an actual NoSQL database. And, while it wasn’t the original motivation, it has also allowed us to more easily change our NoSQL infrastructure (three times!) without disturbing application level code.
Your analogy to “changing all four tires while driving at full speed” refers to your database migration. I’m imagining a pit crew trying to do that at the Raleigh Speedway. As fans of racing know, the crew makes all the difference to a winning team. Tell us more about SAS’ “pit crew.” Who’s on your team?
First, let me broaden that analogy. A database migration is just one of the tires. A server code upgrade may be another. Turning features on or off for a customer is another. Our ad serving ecosystem has many moving parts, and we have to be able to upgrade or service those parts while continuously fulfilling ad requests. Oh, and a pit stop would be a luxury. We have to do it while barreling through Turn 4.
We have a surprisingly small team of talented engineers who develop and manage the entire system: 3 front end developers, 3 back end developers, 2 testers, 3 operations people, our development manager, and our product manager. There are 2 dedicated tech service people, and they are included in our daily scrums so that we have continuous visibility into customer issues. The developers share on-call duties with the operations staff, so we get to feel the pain first hand if something we’ve created goes sideways. It’s a great organization.
In your presentation, you repeat on a few slides the short but potent phrase, “No down time.” Tell us why uptime is so vital for SAS and your users. What’s at stake?
Our customers are web publishers and, of course, make most of their revenue from selling advertising space on their sites. The product we are responsible for, “CI 360 Match”, manages the delivery of ads into their sites, starting and stopping campaigns at the right time, pacing them to meet their goals, serving them to the right audience–in short, making real-time decisions about which is the best ad to serve “right now”. That’s how our customers maximize their revenue. They cannot afford periods when SAS cannot serve, because it has a direct impact on their bottom line. We have to be running 24/7.
The data we keep in NoSQL also has to be available 24/7. It has the information about the web site visitor–their demographics, behavior, and ad serving history. That data is needed to target the ads properly and make the best decisions. Without it, the ads served have less value to our customers.
Things are a little more relaxed on the front end, but not much. Our customers use the front end to enter ad purchases and set up business rules for their delivery. They also do searches for unsold inventory and generate reports. Down time is not as significant as for ad serving, but if it occurs during their work day, it can really interfere with their business processes. We have to be very careful.
Tell us about SAS’ deployment.
CI 360 Match is a hosted product deployed in Amazon AWS. Ten years ago, when the product began as part of a startup, we were deploying static instances in one US region. Over time, we have spread to Europe, Australia, and Japan. There are no longer static instances for ad serving or front end, but autoscaling clusters. Our use of automation, both within AWS and in our own tooling, has been critical to getting where we are and making the system manageable with a small staff. We’re running 200-300 instances in AWS at this point.
We deploy new front end and back end code weekly. It’s a great pleasure to be in an environment where we can measure time to new feature deployment in days–not weeks or months.
What is one thing that most people in the tech industry don’t know about yourself that you’d like to share?
I like to sing. You won’t find me on American Idol, but I have fun in our church choir. It’s about the only time during the week when all this tech stuff goes out of my head!
Thanks for taking the time to talk with me today, David. I’m sure our users will be eager to attend your session to find out more in depth!
