See all blog posts

Equifax and ‘if it works, don’t touch it’

Learning from the Equifax breach

I guess it should come as no surprise that the Equifax breach could have been prevented. This time it was Equifax but next time it could be you. While it’s fun to see someone else take the hit, the heat, and watch people mock the CIO’s education:

Equifax

The analysis reveals that a failure to patch a two-month-old bug led to massive Equifax breach. Such an issue can happen to 99.9% of companies. Here’s why:

“Most breaches we become aware of are caused by failure to update software components that are known to be vulnerable for months or even years,” René Gielen, the vice president of Apache Struts, wrote. Struts is the faulty software component that brought Equifax to its knees.

“This vulnerability was disclosed back in March. There were clear and simple instructions of how to remedy the situation. The responsibility is then on companies to have procedures in place to follow such advice promptly,” says Bas van Schaik, a product manager, and researcher at Semmle, an analytics security firm. “The fact that Equifax was subsequently attacked in May means that Equifax did not follow that advice. Had they done so this breach would not have occurred.”

Now, with a hand over your heart, do all of your servers run the latest and greatest releases?

We at ScyllaDB have way too many open source users who choose to run ScyllaDB 1.4 and 1.5. They are ancient! Databases need to be secure even if they are behind the firewall. You’d like to guard and authenticate the access to your precious data and sometimes may even want to encrypt it as well.

Most Operating Systems are way behind the latest updates. Spending four years on Red Hat’s core team taught me that no software is safe from CVEs (Common Vulnerability Exposure) and this is actually the number one thing attackers look for. Only a fraction of users run the latest RHEL/CentOS releases and many compromise their security by running older copies or by using non tier-1 vendors.

Cutting-edge risk vs Security risk, who has the upper hand?

A common claim is that running a cutting-edge release is risky and can be the source of bugs, downtime, and even a security risk on its own. While a new release imposes some level of risk, mainly about bugs and downtime, I claim that it’s nothing compared to a security breach risk caused by unpatched software.

Usually, developers who run older software (any day beyond zero-day fix) are fed by FUD at best or most likely, too busy or not professional enough. CentOS 7.2 is almost two years old and you’ll be surprised how many CVEs are relevant to it (RHEL/CentOS are just an example and are among the top Linux distributions). The slightest hesitation or delay in applying a new release can materially hurt your company’s reputation and your job. I just wish my SSN won’t be disclosed with it ;).

Can one safely run on cutting-edge software?

First, we’re not really speaking about cutting-edge software. Most open source enterprise software is a fork of the open source release that had stabilized for months or longer. So more accurately we’re speaking about the stable-cutting-edge. That’s much less fear-inducing and far more confidence inspiring.

The key is to be able to run software updates with an agile process on a regular basis. If you have a Continuous Integration/Deployment environment you’re already 90% there. The more often you do it, the less costly every update becomes. You should gain the ability and the mindset to apply updates as soon as they are released, test them initially in a separate environment (trivial in the cloud/container world), and then apply them to the rest of your production infrastructure. ScyllaDB supports rolling upgrades and its feature discovery kicks in only once all the cluster nodes are running the new software. This is a topic for a full blog which I’ll queue for later.

Yes, there will be times when an issue will escape your sandbox or will hit the production environment. But you should ask yourself, would such an issue, even resulting in an hour of downtime, be worse than the Equifax effect?

Happy Yum update.

 

 

About Dor Laor

Dor Laor is co-founder and CEO of ScyllaDB. Previously, Dor was part of the founding team of the KVM hypervisor under Qumranet that was acquired by Red Hat. At Red Hat Dor was managing the KVM and Xen development for several years. Dor holds an MSc from the Technion and a PhD in snowboarding.