Finding a Scalable Way to Stop Attackers
by Tony Albanese
August 15, 2014
| 3 min read
Evan Gilman, operations engineer at PagerDuty, recently spoke at a meetup at PagerDuty HQ. The first thing Evan noted, “Security is hard.” Whether you’re a small shop with a constantly evolving codebase or a huge enterprise with many thousands of employees, attackers will keep coming so you need to find a scalable way to stop them.
Secure by default
Evan emphasized the importance of being “secure by default” in relation to file permissions and security privileges. If you put security checks in place, make it a pain in the ass for others to work around it. What good are rules if people know they are bendable? The checks weren’t added for an arbitrary reason – they are there to protect your customer and team. Also, secure everything. Your logs may not have passwords but they may contain sensitive data such as customer information so you need to secure anything.
Be paranoid
As a general rule, Evan noted, you should assume your network is hostile. That’s especially true in the cloud.
“You have no idea what else [is] running on the rack next to you.”
Encrypt all network traffic – both inter and intra-dc – and the success we’ve found is to encrypt at the transport layer. Also, remember to sanitize the data leaving your data infrastructure as well because you can’t trust the provider to watch after what you leave behind.
Automate and distribute enforcement as much as you can
For Evan and the PagerDuty team, automation is about distributed security policy enforcement. Create a centralized ruleset to manage policy and then push it out to individual nodes so they can enforce themselves.
For example, below is a snippet of code we have to distribute enforcement which reads: Cassandra storage on port 7000 should only be accessible by those with nodes with the Cassandra role.
Take action when there’s something wrong
Ultimately, whatever security solutions you opt for will have to be user-responsive – whether your users are people within your organization, the general public or both.
You need to set up monitoring and alerts to let you know when things aren’t going right. Evan suggested monitoring the level of encryption in your data traffic. If you know 80% of your traffic should be encrypted but only 25% are over a given period, there’s something wrong and someone needs to get paged immediately.
Since PagerDuty is distributed across multiple 3rd party providers, we don’t have VPC available to use so we leverage host-based intrusion detection (HIDS) to let us know when there are problems.
The most important advice from Evan? Start today. You’re going to have to do it eventually and by starting now, you can reduce technical debt and help churn out the bad stuff you already have. Watch his talk below:
Want to see more talks from this meetup, check out:
- Defending the Bird: Product Security Engineering at Twitter
- Identify Identity Lifecycles for Cloud App Security
Or learn more about Security at PagerDuty:
