Security Teams & SREs Want the Same Thing: Let’s Make It Happen

Access controls (or, in the SRE world, dependency control systems): You can use access control systems to give you dependency control systems. No one likes it when people and systems touch things they’re not supposed to. It’s amazing how few people realize that, with careful design, we can build one system for both security teams and SREs — and when I show up and offer an SRE team a dependency control system for the low, low price of working with security to roll out better access controls, I become their instant best friend.

Network design: If possible, we don’t want unauthorized folks hitting the access controls. For example, we want to shut down denial-of-service (DoS) attacks (whether from outside attackers or from your own systems because of things like buggy retry logic) before they even take up the resources for a network handshake. And why give potential attackers the chance to look for or exploit a security flaw? Also, Amazon charges a lot of money for that network address translation (NAT) gateway. Running your own proxy, like Envoy, means you get egress controls (which security loves) and you pay less for it. SRE would definitely like to spend that money on something else. Who wouldn’t?

Observability: Both our teams are trying to detect incidents — albeit different types. Tracking an attacker is a bit different than debugging, but the information you need for each of these — such as traffic flow, tracing, and crash dumps — heavily overlaps. Both teams want this to be a comprehensive process, and neither of us wants to do redundant work. 

Releases: Patching — especially predictable patching — is what gets a lot of people in trouble. Production services get hacked because they’re not up to date on patches or are only partially patched. Having a mess of slow and partial patching is a total headache to run or debug, even without attackers in the mix.

Release engineering: Fast, safe releases are how you fix incidents fast — so release speed is a security feature. 

Incident response: Enough said.

Eliminating toil: We’ve both got way more than enough to do, thank you — and need to move fast in a crisis. 

Error budget: SREs have an error budget, which (usually) isn’t zero. One security error is still a security incident. One of the places where this crops up is tiny experimental launches — a tiny experimental vulnerability is still a vulnerability for a security person, and someone who exploits it is still a problem after the launch is rolled back. The attacker might even still be in the system! With that in mind, security teams tend to be a lot more concerned about launches than SREs are.

Measuring: Security can’t always be as good at measuring. A lot of what we’re trying to measure is actively hiding from us, and we almost never get ground truth to work with, so this continues to be a challenge. 

Compliance: Most engineers hate compliance, probably because it’s both inflexible and sometimes seems useless. And hey, I agree. However, we have to do compliance anyway, even if some bits are annoying, because it turns out that at least one of two things are true for most compliance frameworks: that people will give you money for doing it, and that governments will get involved if you don’t do it.

Respect: People are different. Companies are different. This is a feature, not a bug. If we’re going to do a good job at security, privacy, trust and safety, or building any kind of product or system, we need to be solving problems for actual people — otherwise we’re not solving problems at all.

Collaboration: We can elevate each other’s work, even when it’s not immediately apparent why we should be doing it. One way this can work is by raising the priority of projects that SRE teams really want to do, but couldn’t get prioritized previously, and that security can’t do on their own. Security can help by approaching them as joint projects and critical for the company. Dashboards and monitoring can also make a big difference in your collaboration efforts — they help security teams get over their measuring obstacles, and support SREs in gaining helpful visibility into security’s work. And remember: Dashboards are not to inform people or make sure they know things. What you need is a dashboard where when people look at it they do things — like patch their vulnerabilities.

Choosing each other: Many chief information security officers (CISOs) are not engineers and, as such, do not know how to get engineers to want to work with them. If you’re not working closely with your CISO already, I suggest looking at a list of the security priorities and saying, “Hey, I can solve some of your problems.” They would love to team up and help.