BSidesSF 2018 - Managing Secrets in Your Cloud Environment (Evan Johnson • Maya Kaczorowski)

[Music]

hi hello everybody see so you're probably wondering why are we qualified to talk about this it's amazing this giant slide it's so cool okay so I'm Evan I work at a company called segment which is a platform for helping you manage data infrastructure and customer data and I'm a security engineering lead there and I'm I am a product manager at Google I work on container runtime security but previously worked on encryption so I care a lot about secrets quick disclaimer nothing that I say is an endorsement of any particular solution it's important so what is the secret what are we talking about well the secret could be a lot of things it's an applique it's anything that an

application needs that build our runtime some common examples are credentials API keys username password combos a really popular secret type is like your stripe API key it gets access to your payment system and it's pretty critical if you were to lose it or if an attacker got access to it something that's not a secret your entire application code all of your config file there might be parts of it that are secret but probably the whole thing is not a secret so today we're gonna talked about how to do secrets management and why there's really no excuses at this point in time to do a good job of it yeah I also think it's kind of important to note that I

feel like secrets management was a big thing that blasted onto the scene in like 2015 2016 and there's a bunch of talks back then and like we're doing hallway Connaught here 30 minutes ago people are still coming up to me saying oh yeah we're still working on that like we still need a solution for that they didn't read your article they didn't read the article and so it's I think we we have a pretty between the two of us we've kicked the tires on a ton of different secrets management solutions and I think we like have a pretty good idea about what works and what you should be looking for when you actually go and solve this at Raqqah excuse me at

your company cool so there's a quote here which is secrets are the very root of cool so obviously we're working on cool thing yes super cool so I like to think about secrets in the context of my company segments which like we are all on AWS we use a ton of different SAS providers and I think just to talk about why management of Secrets is so important like we have AWS keys and we have stripe keys and I think everybody can everybody knows that those are really important but we also have like hundreds of different SAS tools that we use how many secrets do you guys have we have actually in the thousands so if you look

at it it's like eh just I have a list here HR software payroll software blogging software community site POC tech Salesforce slack send us Optimizely github and that's like just barely scratching the surface so it's it is a big problem and so I also really think that all of those secrets are supposed to be treated really sensitively there's no such thing as like this is super high value versus this is super low value because you really want to enforce a positive behavior pattern with engineers that like they don't have to distinguish between important and unimportant secrets and I think like these are examples of companies that maybe this is too large of a slide for you guys to see

but it's an example of companies that have made mistakes with their secrets management like trust Co a few months ago it's like the private keys shouldn't be accessible by the CEO even though they shouldn't have private keys in the first place there's like you shouldn't be storing AWS keys in a public s3 bucket I think we all know that one so it I think it's yeah the examples are there there's like a large bucket of examples to choose from so so how people think about secrets today we'll see this wonderful line graph that we've made of different options and the line graph is supposed to sort of represent how many people I would guesstimate her in each category

so one end of the spectrum you have people keeping secrets in code yes this still happens then some people with those in config files or environment variables at the low point you go people are like oh I'll put them in a storage bucket and it's like well if you're already putting everything something everything somewhere is central why don't you actually get some additional functionality out of it and so you see what's happening over here which is centralized secret management with deployment managers or environment variables sorry secret objects like NS key or committee secrets and secret management tools like hasha core fault as your key vaults confidant key ways etc the real important message here is

that people are shifting from the left to the right they're seeing that they have secrets in code and decentralized that it's hard to manage it's hard to keep versions consistent across multiple places and they're choosing to centralize everything there's really no point at this time to keep things decentralized and you're like wait people so people are still keeping secrets in code it's like yes yes they are please please stop doing this there was a talk yesterday on private keys and Android apps and it was really good you should go watch it I think he scanned 1.8 million apps and found 6000 private keys please don't put your secrets in code so what are we looking what have we

learned in terms of what not to do putting secrets in code hopefully if I just say it another ten times you'll take that message away not rotating secrets so Evan will get a bit more into this but this is a trend that we're seeing in the industry right now where it's actually fairly simple for you to change your secrets regularly so you should be doing it not backing them up they're important it's great that they're super secure but if you lose them you're probably screwed and not having a strong concept of identity so so you need something that can actually authenticate and and that's right that can be authenticated and authorized to access those secrets and the last one

here is about differential protection if you protect the seat your secrets the same way you protect everything else in your infrastructure what's what's the point of them being secrets you're not actually using your resources to the best your ability and doing like / risk-management cool so these are I think when you look at most of the really high-end secrets management and popular secret's management tools out here these are the properties that are kind of emerging as the most important ones the two that I care a lot about one is identity I've seen people try to create a secrets management tool without a concept of identity of all of their infrastructure services and hosts and it

just does not work I've watched it happen and it just was a bad tool and it didn't work and it wasn't secure so you really need the concept of identity and that's really the foundation of authentication and authorization and auditing you have to know when secrets change when when they were created who created them who who fetched them last and all sorts of different just data points about your secrets to respond to incidents and understanding the blast radius of issues that arise yeah and to talk about the other three properties here encryption ideally you're encrypting all of your data but probably you want to be encrypting your secrets at the application layer if you're worried about exfiltration right if

somebody can get in and your data's encrypted a lower layer but they can still get to those secrets and plaintext that doesn't really protect those secrets the second one here rotation I talked about really briefly so you probably want two different things here you want the ability to rotate things on a regular basis as part of like a general best practice I want to change my secrets once a month or whatever it happens to be and then this is the one that people don't test which is suppose everything's been compromised I need to be able to quickly go change all of the secrets of my infrastructure and that's the kind of thing you want to be doing

as part of like an annual testing exercise but that's actually pretty hard to do right now in most environments and the last one here isolation one part is kind of the concept of separation of duties right the people who are managing the secrets shouldn't be the same people who are using the secrets and similarly where you're storing the secrets is probably hopefully not where you're using them that goes back to the idea of centralizing all your secrets somewhere making it easier for them to manage but also meaning that if one application is compromised making all of your applications compromised so what do we what are we looking for well it really depends on the environment that you're in and since

we're at besides SF and we're super super cool you're either in containers or you're in the cloud right like that's that's normal right who's who's some on some public cloud in some fashion yeah okay like 40% in the room and who's in containers maybe 30% the same group who got told they're gonna use containers soon that's usually that's most of the conversations I have now people are like my managers manager told me that I have to use kubernetes and I'm scared from this lift and shift it'll go fine we should talk but so you're probably looking at a couple different things if you're not in the cloud and not in containers having a standalone secret

management solution that is independent of all those things is great or if you're in multiple clouds or in multiple environments you want something that can be completely platform agnostic we'll talk a bit about the tool-house record vault that that fits that category pretty well if you're running in a particular cloud very heavily that cloud provider probably for us a pretty good solution and if you're running mostly in containers then you probably want to go with that specific container Orchestrator solution now if you're running in the cloud and in containers because San Francisco then you have a lot of choice you can kind of do whatever you want and and it's it's more about usability at that point in time so

let's talk about this whole standalone secret management thing and the first tool here is vaults so what what is vault and how does it work vault is a pretty popular open source secret management solution by a company called Hashim Corp there's a an open-source version that you can run wherever you like it requires some manual management you have lots of options and flexibility you can dynamically generate secrets such as like database credentials and passwords and things like that and authenticate in a bunch of current ways so how does it practically work well you have a storage configuration which is where so sorry first you're running vaults probably in a VM or in a container okay you have a

storage configuration which stores your secrets at rest encrypted in storage somewhere you have a seal configuration which is how you manage is the unsealed keys for vault at this point I'm you're like what are unseal keys vaults does this thing that when you spin it up its sealed for you to unseal it so that you can actually access the secrets you there's a shimmy or secret sharing scheme to to get back a plain text version of vault no comment no okay next and authentication method so you go back to what Evans I'm saying earlier with a strong notion of identity you need to present some identity to vault for it to know who you are you can

prevent present lots of different identities based on what environment you're running in and an audit device so you're writing all of your logs to something like our syslog or storage or whatever it happens to be and then you're generating secrets so vault lets you dynamically generate secrets using secrets engines like AWS I am credentials database passwords it also has a couple other useful secrets engines one is kv-4 key value just stores a very basic secret has a PKI back-end a secret engine sorry and it has a transit secret engine which like basically encrypts data in line so you're like this is great my oh I have storage seal configuration off method audit device they're all just backends

just think about them as backends you have lots of plug ability here and you can make it work for any environment you want to make it working so again some examples here for if you're running in the cloud you have storage backends sorry storage configurations see what I did there for a Google Cloud spanner which is brand new s3 Azure etc if for your auth methods you have Google i/o Google GCE which is the VM system AWS I am then ec2 are just released a vm one as of I think last week in ballpoint 0 10 and cribben is you can actually authenticate to vault directly using your criminality service accounts and in terms of dynamic secrets like I

mentioned already key value pairs transit DK are probably the most popular options you can also dynamically generate I am credentials for AWS and as of last week involved 0.10 for GCP cool so when is vault for you you're already using vault and you want to use them in the cloud now that's a really good reason to keep using vault you're using multiple clouds that's another very strong reason because you can authenticate it I'll tend to get to it in multiple ways and actually be running a none multiple multiple cause at the same time if you need to and you have some dedicated engineers so we didn't get into it that much but the unseal thing keeping that keeping vault

up and running can be kind of complicated so you probably need to have an image team somewhere who was going to be who's gonna respond to that if fault goes down yeah vault is very heavyweight and if you don't have engineers it's like and your secrets management solution goes down in your services camp boot or your hosts no longer can fetch their secrets it's like it's it is an operational burden that you have to be ready for so what's next so yeah so I'm gonna talk about AWS I've kind of gone all-in on the AWS secrets management game and there's a ton of different ways you can do it there's actually they have a lot of different

services and just in the last two weeks they launched this secrets manager service and that's pretty neat it's kind of a low cost if you have thousands of Secrets like me it's not suit it's like not a it's not certainly not free it'd be a couple hundred dollars a month which is not cheap but yeah the the main services that will help you on your secrets management journey and AWS is kms parameter store and then secrets manager depending on like the details of your environment on AWS so just to hop into it AWS kms key management service this is a API for doing secure encryption and decryption it's actually like super bare-bones you can do some crazy stuff

though like delegate keys to other AWS accounts that can be used by them you can like encrypt you can decrypt you can generate data in the form of a data key you can make AWS handle the customer lifecycle or the key lifecycle by rotating your key every year so it's it's pretty neat and you can have granular you can give you can use I am to grant people granular access to specific key so the way AWS kms helps people with their secrets management is they usually use it with another Amazon service like s3 or DynamoDB and they encrypt their secrets and enforce access control to s3 or DynamoDB using an IM policy for each service and then this is like the scheme

cred stash if you've if you've ever looked into how cred stash works and this is actually a really flexible and pretty easy to implement scheme but you do have to implement it yourself it's not free SSM parameter store this is my favorite and what we use at segments really every every engineer knows what parameter store is and it's working really well it's kind of come out onto the scene in the last year and people really seem to like it it's kind of like the simplest answer you have a secret you want to store it you want to fetch it you want to like limit access by service and isolate it per host or service parameter stores really good for

this it's actually really km/s is simple parameter stores simpler you can save strings string lists and secure strings that's all you can save in parameter store and it's not specifically for secrets but it's for like config as well and whatever whatever else your code might mean what's a secure string versus a string secure string is kms backed so they behind the scenes they'll encrypt it with a KMS key for you so it's encrypted at rest and this actually can is this actually even visible cool it's not for me so the the this is actually the ion policy that every service this is a code in terraform that every service that segment has that this is

the code for every service that every at segment this is a mouthful that uh that actually enforces access control between secrets that actually enforces like one service can't fetch the secrets of another service so based on the service name they get a little like I am policy that has their secrets named prepended to the to this block of the secrets store in parameter store and every single service and host get that at segments and it works really well this is actually what it looks like we've built this command line tool called chamber which will fetch will like put secrets fetch secrets and exec programs after fetching secrets for the program this is an example of me listing all of

our secrets in the Gateway API on stage and so the gateway API is the only service at segment that can fetch these secrets except for me I can also fetch them so it's pretty neat it tells you the version of the secret so if you change this if you rotate this secret or create a new one or whatever you do to it it'll tell you who did it what time and what version you're on you can also roll back or fetch a specific version so you can say okay I want version 2 instead of version 3 and that's pretty neat I think this slide is really impactful into why nobody's heard of parameter store because it's way down here in the bottom

left of the ec2 console like good luck finding it if you didn't know is there you're just not gonna find it but it's super useful it's the most useful AWS service that nobody's heard of in my opinion so this is kind of like the playbook if you're trying to build secrets management all in AWS if you have a significant amount of infrastructure that's not on AWS and you need this super like flexible and secrets management set use vault if you just have a simple setup where you just want to have secrets and fetch secrets and use parameter store if you have ridiculously high load requirements don't use parameter store we've had issues where like we try to boot up thousands of

services that all call the parameter store at once and they get unhappy with us and rate limit us and our services don't boot healthfully and that's bad so you'll want to build something with kms and s3 or DynamoDB and then finally if you really have like rotation requirements and and you're not worried about the cost secrets manager is probably a good option for you cool and on GCP we also have cloud KMS which is our cloud key management service and very similar idea you could encrypt secrets with cloud KMS and then store them in GCS which is our storage service use IAM and cloud or logging to track access to that the recommendation there is actually deterrent on cloud audit

Logging's data access logs so you would actually log every single time that that secret is read there is thoroughly detailed instructions on how to do that at this link but it's not automated so there's no kind of parameter store equivalent or secrets manager equivalent on GCP yet so if you're running on GC p what should you do if you just need your secrets to be like relatively secure and encrypted bonus is that Google cloud platform encrypts all your data at rest anyways so you could just keep it in spanner that's a pretty really popular option if you want to further encrypt things at the application layer follow the instructions from the previous slide and store them and either spanner or GCS

and if you have more complex requirements or you're running on GCP and other clouds and probably hash your core vault is gonna be a pretty good choice so the last area here which is you're running in containers and or you're running in containers and on the cloud so some of you might be familiar with docker swarm Gorham has a so first for people who do not know what containers are you have worker notes that are going to run jobs and you have sworn managers that are going to tell the worker notes what drops to run and that's the basics of container orchestration and the store managers keep a state including secrets in a distributor drafts store and that's

how they keep track of what's going on docker has secrets as a concept built in so and you're like wait a second I use docker I've never seen docker secret before they're not a part of the container the part of the service you have to be using docker swarm not just docker and they're stored in the internal distributed States store the bonuses that they're already encrypted at rest that's great you don't want to do anything else and they have there's mutual TLS between the swarm manager and the workers so your secrets are distributed securely all the way to your worker which is really nice it's actually pretty straightforward and simple on docker swarm and we really

wanted to talk about docker just to show this really cute photo I love that crab so what happens on kerbin any is cumin a similar idea to docker I have nodes that have pods that have containers and I have masters that tell the the containers on all of these pods what to do and I store my secrets in at CD the the only reason I want to call out pod specifically here is unlike in docker where I believe the secrets are scoped to the notes incriminated secrets are scoped to the pods that's a that's a difference towards calling out so secrets originally incriminating were not encrypted at rest as of 1:7 they are so you can have secrets encrypted in at

CD at the application layer using AES CB CAS GCM or secret box modes and you're like that's great but I have a central secret management solution and I don't want to deal with this no problem incriminate E's 110 you have an external kms key that you can use to encrypt the secrets that you store NAT CD there are plugins already available for as your key vault and Google Cloud cameras those are not in karate score so if you're looking for those are not finding them you have to go to the Microsoft and Google github separately and I know that the community is also working on developing one for house record vault cool I also think it's worth noting like

the architecture of the swarm versus the kubernetes like the slides were almost exactly the same except the the name docker swarm was replaced with kubernetes it looked like to me what do you think and I think that's really telling that container orchestrating frameworks and like cloud platform as-a-service are really going towards providing this as a first-class service and a first-class feature of like using docker swarm are using kubernetes and I think people really see that as like a really important feature in a differentiator and I think like this whole ecosystem will get simpler as you further embrace the cloud I agree so so what are we talk about today we talked about a whole bunch of different secret

management options which include hash reports AWS parameter store secrets manager GCP cloud cam as dr. storm and kubernetes and we talked about some of the requirements that we have around those identity auditing encryption sorry it's hard to read rotation and isolation I don't remember what I'm supposed to say here so I think Maya has covered it that's what we've talked about all the solutions that we talked about had some sort of notion of identity auditing and isolation so if you have a specific requirement around encryption or rotation look for that specifically not everything is doing that as easily the second brand-new rotation in particular I think is becoming very trendy the second the second point here is that

basically every solution does a pretty good job and exists in every environment so you're probably going to pick something based on the environment that you're in which is why we've talked about containers and clouds specifically and unusable 'ti so do you not just listen to us but go read simple but how easy or hard it is to use these things yeah usability is key because if you actually want engineers to use it it better be usable and so like coming down and from the security ivory tower and saying like this is the secrets management solution we use like get with the program no never effective in moving security forward so what didn't we cover what's

not currently covered in terms of secret management one is usability we just called out to is a root secret so I mentioned earlier for example hash report fault has this unseal where you do a three out of five similar secret sharing scheme I mean like in kms there's a root key somewhere that's managed by the cloud provider and in you know urbanity is if you're just op encrypting at the application there that root key stops to be stored somewhere it's stored in that CD as well unless you're using the external kms so it's it's still that is still potentially an unsolved problem at the end of the day it's Turtles all the way down so how do

you how do you protect that first secret yeah yeah the you know and this is a good overview of like 20 18 secrets management and I hope 2019 secrets management it's like like rotations bigger and more important and better features around it like what secrets manager just came out and that specifically is supposed to support rotation and besides that just more usability I think that'll really help just better tooling around this whole pulse problem so what do we what do we learn today just encrypt it's that easy put them in code that's bad yeah great does anybody have questions we they're shaking their head no of me because I don't have time but feel free to find me

or Maya alright thank you so much [Applause]