Secret Hunting

there we go we're going to talk about secrets we're going to talk about what the heck they are finding them rotating them storing them safely uh it's easy all of you are going to rock at this it's very simple um I would like you to please try this at home and work a lot of people are like don't try this at home I really want you to try this I suspect that all of you will successfully find a secret unless you work at an amazing place that I probably would love to work um so about me I'm Tanya I'm the head nerd at we hack purple I do security Champions secure coding training and stuff I wrote a book about appsec this is year 26 for me in Tech I've done a bunch of stuff but the key takeaway is that I am a nerd on the internet who really likes to talk about securing software that's the key take and when I brush my hair I look very good um so right now half the audience is like everyone knows not to put secrets in the code everyone knows this already uh but uh you know you say that but um you know late 2021 one of my clients was like I know you don't pen test anymore but could you just take a quick look we're going to buy this company and 30 minutes in I was like I have their banking like their legit banking stuff for the whole company ah um so I called the company they fixed it and then the other company bought them for less money but so this is still a giant problem this is still happening all the time and I was a developer a lot longer than I worked in security and I put a lot of secrets in code I have like a lot of work I need to make up for now and so yeah um this happens all the time and so luckily it's super easy to actually find the secrets in the code it's not that hard and so we only have a you know 22 minutes or so so we're gonna go a bit fast bear with me but let's start at the beginning like what the heck are secrets um secrets are what computers use to identify identify each other so when I'm like hey I'm an app and I would like to connect to you database they're like are you really the app I should be talking to I'm like yes I am here you go so we have API actually I'll give you a bunch of examples in a minute but computers don't have eyes they can't look at us and be like Oh yeah I remember you from last year you're okay they don't know these things and so secrets are the way they do that you know when you go into a bar or you go to like a governmental organization and you're trying to do a thing you go to the hospital whatever they check your ID it's similar to that except computers don't really get to have a cute license plate or a driver's license Etc and so examples of Secrets a password or a connection string which is uh the one that I used to put in the code all the time API Keys tokens hashes connection strings anything to say like hey you're supposed to talk to me here I can prove it I'm going to show you this and so finding these in code is bad because malicious actors can also find them they're really easy to find and then they can use them if I was a terrible human I would have been like banking credentials awesome I'm gonna commit robbery but I'm not like that I'm a security person so I just told them right away and we rotated that that night okay so why do we need Secrets like Secrets authenticate to other computers so authentic this is a very beginner talk so if you're like I already know this Tanya that's okay authentication is like are you the real authentic you are you a person that just looks like my friend Sarah or are you actually the real Sarah and so this is how computers talk to each other so when an application wants to you know query a database it sends a connection string and it's like I want this table I want these things this is the level of access I'm allowed PS here's the password let me in and so basically um then the database response sure thing here you go and they let you do stuff and you finish off all the things if they were smart enough to figure this out for themselves we wouldn't need secrets and lots of us wouldn't have jobs anymore computers are pretty dumb compared to humans I know they can do things very quickly but they can only do it as well as we tell them to do it and so you might be thinking why would someone do this well as a person that was a developer for 17 years before she switched to security well I guess I did security a bit anyway the winners I would put it in the code in the comments for the next Dev to help them this is what we were taught in college I kid you not I went to college in the 90s and they're like oh yeah well you know that's nice that you're writing this app Tanya but what about when Brad needs to work on it when you're on vacation Brad needs to know how to contact the database and Duel of these things we still have literally like we would have all the passwords and everything on a big piece of paper and everyone's cubicle so that we would know how to connect we thought we were doing a great job ensuring we gave more reliable service things are in the code because we thought and we were taught to do it that way now we've had many malicious actors use those for evil like I thought I was an awesome Dev my you know client would call me and be like I forgot my password I'm like don't worry I'll look it up for you and then I just like look it up for them I decrypt them like here you go it's this thanks Ann this is this although it's convenient and might make it seem like we're doing better stuff but it turns out that's a huge security flaw and so he's hanging his head in shame um okay and so one thing I want to just say briefly is that ideally when you're gonna store a password specifically so when we have secrets we'll do all sorts of things with them but when you're storing a user password this is not a secret is different than a secret it's for humans it's not for computers we treat it differently and I mentioned this because I showed this to someone else and they're like but what about my users passwords those aren't secrets and you want to take them you want to Salt them which means add a long unique value to the front of it then you encrypt the whole thing or then you hash the whole thing which is a one-way crypto cryptographic uh action so you can never get it back and then you store that in the database I used to encrypt them encryption means you can get it back we don't want it back which sounds kind of weird right but what you do is you get the value from the user you add on the salt you do the hashing and then you just compare to what you have in the database what this means is that if someday your application gets breached uh all of that stuff's completely useless to malicious actors so you might be in have ibimponed.com you might receive an email saying it's been breached but the actual malicious actors they they can't use a rainbow table they can't get that value back because it's salted and tashed but that's not a secret that is a user's a user's password it's not the same secrets are just for computers okay so what is the risk secrets in our code can have all sorts of potential attacks breaches embarrassment malicious actors can use these secrets to talk to your computers with all the authority that that secret holds so just like when I found the banking passwords if I was an evil stealer type of robbery person I would go commit theft I'm not awesome um but if you get an API secret you can then talk to apis you shouldn't be allowed to if you're a malicious actor if you get a connection string you can connect to that database and copy the data change the data erase all of the data there's a whole bunch of things that you can do and so basically all of the things that you've been trying to protect are no longer protected if someone can get those secrets and so none of the outcomes are good from this error unfortunately so what what can we do about it we find the secrets we rotate them and then from then on we use a secret management tool to manage them for us it's way better um so if we put all of this stuff as we find them we change the secret which I'm going to explain why in a minute and then we stick in our secret management tool and we access it programmatically from then on so that means in your CI CD it can say oh there's supposed to be a secret there cool I'm going to go talk to the Vault I'm going to authenticate to it I'm going to grab that thing and now I can put in the password or connection string or whatever the thing is as I'm testing and building your code awesome you can also access it programmatically from within your application it's great human beings don't have to manage this I don't know about you but I don't want to manage Secrets it's boring and hard work okay so yay high fives um so the hunt begins so we want to go and find some secrets so how do we hunt this might sound really simplified and that's because it's actually really really easy I remember thinking this seemed really hard and then the first time I went to do it I was like pop pop pop pop I found tons of stuff and I was like I feel very smart so you download a copy of your code so if you use GitHub bitbucket whatever you're using I just download the whole thing and then I scan it with a secret scanner there's tons of free ones I'm going to give you a giant list and then you sift through the results the boring stuff so I actually suggest that you put on music get like some caffeine get a snack get like a comfy chair and get ready because you're going to spend a few hours going through these results okay so I apologize if you thought oh she's gonna have a sword she's gonna like she's gonna have spirits and guns and she's gonna hunt things it may seem kind of anticlimactic because it's actually so easy compared to all the other appsec activities I would say this is by far the easiest one and if you are junior or new to appsec or you're going to hire an intern or you're going to try to train someone up this is a really good activity that can make them feel useful right away and give value right away and not discourage them because there are some app Tech activities that are really hard this is not one of them so how these tools work so they use something called regular Expressions who here has written a regular expression before yeah yeah like the first time you write one you're like oh it's kind of weird but as soon as you start writing them it's actually really quite easy and so when you like use a regular expression what it does is you write out a pattern and it matches it throughout your entire application or all of your coats all oops sorry I'll go into a folder where all the code is and then you run the secret scanner it's just doing regex against the whole thing regex is short form for regular expression they also search for entropy so entropy in this case means super long random stuff so people are supposed to name their variables things like um username because it's a username password because it's a password you usually name the thing after what it is sometimes you'll add like Str at the end or at the beginning to say it's a string and that's helpful if you see a big long thing that's like 45 characters and it's all just random there's no real words there's numbers there's uppercase there's lowercase that's probably a secret it's probably not a variable name and it is a if it is a variable name you are an unpopular developer at your office because it's really hard to tell what you're doing with that and so it looks for all these long random things and they're almost always a secret keywords it also looks for those like password key API key connection string Etc if it says that it's probably a secret um it's not exactly rocket science so again you might be like oh this talks so easy yes I wanted to write a talk that anyone could understand I want to write a talk that would give you all the confidence to go and try to do this because if you are just starting or especially if you're at a new office and they're like read these documents you don't really have access to anything if you have read access only to the code repository and you have no tools yet no permissions go find a whole bunch of Secrets and you just earned your paycheck so this is a giant list of secret scanners and all of them are free except cred scan from Microsoft and although I think credscan is quite Nifty I'd probably go with one of the 20 free ones on the page instead if you are looking for a secret scanner like all of these are decent I really like truffle hog but mostly because I've used it the most and so any of these will go through and look for stuff Chekov specifically which is on the right side third from the top it looks at infrastructure as code we don't want Secrets there either infrastructure is code regular app code does not matter I don't want secrets in there okay so next false positives kind of suck but you're gonna get some unfortunately so I have a fun little story about this so I was coaching a person to become an abstract professional and I was like we should do this on your first week this will be really easy and I remember he'd never written code before in his whole life and basically we we went through and we found thank you we went through and we found a whole bunch of Secrets but one of them was a male license key and there's this weird email converging software or whatever and we found out uh that that company went out of business 10 years ago and so although it was a license key and normally it would be a secret uh we found out we weren't breaking any copyright and they were using it all over the company and we decided it wasn't a secret anymore and so we suppressed those from then on and so sometimes you're going to find poor coding choices that aren't quite a violation of the secret but basically you should go through and at first tutor the person and just say if you're not sure just come and get me okay I have work to do I'll come see you in an hour and then just check on them every now and again and encourage them um you might also find some scary things I have found some passwords to things that are quite important to the company I was working for where I was pretty darn shocked it was in the code I bet a whole bunch of people in this audience and in this entire building have found tons okay so why are interns awesome I am from Canada you might have noticed my accent where I ended a sentence at some point and said a at the end which means that was a question and the only appropriate answer is yeah um so in Canada we pay interns I know that that might not be true everywhere in the world but we pay them and you can teach a total Noob and by that I mean an intern a co-op student whatever you call them to do this in about an hour you can have them going and you their first week you can just make them do this most the time when you don't have time for them they might not get it perfect at the beginning but very very quickly they will find a whole bunch of things and if they just find one secret they just saved you running your entire incident response process right like you go through oh and also when you find a secret go check to see if you're the second person and not the first person to notice this so assume breach and go and like see if anyone else has done this before okay so tip so someone else gave a talk about this yesterday A Plus to that person you can commit or you can create a pre-commit hook that will then um block new secrets from ever being checked in again so whenever a Dev goes to check something in they just do the regex very quickly and they're like that looks like a secret no sir you need to like try again later okay so rotating secrets if you spill a secret so that means you put in the code or somewhere you weren't supposed to you rotate it no ifs ands or buts um I have this friend that does a live streaming all the time her name sues Hinton she's pretty amazing if you like smart things in programming them she's awesome and she spilled a secret Walsh was live streaming once and then she showed everyone how to rotate it she's like yeah that's right in two minutes you could just rotate it and check it back in and so he's like you rock sues so you if it gets billed you have to rotate it you do not take that spilled secret and put it in your secret management tool you've just wasted all your time so if you app so you must rotate it do not check it into the secret management tool and pretend everything is fine because like I said you're probably not the first person to find it so how do you rotate a secret you literally just generate a new one you just say change my password that's it you just generate a new one so if it's an API key you just literally press the little refresh button it makes you a new one if it's a password you do the change my password and just make a new one it doesn't matter what the secret is you must rotate it which means make a new one and check the new one into the code repository okay so setting up a secret management tool um so unfortunately none of them are free all of them cost money I didn't find a single one that was free lots of these are cool I've used two or three of them not all of them some of them are really expensive and it makes me think that I'm in the wrong business when I see how much my clients pay for these but here are a whole bunch of them I don't recommend specific ones I just recommend that you get one of them so your entire org should use the same tool but you should have separate volts so each system so each like application should have its own Vault because if somehow someone gets into your secret management tool if it was all just in the same Vault they would get access to all of your applications and life would be terrible so each so same product or system or whatever for everyone separate vaults for each system you built that's really important yay Hearts okay so secret management uh the musical wait no sorry best practices I get those confused okay so each application should have its own Vault back up the Vault like your livelihood depends on it because it does use a service account for Access not your personal employee account so not t Jenga you want to have whatever the name of the app is connect it to your CI CD for each different environment and never save a copy somewhere else of that secret I've seen so many devs be like oh yeah but I also stored it in this folder I'm like ah please don't do that um if you find a secret somewhere it shouldn't be and it's an important secret kick off the incident response process because like I said before I'm seeing it for the third time because I really hope you do that conclusion what we learned very quickly What secrets are why we must protect them how to find secrets and then rotate the secrets how to manage them safely how to prevent secrets from getting into your code and what is up next in the four minutes I have remaining are some resources I always give resources at the end and the first one is would you like to have my slides this link will also be on the very end of the like last slide where I answer questions but just so you know PDF summary I have an online community called we hack purple and every single thing inside of it is free and we have community members present and teach stuff to each other all the time and earlier this month I had the OAS prong Secrets project team come and like just do like this two-hour Workshop where we all smashed and we found secrets and it was awesome and so if you want to practice finding Secrets not at work so you look super cool before you go to work they the the wonderful humans from OAS again have created an amazing free tool also there's a video on my YouTube channel we hack purple where you can see them even though you missed it like we still have a recording and it was great the books are not free all of these books are about devops except the last one which I wrote is about apps