Git Those Passwords Out Of Your Repos!

right so uh thank you for coming to my talk I'm going to talk about uh getting those passwords out of your repo um detecting Secrets at scale so uh as like the previous talker Simon uh I've been unable to put my name on my introduction slide as well uh rather helpfully um we're both co-directors at Punk security who is a devsecops uh company and we're both devsecops enthusiasts I love terraform so anything that I can build is IAC and then integrate security into it I will absolutely do love doing security um I've got a long history of breaking stuff and not necessarily doing things in the right order and anything that's geeky I am totally addicted to like um

I've just bought a pair of new works landroids and I'm desperate to start messing around with them so they're busy cutting my grass at the moment um and they'll soon be being taken apart and as Simon said before as you might for those that might have missed it and we are a devsecops company we do have four open source tools one of them I'm going to be talking about today which is secret magpie we also have DNS Reaper SM beagle and pound spoof and if you're interested in those they're all free they're all open source they're all on our GitHub go and have a look at it we are the home of the devsecop CTF we ran that last month for

our second year of trading which was on May the 4th we didn't intentionally pick May the 4th to start our company but we did and we also ran it for crescon as well so if if anybody's interested in potentially running a devsecott CTF for their own personal companies uh hit us up with the source code for the CTF platform and everything is open source and if you want the challenges we'll quite happily let's share the challenges I think is that right Simon yeah he's not in his head he's saying yes so what am I going to be talking about I'm going to have a look at what the problem is and then what are the different types of secrets

that we can have leaked what can go wrong how we can defend it and then a little bit about our tool that and how we would wreck recommend you using it if you wanted to use it so the first thing is secrets are publicly accessible whether they should be or they shouldn't that is mainly what the problem is so if we've got an AWS key and we want to have uh somebody been able to be to use that AWS key to read a file out of an S3 bucket but somebody accidentally gives it too many permissions they can use that API key to then abuse the systems or somebody has created an EMV file or an

environment file or hard-coded a database password which then allows them to be able to go and access something so those are two things that we shouldn't be having with inside our git repositories and secret is not managed correctly so this is where we are um where we're not managing our secrets in a central location so when we might be writing secrets into a log file that we didn't know about or those Secrets can be used for privilege escalation like we were just discussing with the AWS S3 bucket or an individual might use those API keys to access sensitive data instead of using their own credentials that has actually happened we've actually seen that where developers who wanted to

access the production systems to be able to check stuff rather than logging in with their own individual accounts used an API key to be able to access the database to then extract data so we need to be able to try and control some of these secrets as well as keeping them out of our repos four main types of secrets that we're going to be looking for in our repos they're going to be hard-coded passwords to databases or to systems there's going to be API Keys typically AWS or some kind of cloud system we're also going to have tokens that we're going to be wanting to look for to make sure that we don't store those and private keys or

SSH keys in our repos we've also seen that before as well in in organizations and when you come across these SSH Keys you go well what what's that for is it still valid which system is it for and then you've got to go start trying to hunt down where that SSH key is valid so where can these secrets exist or they can exist in a couple of locations um one is in the files themselves and the other one is with inside the git Miss messages or the history and the problem is that if we've got these hard-coded in and we've got them in our public repos bad guys can come along and start reading them and can start

extracting the information and using it against you um we've had an example with a consultancy that we've worked with who's had this happen to them twice they've accidentally left AWS keys inside their public repo and nobody noticed but then over the bank holiday weekend an attacker then used those keys to go spin up an ec2 instance and start Crypt running crypto Miners And it cost them a fortune so we need to just make sure that we're we're checking for these things and making sure that they're not in our git repos and that we're not leaving them in places where they shouldn't be so this is a typical path of a a git Branch so in this branching there are many

different ways that you can do branching some people just develop straight into Main in this instance here we've got it the the developers should be using feature branches so this is where they will make a copy of the code so then they can edit the code at the same time as another developer and then they'll move they'll merge it all back together with a pull request into the main branch which is where the main applications code it so in this case uh our we're going to use a poor Young Junior developer who's a little bit inexperienced and he didn't bother creating a feature branch and he just created an EMV file with a username and password in and he's uploaded it

into the git main branch and as we can see there he's also put the commit of AD creds that set off alarm Bells with one of the senior developers who then quite rightly told him off and changed it back to be a placeholder for a password now the problem is that that password still exists with inside the git history and anybody can go back through the git logs and easily be able to access that file as it was in that point of time so our bad guy Mr hacker can now go back and go see what that password was even though it's now removed from the main current head of that branch is everywhere with me yeah excellent so

it's just like a written history now there are some people that would say oh go back and rewrite history I I'm going to cover that in my how to defend about against this in a minute well let's have a look and see what happens if they do this with inside a proper feature branch so again our young developer has gone and put admin creds in an EMV file he's put it up there I will see any developers gone you idiot get those out and they've removed them they've put a placeholder in there again and then they've merged it into the main branch or they could have even done a squash and put it back into the main branch the

problem is again even though it was a a branch that hit that hash for that that history still exists and the only time that the if you were doing a squash Branch um and it would be removed as if you run a git cycle and it was cleaning up the git history again an attacker can be able to go back into those feature branches and go and extract those Secrets um so even if it's something that happened six months ago if you haven't rotated those passwords and you've stored them in your somebody stored them and you get history the attackers can still go and extract that password so how are we going to defend against

this well there's this wonderful thing called devops and somebody quite rightly said let's squash secret let's set let's squash seek uh security into the middle of it so one of the things that we can use are secret scanning tools and there are two main locations where you would put these is either on what we call a pre-commit and this basically runs on the developers machine and it checks for Secrets before the developer can commit that code up into the git repo now there are some exclusions to pre-commits like if you were stashing something um developers can stash their code and that will like that will also get leaked in but um if they were doing a push or or

um doing their commits a pre-commit will check to make sure that they don't do they don't push anything up there into the git uh into the git repository either in your cloud or on your main git server the second place to check is also on you is a CI pipeline when you up when you get his uh git server so this is where every time you do a git push a CI pipeline or a continuous integration pipeline will run to check that that code that's been submitted uh doesn't contain any secrets and if it does it'll flag alarm Bells trigger an alert and then you can go and get it sorted out now me personally I would

probably encourage you to run both um the pre-commits um are really really good but don't just rely on them solely because developers can turn them off uh and if it's if they've accidentally stashed it or they've committed it up because they've turned the pre-commits off the CI pipeline will be your fail back it's site belts and braces approach make sure you scan in both places you can also use two different uh git scan secret scanning tools as well so maybe use git leaks on your developer side and then maybe use something like trufflehog up in your CI pipeline or even trivi uh Aqua security do trivia now and they do Secret scanning there are other ones as well that you can pay

for like git Guardian if you've got Deep Pockets and you don't want to use open source then you can pay get git Guardian for it foreign so what do we do when we find these findings well first thing I would always highly recommend is create a ticket yes we we all hate tickets but if you create a ticket it creates that bit of history about what happened at that time and then when security come along later on or in order to come along later on and go did you rotate that secret you can go yes this is the ticket this is who rotated it and this is what happened and that's the easiest thing to do in my

view is just rotate the secret change it don't worry about rewriting the git history and trying to strip it out if you've changed the password who cares what's in the history if anything it might put some send an attacker off in the wrong direction trying to Brute Force something that's no longer there and provide training to the developers and also to the operations teams explain the reasons why we're trying to do this Secrets management and why we need to make sure you make sure that we're not storing Secrets inside files um I mean I can think of one recent attack maybe Uber where um operations teams we're storing uh service accounts passwords in clear text

it's not quite the same as git but it's very similar and maybe ask our red teams and Pen testers to verify that there are no secrets in our git repositories I know it's expanding our pen testing again to dilute the time that the pen testers have got but it's a good check another thing we also need to think about is where we are managing our secrets so currently um or what used to happen in yesteryears especially when I was younger when I was a system administrator we'd had an Excel spreadsheet and we would store all of our service accounts passwords and all our service accounts all together in one big spreadsheet because we'd put it on a

a file share and it would be secure because only the domain admins would be able to access it I'm sure nobody does that anymore um so we've got things like AWS Vault and we've got other uh we've got Azure we've got Azure Vol and we've got AWS Secrets manager is it SCCM as yeah that's it SCCM so let's start using those things because then we can start logging who's accessing these secrets we can record who's accessed them from which service account what time was it accessed and if we're storing them somewhere centrally and pulling them down and into our applications it means we can also regularly rotate those Secrets as well so who cares if somebody

gets hold of it if we rotate it every time we're doing a release we can also Version Control our secrets as well so if we are rotating them every release and we need to roll back a release we can also roll that secret back at the same time and think about if encryption and where you're storing these secrets and making sure that you just have some decent um hygiene around them so how would you use secret Magpie well we recommend that you use it as a pre-secret scanning tool so almost think of it as like a scan before you enable secret scanning what secret Magpie will do is you will give it access to your repos and it will

then scan all of your repos and tell you where all of your password or where all your secrets are and in which file and give you a link to be able to get to it I'm going to show you that in a few minutes you can add it into a pipeline um we did have a very large digital payments company that we worked with across in Europe who asked us to add a feature and so then they could add it into every Pipeline and they could feed in which um which repos needed to be scanned every time they ran the pipeline and we've also integrated two secret scanners into this so we've added gitlix and we've also added truffle hog because

we found that both of those two found slightly different things get leaks was a little easier to configure so then we can put custom uh effectively regex's in there and we also wanted it to be able to be easily readable so we've recently just enhanced secret Magpie to also have a HTML interface and I'm going to be showing you that after the after the little kind of fake demo yeah because I don't believe in the demo gods and last time I tried to run a live demo it blew up in my face so don't worry I've pre-recorded this so what we're going to do here is I'm going to show you how it's running currently in a Docker container because

you can either run this as a python script or inside Docker and as with every good tool screw it's all it's got wonderful ASCII art we have um oh I wanted to pause that we have it's working as well for me as it did for Psy right we have documented all of the switches and we have documented all of the different formats so you can see inside here things like you can disable truffle hog if you didn't want that or you can get it to run against a single Branch so rather than scanning like all of the feature branches you might just want to specifically Target one particular Branch we can um we can also add extra

things hang on let's when it Scrolls back down so in the new version it does like a little web interface so if you don't want to have like the htmls file saved out yeah I did say that I wasn't very good at this didn't I

come on right when it Scrolls back down I will pause it this time come on scroll down right and if you don't want to save the file out was HTML it has got a built-in uh web um web server so it's just using Python's default simple HTTP service so then you can spin it up and have a look at it and you can point this directly into your repos so you can point it at GitHub or Azure devops or gitlab or bitbucket or if you wanted to store them all as a flat file system you can just point it to a path and it'll just scan the git clone that you've already created so what we're going to do now is we're

going to run this um against um the oh what the um the punk security demo uh organization which has forked um the O wasp wrong Secrets repo has anybody come across o wasps wrong secret repo before a couple of shaking heads a couple of nodding all right so basically it's a repo that's uh very very similar to like uh web go or juice box it's filled with secrets and then you can then test secret scanning and test your policies have been able to find secrets with inside repos so it's it's a nice easy thing to do what we're doing here is we're running um we're running it as a Docker service we're passing through a volume a local

folder called repo and then we're going to Output the results in a HTML file inside that repo folder hopefully so go I am playing yep that's it started so it'll display a nice bit of ASCII art and for the sake of your time I did pause it whilst it was busy running because it doesn't take very long but it it you know it takes a few seconds but I didn't want to waste everybody's time because I can quite obviously use a pause button um so inside here this is what you'll get when you just run it as a CLI it'll tell you how many repos it's got it'll also tell you how many Secrets it's detected how many unique Secrets uh

how many Secrets each tool has found what type of Secrets have been identified and what kind of files have been seen so if I now just stop that I'll Escape out of that and hopefully it'll let me show you this yes so this is the HTML interface that we've created I'm just going to refresh that because I want to clear the screen so basically it will load in all of your secrets uh as a as a table and then you'll see Bell C down here the repository the commit where it was committed or found the file it was found in and then some other information as well down the side here we've got a nice quick easy bar for you

to be able to Pivot through so you can detect what kind of secrets that you're interested in um you can also determine whether they've the secrets have been fixed or anything like that so if we have a quick look at that one which was some AWS Keys we can sorry it was a private key we can quickly click on that and oh brilliant it my uh my phone's not hot spotting anymore we can see there that we got a private key um but we may well have already rotated those private keys so we can go and select those and mark them as uh confirmed rotated we can also export the results as a CSV file if you really want

to but basically this HTML interface is there to help you pivot through the data really quickly be able to drill down into those Secrets at scale and be able to Market me the resolved or if you need to go and start rotating stuff so in conclusion um secret scanning is reasonably easy but you do need to just be aware that if you ask if you are scanning make sure that you s run you know you check that the repositories clean first before you enable the secret scanning otherwise you will end up blocking your um your developers from working uh has anybody got any questions

no all right thank you [Applause]