BSidesSLC 2022 - Stop Committing Your Secrets - Git Hooks To The Rescue! - Dwayne McDaniel

okay I was told I can go ahead and start so go ahead and start hey I'm Dwayne thanks for coming to my talk stop committing your secrets get hooks to the rescue quick curiosity before we start who here actually uses git hooks on the regular already one I see one hand good you can leave everybody else this is talks for you I'm Dwayne I came down here from Chicago I've been a developer Advocate since 2016. find me out there on Twitter at mcdwayne it's down the bottom of the screen if you forget um happy to talk to you about anything Tech related or rock and roll or a lot of other things I work for a company called get guardian

we are a platform focused on secret leakage detection and Remediation happy to tell you more about that actually be mentioning them later in the talk missing one of our products so enough of that for right now we live in this constant game of cat and mouse that's the world of security we're constantly trying to outdo them and they're trying to outdo us and I wouldn't say they're bad people in general I think they're just there for the money and the resources the more I learn about security the more I think that's true but they are after our resources and our data constantly what this looks like in the wild if you were at the talk earlier today much more

in depth and I'm going to talk about it but Uber uh was hacked by lapses going in through a fishing attempt successful finishing attempt and then through Powershell scripts found a bunch of hard-coded Secrets that were then used to pull an Uber Toyota is one of my favorite breaches this is how I got on the front of Hacker News for about eight hours by writing about this um Toyota had hired a subcontractor that maintained their t-connect repo accidentally pushed part of it public maybe accidentally we don't know the point is it had a key to a data server sat there for five years before anybody noticed Samsung who here has a Samsung Galaxy nobody's hand went up

um everybody knows the source code for Samsung Galaxies now thanks again to lapses inside of the code base we found over 6 000 Secrets laying in there and back in August another breach happened and we think it's related but there's no direct evidence that that's true all we know is that Secrets got hard-coded and released this is one of my favorites AstraZeneca this happened last month developer hard coded a credential for a testing environment which should be all good because it's test environment right except somebody else pushed real data into it which isn't great so credentials for an undisclosed amount of patients was released or leaked we still don't know the full impact they're really

tight-lipped about this one and all these stories these are companies that take security very seriously but someone left the key out front you can have the greatest defense in the entire universe but if someone just literally hands out the keys they're going to get in and here's one of my favorite examples of good Security in action because they bought a really secure lock and I don't know what's on the other side of this door but I'm guessing it depends or staplers or something that's yeah we shouldn't we should put it behind a lock but at the same time people just got to get their jobs done let's slap this number up there because that's a complicated lock for for pens

and of course that's silly we would never do this in computer science we would never do this as developers except we do every single day I know I've done this I just got to make sure that database works I just got to make sure that API endpoint is up and how's the easiest way to do that well I'll just hard code it real quick but I'll take it right back out I promise except now I have a fire over here to fight and in slack I'm hitting three different places where I need to check out another Branch so oh I'm not online nine anymore I'm way down in the code like line 400 working on something

else I just got to commit push and move on to the next thing this is a serious problem so get guardian we make a state of secret sprawl report every year that's the URL at the bottom if you want to go read it yourself from last year we found over 6 million Secrets just hanging out in GitHub repos publicly we know it's way more than that across all the other hosts out there the number that I was talk about is the three out of every thousand commits that we see has some kind of credential in it and this is constantly increasing I can tell you next year that number is going to be higher and it's not a good thing

so who's ultimately responsible for this unfortunately it's everybody's job which means it's no one's job because everybody's in devops you sitting in this room you were part of the devops cycle somehow some way I promise you and the best of organizations the security team is outnumbered a hundred to one by developers this is a quote from Alex rice from hacker one at security at 2022 earlier this year and that number really stuck with me because this is the best of organizations if we do the math exploration for a team of about 400 Developers that's from what we've seen working with customers on average per uh security expert there's about three and a half thousand hard-coded Secrets

inside the orc which is a staggering number so it has to be hey this shift left and let's push this over onto the shoulders of the Developers this might seem like a complete non-sequitur but if you're going to speak to a developer you need to speak to them in their language and get love it or hate it confused by it or mastered it get is the default de facto language of how devops works and how developers think about code it doesn't matter if it's PHP JavaScript whatever python at the end of the day what they do with that code translates into get and get is awesome get almost all developers use this 93.6 of all code bases on Earth are in

git right now according to the last numbers I saw 96 of all developers use it daily so git is awesome and it's this universal language we all use but it's also the stupid content tracker that's what Linus called it that's still if you type man get into Bash that's still what it spits out um because it's stupid it doesn't know what it's doing it knows what you told it but in and of itself it doesn't make your code any more less secure so everybody using git yeah it could be a good thing git does have a way to help with security get ignore for instance um does anybody can anybody spot the the problem with this

there's an I'll give you a hint it's right here yeah they commented out the dot EMV file in this particular instance of get ignore so getting more is awesome because it will say all right get don't pay attention to this file or this path at all and you can safely store things in EnV files and um secrets.jsons and things like that and we should be but if we also are commenting out that line or not even adding the git ignore then it's going to include that in Source control and again we push that out to the environment out to the universe and people are going to find it and get into our stuff and there are awesome Solutions out

there everybody should be using hashicorp Vault I don't work for hashicorp probably won't ever but I love them I love Vault uh again I don't work for Azure but Azure key vault is one of my favorite architectures for Key Management just is personal personal preference um and their documentation is really cool and if everybody would just use a combination of get ignore and Key Management then I could stop talking and I would be able to stop giving this talk and I have much I'd sleep much better at night but unfortunately developers keep doing this and I say developer with me included because eventually we have to just test and see if something worked and the problem is that we tested and

see if it worked the problem is we forgot to take it out and then we committed it and then we pushed it not least a bad time because in theory you can get it back out if your next commit is you removing the secret and making the comment removed that thing I shouldn't have put in there guess what you've just told every attacker on Earth where to look for your secret it's in the previous commit let's not forget how git works git takes a compression of the entire file every time it's changed a new one and then it Stacks them up on top of each other and that's what our git history is so at any point in time you

can say I'm going to get check out this commit that happened you know 50 commits back 100 commits back the beginning of the project back and it resets the file system to that state so just removing the secret in and of itself doesn't fix the problem you have to remove it completely from history which is not hard it's just painful it's painful for a couple things it's painful because it wrecks everybody else's version of History and now you have to go have that conversation with the entire team on how you need to fix history and also you have to have the conversation that hey I did something really wrong and I've just exposed a lot

of things so what we need is some kind of robot because robots are awesome because they do the same thing over and over they're repetition machines and they can stop us from finding our secrets every time get gives us a way to build this robot I say bot very generically here give us a way to automate this process I should probably probably say githooks is this beautiful automation system built into git it's been there forever been there since the beginning because Linus needed this for certain things and it's basically this I have an idea of something I need to automate when I do a thing and get git will trigger the thing I built and then eventually wipe my mouth with a

napkin githooks.com if you don't go to any other website from a result of viewing this talk go to githooks.com and it's awesome Matthew Hudson has done an amazing job of putting together not only an explanation of how githooks works but also beautiful examples of what you can really do with it but these are the 17 hooks in git and three of them happened before a commit happens and one of them happens after you pushed but before it's received on the other end these are inflection points where you can tell get hey check to make sure I got everything right and if I didn't get everything right well don't let me make that commit and if you're like well my developers

are never going to do that all the time whoever's in charge of your corporate GitHub GitHub Enterprise your gitlab Enterprise your bitbucket Enterprise or however you're managing your version control system across the Enterprise um they can put in pre-received hooks to hey let's make sure we stop those Secrets before they get on to our server so GitHub comes with a bunch of example hooks so if you have a machine in front of you right now you can open up a git repo look in the dot get folder there'll be a hook folder it'll probably contain exactly these and if you go look at those scripts they contain things like rev parse and weird bits of pearl because let's never

forget git came from lioness torball it was specifically a set of scripts he wrote to help him manage the patches when building the Linux kernel he even said in the original email where he introduced it to the community I don't think it's going to become a full-blown SCM it's just something I use for my personal work three months later he hands it over to hunio and that's the story of git so if you go look at the examples like well why would I do that rev parse is one of my favorite commands and get side quick side note if you go look at the manual for it the actual description is retrieve and massage parameters

massage what other part of computer science uses the word massage in its documentation because I still don't know what exactly that means anyway these scripts while there's all those samples if you just take off the word sample if you remove the word sample at the end the extension sample then every time commit message triggers it will run what's ever in there uh this is one I run my because I like Dad jokes point is you can do anything you want hooks are just scripts so if the environment's available the script is valid it's going to go ahead and run it uh props to Ed Thompson uh who used to maintain npm I forget what he's doing

these days but he wrote get Dad which is a little script you can run for uh if you miss type git data instead of git ad it will still do the ad but I'll also tell you a dad joke so that's where I got the idea um but you can make it do anything including the thing that I originally was going to talk about or originally was talking about which is we need something mechanically every time we go to make a commit we'll look for username passwords API Keys security certs anything else you can think of that is a credential and if it's there throw an error don't make the commit and tell me what what

went wrong good news is you can build this yourself with some straightforward regex who loves building regex yeah that's what I thought um regex is fun don't get me wrong but rely on it for production use is terrifying but you can all you're really doing is saying hey uh get grew up git grep is one of the Cool Tools included in git because it's a subset of Linus's tool set um It Go dismisses the problem of what it's searching and just looks for the patterns inside of all the included indexed files so git grep is just a shortcut to use grub so we can say hey go look for things that look like passwords or 20 things

that are 20 characters long that contain all uppercase and all numbers that would be AWS credentials or tokens uh unless there are some errors if that happens and again you can build this by hand but then you got to maintain it and then you gotta account for edge cases and then you got an account for allowed patterns and then you got to account for false positives and then you got to sell it to your team and then and then and then so you probably don't want to build this yourself the good news is that this has already been built a number of times by a number of people um I'm gonna talk about three of them there

are more than three in the universe but I'm going to name three and full disclosure I work for one of them I didn't used to include them on this list but then they hired me uh the first one I want to talk about is AWS Labs get secrets so if you are all 100 bought in AWS everything you do is AWS I would probably start with this one honestly because the AWS Labs team built get secrets to look before a commit is made all three of those checks and see hey is there an AWS credential in here we know what they look like because we built them so let's go ahead and check and they'll

throw an error if we get there the awesome thing is it's free it's open source triple checks you can extend it because it's open source again you have to know regex to really do this effectively but it does it out of the box you can extend it out of the box not too hard but it is AWS specific so extending it means knowing exactly what you're looking for in those other patterns again not impossible people have already done this in open source I'm not going to name their repos because they're a bunch of one-off repos but you can find them and for like Google Cloud product for Azure people have already extended this so second one is truffle hog who here has

actually run truffle hog got three three hands two hands see two hands maybe I saw two um trevolog is an open source repo that will let you use pre-commit hook you can also do other searches known as it can do more than what I'm talking about for this use case but I'm going to narrow it for this use case for today which is looking for things before you commit them it will it'll absolutely let you do that it's free it checks the pre-commit level it does require pre-commit framework to be installed and that is a requirement for your developers to like have to deal with and live with so it's an additional thing to build on

top of git hooks but it'll help you manage GitHub so I actually highly recommend looking into pre-commit if you are interested in githubs in general because their documentation is awesome and they can give you a lot of really good advice uh you can run it as GitHub action but that catches your secret after it's already made it onto a platform this is also why I'm not going to talk about github's Native solution for this because it only checks after the secret is made it onto GitHub because that's where it cares about it um that last one your mileage may vary but go read your own reviews on high false positives neurotesting might prove differently uh then the one I work for get guardian

we make a product called GG Shield it's open source repo it does rely on the platform API so while it is all open source it is calling to a third party platform that's us um one thing I love about it is it also includes the pre-receive hook so you can install this on server side and kick out and stop any credentials from accidentally making there I'd overcome this this week as I started to contributing to docs because guess what I I wrote something that looked very close to a secret and had to call somebody to actually get the commit to be accepted um anyway installation is really straightforward and all it does is throw one line into a

git git hook for pre-commit or pre-push but it does require get Guardian which you can use for free if for individual teams individuals and teams of up to 250 for open source projects um because it's a platform based we're constantly adding new patterns to it right now we're checking for 350 known patterns um and we're always adding more and you can extend it yourself if you need to but it is API rate limited because it is a platform so mileage may vary on that depending on how many times you push might be the rate for you might not be I don't know your testing will tell you what what does this look like in action

because I've told you like what all these things do well I've already seen what this looks like in action because that's what I built earlier with the example of just building out a git grep I try to commit a secret here are my config Gamble and here's git Guardian it specifically spits out our GG Shield spits out hey here's the thing you tried to commit there's what we think it is an API key uh and fail We're not gonna let you do it not gonna show truffle hog not gonna throw AWS but they look basically identical they're the same use case hey we found a secret we stopped it and let's move on today it's a conclusion don't hard code your

secrets if you could just convince everyone on Earth to stop doing this the rest of this is irrelevant please don't hard code your secrets if you do hardcode Secrets do not commit those secrets and push them again if we could just stop there everybody'd be happy use automation to help you though there's a lot of ways you can automate I didn't really dig into it but now I'm seeing I got a couple more minutes here so I'll just rant on it for a second I'm talking about one specific security use case of git hooks you're only limited by your imagination you wish your developers would run these let's say five different testing tools every single time they made a

commit but you know they're just not going to manually do it build them a GitHub test it out your machine and then give it to them and say hey just throw these in place it takes you no extra time it's non-disruptive to your workflow assuming all the tests passed and if they don't pass then we've just saved a lot of time the farther left you can find the problem the cheaper it is to solve if you can solve something on a whiteboard whiteboard ink is really cheap if you find it in a production that could mean your business goes away that's the difference between those two edges so use automation as early as you can

and there's open source tools to help you you don't have to build this yourself in fact I recommend not building this yourself for the most use cases um but I'm a Dwayne I live in Chicago rock and roll improv karaoke hit me up about any of that stuff or more and then see Dwayne on Twitter and thanks very much for coming to talk

I got time for like one question foreign

all right well thank you