Stop Committing Your Secrets Git Hooks To The Rescue!1

good morning everyone i'm very happy to introduce drive mcneal he's a developer inventorist at get karateki helping people figure out how to best use get and get base the tools to get their work done outside of that tag right is that improving sir in chicago and also is a huge fan of the karaoke and the kent and the clock chart feel free to ask him about any of those and over there in a tinker where you can find him as a mac drank m c d w a y n e hope this helps and now i give the uh the the you know everything to speaker and i will record the the voice thank you

thank you very much for that introduction david um hey everybody at b-sides tampa i'm very very honored and pleased to be here with you all so as i was introduced i'm duane let me get my slides moving there we go um so i'm duane uh i live in chicago illinois so i'm doing this remotely i tried to come down that way but just did not work out and also excuse me if you see me looking up into the left i'm on two monitors here and just making sure that i'm monitoring the chat as well um so if you do have questions feel free to throw them in the chat here inside of the web app and i will try to

reply to them as quick as i can um yeah i was introduced i'm also a improviser here in chicago uh affiliated with the annoyance and a few other theaters uh as well as a huge fan of knit and crochet i see a lot of overlap between yarn craft and what we do in computer science uh repeatedly doing loops and end up with beautiful projects we can talk more about that later if you have any questions about anything feel free to hit me up on twitter that's the best way to get a hold of me or feel free to email me at the end of this when i'll share my slides uh i'll also share my email address with you

so i work for a company called get kraken we make legendary get tools a desktop client that makes it really easy to drag and drop things around and understand your histories and everything very simply get lens which is the number one vs code extension for get and get integration for jira which is a tool that makes your life a little bit easier if you're using jira to manage your projects all right enough about that that's not why you came in here but did want to give a shout out to the people that sponsored me to give talks um so why am i talking about get at a security conference this typically is a topic a lot of people associate with well that

before a specific code realm uh or a specific uh programming language maybe a devops summit and true i've given this talk at a number of places but i think it fits in with security folks extremely well because everybody uses git well not everybody everybody but almost everyone you'll ever meet that pushes code uses git and where they push that code to to go live somewhere on a server uh is somewhere like azure or aws or google cloud product or their own server infrastructure wherever it is it has a bunch of secrets associated with it as well as the repository itself github gitlab bitbucket wherever the code lives that eventually gets pushed to that devops service

it requires a lot of secrets api keys credentials passwords uh tokens those long really well thought out hashes that we use to plug in to verify we are who we say we are why i'm giving this talk is something i'm trying to raise awareness of out in the world is that people put things in their repos that they're not really supposed to they put these secrets in places where people can easily get a hold of them uh get guardian did a report back in 2020 i know it's 2022 right now but they didn't put one out last year for some reason um but they found two million secrets on github alone didn't even look at the

other services out there and what they found was it wasn't really malicious that people were hey i'm exposing this on purpose it was hey i copied a secret to the wrong place and pushed it to a repo i wouldn't supposed to we're playing this constant game of cat and mouse with the world where we're trying to protect our data and people keep finding new ways to get at it you can have the best security setup in the world you can spend millions on rock solid infrastructure really pen test everything make sure everything is up to snuff but if someone just leaves the key out front if you walk up to a building and see hey there's a key under a rock here

it's pretty easy to get in and again i don't think anyone's doing this specifically maliciously well some people probably are there are bad actors out in the world but i think it results from something like this that is a really ridiculously complex door code and i'm guessing whatever's behind that the people in that building the managers of this facility thought well it's just pens it's just staplers back here like people ah we're we're impending our flow we are stopping people from getting their job done by implementing too good of security and i know everyone cringes when you say too good of security yeah this is a great door lock it would take you forever to figure out

how to get in here but again someone pasted the key out in front for everybody to see and again for this particular use case maybe it wasn't that bad maybe there's nothing back there that is going to cause catastrophic failure for their company or cause a data breach but okay what does this have to do with code and get well you might not do this on your code repo and say here is the secret but this happens literally all the time uh this is a little go script i'm borrowing from someone and when you look at the nice speaker notes if you get your hands on these slides uh it links back to the medium article i grabbed this but i

thought this was a good example this is a bad idea i think everyone that programs anything would agree this is a terrible idea you should not be hard-coding secrets however we all do it why do we do it because we need to make sure that database exists we need to make sure the credentials are the right credentials we need just a quick test that hey this does what it says it does in a perfect world we would put this in here test it yes it's up and immediately delete it and we would be safe however we know that these things keep getting leaked and again i don't think the problem is people say hey i'm going to

put this in here and i'm going to leave it in there i think what happens is people start checking like okay this api endpoint is up but wait a minute i got this other problem down here in the code and they start scrolling down and now they're no longer on line 10 they're working online 180. meanwhile the rest of the world keeps going on they get a message from their partner on their phone someone on slack said hey we got an emergency in production can you like please jump over to this channel and work with us and by the time we get back we're human beings we only have so much capacity for focus we'll forget that we did this i think

that happens a lot to the tune of millions of secrets get leaked a year i have a friend who works for a company that just rotated 5 000 keys because they realized that in one error message it spit back an ssh key because the way they configured it it didn't just say the name of the ssh key spit back the whole key they didn't know how long that had been going on and rotate 5000 keys that was one company those logs get into databases those logs get into repos they get exposed real quick take a step back and say well okay what does git have to do with this still let's let's take a quick reminder

of what gets rolling is is in all of this and how it's helping or hurting our security situation so git is a version control system then by linus torvald with the linus linux project 2005. uh junio maintains it these days gitster but basically you take a snapshot of your work over time this is awesome because you have access to all of those versions forever you can stack those snapshots you take on top of each other one on top of another and you get these awesome graphs and you can do very complex workflows with it if that get gives you a lot of awesome superpowers like time travel where you can go back to any specific version you

want you can also branch off into multiple universes dealing with one universe and moving forward in time is really complicated we're really good at it because we do it all the time unwillingly that's just how reality works once you introduce concepts of parallel universe or branching and time travel with revert and checkouts and switch wow that's a complicated universe to live in and now there's a lot to keep track of and then you tie it into the world of devops git sits in the middle of all of it without a transport layer to easily push around deltas none of this works get ops is a thing for a reason devops only works on the back of version control

we can argue it could exist with other versions of version control than git uh i'm not saying it couldn't i'm saying get one get is the de facto and all of this everything you're seeing on the screen is built on the back of git well built to use git anyway so git is awesome but it itself is not a security tool in fact the people that invented it the people that still write the manual for it still to this day call it the stupid content tracker if you type man git into a terminal that's exactly what it tells you it is so it's not a security tool into itself but it does have a way to help with your security uh if

you know that there are things that you shouldn't ever commit like your gem files or other things like secrets.json are a aws file you can make a git ignore file and say hey git ignore these things and every repo should have at least one git ignore file that lists at least these kind of secret checks um in a perfect world we would never ever actually put our keys into our code but again we don't live in a perfect world if we all kept our keys in you know secrets.json and then just get ignored them that'd be the end of the talk i would never have to bring up the rest of this talk again that would be a perfect

situation and again in a perfect world people wouldn't be hardcoded in their secrets in the first place so if we really study this and i've looked at this a lot and thought about this a lot the secret really isn't or the problem really isn't that people do those quick tests you're never going to be able to stop that that's how development works you just need to know that api endpoint is up the secret the the problem is that you've committed that secret that it didn't just stay temporarily on your local machine for like a few seconds that you committed it and then you pushed it somewhere now you have problems if you commit your secrets

and then push it out in the world you're gonna have a bad time there's just no other way to really say that once it's pushed there are tools that can immediately detect what's going on uh so like sonarland deep source advanced security we've all got the security alerts of like hey something's gone wrong somewhere but it's like that old song from the offspring by the time you hear the sirens it's already too late uh something bad has already probably happened again because we're constantly battling that the dark forces of bad actors out there uh they're looking for any way in and with automation they're constantly scanning the same way that white hats are scanning for

hey there's been a breach they're scanning for any way they can get in you can remove your secrets from your history from your branches once they're pushed not saying you can't uh but it's painful it's very painful it's not just painful in the sense that okay now i got to do surgery on my get repo that's fairly trivial that's a bunch of command line runs or just thinking about how your data is stored but now you gotta go rotate those keys how many keys would need rotated if one secret got leaked in your org the thought of that should be overwhelmingly painful there's a lot you need to do when you're rotating keys anyway um so what we need

is some kind of automatron some kind of robot that just stops us from doing this now if we do this over the wire if we do this as a service then we start introducing problems of hey we have man in the middle attack the vulnerabilities we have encryption problems over the wire there's a lot of things that can go wrong once we start relying on third-party services again once we've pushed our code and it's been detected it's already too late so we need something local well we already have something local it's already baked into the entire process it's the underlying lunch pen of devops it's git inside of your git folder if you've never looked in your git folders

i encourage it like you probably have one on your desktop right now go open up a repo and dig into your git folder a lot of cool stuff in there it's not as nearly as scary as you think it is but one of those folders is the hooks folder hooks lets you build your own anything the very simple way hooks work are something triggers and get get causes a script to run that's basically it i like this um picture rube goldberg this is one off wikipedia but a very complicated device that does something very simple there are 17 hooks available to you um though the ones we're caring about the ones for how we're going to stop ourselves from

committing something is we need to well look before we commit something there are three hooks that fire off before we actually make that commit precommit prepare commit message and commit message there's all sorts of things you can do with this and one of my goals here is to encourage everyone to start thinking about well how else could i incorporate other security tools how else could i also incorporate other security checks through this process because every bit of code again almost every bit of code in the world that goes through a devops cycle goes through all of these at some point um so if you want to learn more about this i highly recommend githubs.com matt

houston has done a tremendous job of assembling great examples and deep explanation of what's going on with get hooks um but basically you can put anything in a script so here is an example script uh so if you open up a git hooks from a new um git hooks folder from a newly initiated a project you're going to find some samples in there so there's apply pass message sample pre-commit sample this is what pre-commit sample actually looks like if you erase the dot sample then it will run then git will pick it up when it goes to commit it'll run pre-commit all right now i'm gonna go run this script so this code is actually written by linus i believe or

someone early on in the project and unless you are fairly gifted at scripting um or no get plumbing pretty good maybe this doesn't make a lot of sense to you however you can make do anything you want like i say this is your script so this is an actual script i run on commit message in my actual workflows i like having dad jokes spit back out at me in the terminal maybe it's a little dumb but shout out to ed thompson uh who maintains npm these days he was the maintainer or the chief lead on github actions and he also maintains a project called libgit2 which is how a lot of the world uses git uh but

he also wrote this thing called i can has dad joke um i'm sorry get dad as the project he wrote based on i cannot get that joke basically you curl i can ask dadjoke.com and it spits back a dad joke is it dumb yes does it make me laugh every single time well about half the time that's true anyway this is just an example you can do anything anything you can script you can absolutely make run so getting back to our problem at hand how does get hooks play into that well let's think about the solution if we were to build a robot we would tell it in a modified gherkin language here um before i commit before the commit is

actually finished i want you to check my code to make sure i didn't hard code any secrets and if i did stop what i'm doing throw me an error and do not make the commit pretty straightforward it actually is very straightforward um you can build this yourself it doesn't take that much scripting know-how but it does take a little bit of regular expression so i'm using git grep um git grep is well a tool built into git it works just like regular grep except instead of the scope being all the files it can see and then you have to specifically tell it which files gitgrepple just looked at the indexed files all the files that git

already has access to so it narrows that scope it makes it a little bit easier to search just your repo um so regular expression i'm looking for anything that looks like a password uh maybe it looks like that or maybe it's a key that's all capital letters and numbers and it's 20 characters long that's what regular expression means there and just spit me out an error if i try to commit something that looks like this and since the password was the first thing it saw it saw like all right i see their great no hard code password that's what we told us to sell me yep and it worked as expected truth is you can build anything

to do something like this uh or you can build anything to um look for any kind of pattern you want but then the ultimate problem with that becomes um you got to maintain it like you built a solution great uh one of my favorite quotes in the drupal world ever uh was a guy named jim burch he wrote a module for the drupal community and he said the best news is there's a hundred thousand people using it the bad news is i'm supporting a hundred thousand people using it and none of them are paying me um you don't get unless that's specifically your job to go out and build security tools and infrastructure maybe you don't want to get involved in

this directly um you also got to go sell it to your team you also got to get buy-in that this is okay to run you're gonna have to get this vetted keep track of what's going on what about allowing code or allowing example passwords like this is an example password no one's ever going to use that as an actual password so it should be okay to leave that in there um it goes on and on there's just a lot to think about well good news good hooks to the rescue our open source to the rescue using git hooks aws labs they have a pretty big stake in the devops community they're a pretty big player out there they um

they built something called get secrets that specifically meant to help people not commit their passwords and api keys when using aws basically this isn't a full-blown tech demo of it but it allows you to install the tool on your machine so you have a globally it's there uh brew or powerscript or powershell script or what have you um then per repo you declare that hey we're installing git secrets here so we'll run on this repo and then once getsecrets is up and running you tell it what to register and we'll talk about that register here in a second um then you can start going through and say all right we know this pattern for our

company this is how we use example passwords or this is what our example api keys look like you can start adding those in there as you see fit once it's installed this will run every single time um that's a great question sean i see there and i'll get to it in a bit so what does it actually do well what it's looking for are these things out of the box is what the aws team said hey this is what um our keys look like and these are the patterns that we want to disallow so if someone said a key assignment equal or colon um but also let's start allowing this pattern this is the de facto amazon aws

example keys these were copy pasted from their docs also aws allows you to have a aws folder so that's where you keep it's the equivalent of a secrets.json but aws slash credentials folder um should stay way out of your repo you should put it somewhere like your home folder where you're never going to commit your home folder and just keep all of that away from your code as much as you can but it's looking it will look in there and say all right i'll look through your credentials folder and all right uh if any of this is in your code we will just stop you from doing this what does this actually look like from a

code perspective is this onerous this is a lot of code to manage nah actually it's not um inside of every uh those three um inside of the three hooks i mentioned commit message pre-commit and prepare commit message it adds a single line it says hey run get secrets run that particular flag and then do it for the input uh whatever your computer is looking at whatever bash or zish is looking at and then it adds into your git config so again if you haven't looked in your git folder look in there there's a config file that overrides uh your global so you have your global git config that announces who you are but if you are working as

multiple personalities or multiple personas it's pretty easy to go into individual folders and make a config file for that folder so in that file for that repo it adds all of these patterns and it's already written the regular expression for you in my experience the hardest thing in the world is regular expression it just is the fact that this team already did it is pretty awesome so what does this actually look like when you run this um here's an example i was working on i did this in readme and i did this with totally fake keys so nobody worry i'm not exposing any secret here that shouldn't be exposed these are made up based on

the example repos or example code they have but anyway um i tried to commit it and here's what it spit out now you notice it actually detected two different things on two different lines and they gave me a bunch of mitigations again something i wouldn't have thought to build in if i was building this by hand the fact that i'm going to help myself later with any of this code is cool but it's not top of mind so i'm very again i'm very glad that open source has given this to me and to your point sean um well that's all well and good for ws users but what about me i use something else the last place i gave this talk actually

was at azure spring clean which happened a few weeks ago which is an event all about cleaning up your repos and cleaning up your devops practices well once again open source to the rescue uh because open source uh aws open source their code everyone could go in and see exactly how it works a fun fact go back a few slides this code here is modified from what i found digging through the aws repo uh it's how i learned a little bit more about how regular expression worked and a little bit more about how uh get grep worked and i learned a little bit more about how all of this works together because i could dig

through the code because that is the power of open source i'm a big believer in open source for the code and close config so i think we should all benefit from that anyhow uh for google cloud product uh there are a lot of variations there's a lot of forks these are both forks of the aws get secrets repo um but these are just the first two i found that i said all right these look pretty good when i looked at the code looked at the issue queues and whatnot um i'm not going to try to pronounce um these but you can see them on the screen if anyone else tries to pronounce them out loud feel free um

but they both add the um add aws uh or add gcp so this one has the register gcp which goes through and has the same set of patterns go explore them on your own to see exactly what they add and how they affect the code but aws actually builds in the ad provider you can go in and completely register this yourself uh you can also add anything you want you can add global expressions literals allowed you can modify this to your heart's content and you already have access to this so get secrets as open source go ahead and download it but the underlying framework of how all of this works you already have access to it's already

in your code it's not something you need to go ask permission hey can i use git in general yes if your company's not using git go ask that permission but it's something you already can't access something you can already get your hands on right now and start leveraging right now and this is just the beginning of it um this is one security pattern i'm sure in a security conference like this there's a lot of ideas brewing of like well could i also automate this could i also run this other tool the short answer is yes if you can imagine it if you can physically type into a terminal to execute a thing you can automate it that's one of the

beauties of open source and the awesome things about the command line is the command line is extremely scalable extremely automatable and it's only really limited by your imagination time and ability to research the good news is that almost every problem out there has already been solved by someone else and they've probably released the source code already again go open source so i'm at the end of my talk i am happy to answer more questions but what i want to leave you with is don't hard code secrets and i know i'm not going to stop anyone just by saying that because we all do this we all check occasionally just to make sure that database credential is

right i know i have done that more times than i should um but do not commit those secrets if you do put them in your code get them right back out don't get distracted if you put them in put some giant honking notes around them like hey remember to remove me but also use automation to make sure that it's the big honking thing that yells at you hey you tried to commit something you shouldn't of go ahead and stop it already and make your workflows your own whether it be a dumb example like throwing a dad joke in with i can has dad joke through get dad or doing a full-blown security suite test every single time you try to commit code

it's up to you you do what makes works best for you so i'm duane i work for a company called get crackin you can email me directly at duane.mcdaniel getcracking.com and always feel free to hit me up on twitter i will answer questions about improv knitting crochet uh chicago in general but also tech and git and many many other things so thank you very much again besides tampa for letting me be here and i did reserve about 10 minutes when i ran through this for a question and answer because i said i had 45 minutes but i'm a little bit ahead of schedule so um does anybody have any questions out there for those of you that came in late

because i saw the numbers went up um yes uh i can ask dad joke i wanted to show this get add

thompson's get dad i highly recommend everybody go install this um basically when you type get dad instead of git add and we again all do that we all fat finger things sometimes um it'll just spit back a dad joke but if you look in the core again reason i love open source this is the entire thing this is the whole project uh it's it's a script um it just curls i could have dad joke i've applied that to a number of other projects i've worked on

actually while i'm here i'll go ahead and share these slides with you i think put those into the chat here those are my slides all right anybody else have any questions the other time since i'm here aws secrets

go to google to ask for something on github oh yeah okay got questions coming in um what's the most common non-obvious secret you see committed uh passwords um for what we think of as non-vital systems um i believe is what that report said uh so if we go back to my intro here yeah um get guardians state of secret sprawl 2021. so they did the research in 2020 and then released in 2021 but this is the report um you can see yourself that literally outlines this entire problem set i had nothing to do with this this is just research i am leveraging out there um but yeah it's us passwords are very easily uh putting in put in for just checking to

see make something works and then uh forgotten about oh example something that's not a password or a key um that's a good question um most of my research has always been around you know those major pieces i'm sure if we go through back this report there are some would be some things that are not obvious if anybody else has some ideas out there um feel free to pop in uh okay sean's asking what about writing your code do you use aws password manager as your key vault they have your code check out your keys have you seen people use this as a mitigation absolutely that is the perfect solution um when i was talking to data um azure

aws uh our azure spring clean that came up that specific thing came up and key vault uh for azure is a beautiful system and there are all sorts of secret managers you should be using and then just referencing so we're calling uh so for those of you who might not be familiar um secret manager uh like key vault um let you store your secrets and then manage those secrets you can leave group them together uh give access and permissions to your teams uh and then call them programmatically so instead of calling hardcode your password you're hard coding a call to your secrets management at a certain level though you're just abstracting the problem out one

one abstraction because how do you manage the secrets and tokens and access for key management yes it's a much more secure system yes it's way harder to bust into it's kind of like hosting a site on um github versus a shoe box in your closet uh the shoebox in your closet just not gonna have near the security of github and but github's just a system with passwords like theoretically if someone has the key they can get in and and mess with it um so yes in a perfect world again we would use key management we'd call over the wire uh and purely encrypted maybe tore it up uh make sure that nothing's gonna ever figure out where

those keys are stored and whatnot um and they do help i'm not saying they're bad ideas but reality is people are still going to always hard code secrets occasionally just to check things and they shouldn't but they do and so this team here i don't think anyone at this conference i really don't uh watching the other presentations and understanding like what this conference is about i don't think anyone here is doing this i don't think you're the problem i think the problem lies and people not knowing hey maybe there are tools to help me with this and that's part of my reason to be here is i want you to all help spread the word

if you know a developer send them this repo and say hey did you know this exists did you know you should be managing your secrets a little bit better do you know robots can help you do you know you already have the tool that underlies how all this works um send them my slide send them a recording of this talk there's a page on get kraken's website about this just look up get secrets get cracking and you'll be able to get there um uh but yeah that's that's the whole goal here is i i'm not preaching to you to like change your ways i think this group is really embracing security overall um but you know people that aren't go talk

to those people and and help them see the light all right any other questions out there or anything else you want way to speak on not we can go look at the code real quick um where is this yeah so scans that it would have taken me a while to figure out how to write myself again they're already written um anyway this is another talk to getting diving into code just wanted to show off the repo where i based all this off of because when i found this i got really excited it's like wow everybody should know about this thing out in the world

right any other questions i think david then we can wrap this up um give everybody a few extra minutes to relax get your coffee get your lunch and uh get ready for those next sessions here at besides tampa thank you thank you for your greater presentation i really appreciate it and uh getter and the girl harbour and the gear lava is always you know very sensitive and uh you know the bad guys on the internet that they're always looking for the secrets from the source code and so in that way they can you know uh break into the you know the applications i really appreciate your in the presentation now i will stop the recording and

end this session so thank you bye