Do You Know Where Your Secrets Are? Exploring the Problem of Secret Sprawl

good afternoon welcome back welcome to password con room let's get excited for our next speaker Dwayne McDaniel who's going to be talking about do you know where your secrets are exploring the problem of secret sprawl and secret management maturity let's go sponsors we want to thank you especially our Diamond sponsor Adobe shout out to our gold sponsor blue cat Plex track and Toyota it's their support our other donors and you all which make this possible please turn off your cell phones nobody wants to hear your bone THS and Harmony ringtone okay if you have any questions save it to the end with that further Ado Dwayne what's up hey thanks very much for that intro um

if you don't want to turn off your cell phones uh just turn off the ringtone but yeah please take pictures if you want uh and post them on the Internet please um thanks for all coming this afternoon uh hope everybody is everybody having a good bside so far awesome so so apparently um so I'll before I give my intro I have a prize hat up here I'm asking specific questions who saw the last session in here all right you guys are going to get a chance to win some prizes from me based on what you heard before uh because I work with Mac um so I live in Chicago I've been a developer Advocate since 2016 they abouts uh depending on

how you count veler advocacy in the early days but mostly come from the devop space uh the platform play Space security this is my first besides Las Vegas so thank you all for having me I'm very nervous to be here uh is the biggest impostor syndrome I've had this year this is like my fourth bsides so you guys are all special um if you want to hit me up on X or Twitter or whatever they're calling it these days it's MC Dwayne and it will be until they take the service down uh I'm also on Insta and all the other places MC Dwayne uh including GitHub I should probably put that on here that's a more reliable

platform than all of them and hopefully it's more reliable and hit me up about anything if I'm talking about it today or if you want to talk about karaoke which who's going to karaoke tonight okay it'll be me singing by myself um but if you want to come hear me sing by myself or you want to talk about rock and roll uh you want to see a rock and roll show please hit me up I'm always happy to talk about rock and roll um very quickly I work for G Guardian they're going to come up a lot in this talk because a lot of the research was done by them in fact all of the research

because that's where I work that's just how it works um so we have a booth over there we can talk to you about hardcoded secrets and cheting how they leak and all that um but that'll leave that for the hallway over there so in summary just don't leak your credentials and you're good and you can go home and sleep well at night you've done it success unfortunately I have to keep giving this talk and other talks like it uh so two-thirds of this talk is just going to be a bunch of facts and figures that might be terrifying especially if you weren't aware of them before and some people will just KN along and say hey I've

heard this before I've heard things like this before specifically the guy that talked right before me McKenzie my colleague who's also a developer Advocate at G guardian we didn't plan that that's how the schedule worked out um but if you do feel panicked or fear like you're talking about my company uh just recite the litany against fear and everything will be okay after all we're talking about the internet nobody really dies on the internet do they uh that's maybe not a great joke but uh Uber they're still out there they still exist but this happened to them uh anybody remember this from 2002 or 22 um so super admin gets fished and they're doing it right they got two MFA on or

twor authentication on and the theory is he got flooded with so many requests that is thumb slipped we'll never know maybe he just got mad at Uber and said okay I'm just going to let this one through anyway the request went through uh the spammer or the attacker uh who was from the lapsis group uh which is a UK based hacking group um got in was able to take that Superman's credentials go through the VPN long story short found powery shell scripts chock full of hard-coded credentials for everything and he got into everything including their hacker One account and told hacker one hey I hacked into Uber I'm cool and they're like yeah this is

that's probably not true so he floods their uh their slack with memes and takes over all the channels and they're like yeah this is some prankster the next person you talk to is the New York Times that's why we know about the story what did he actually steal we'll never know did we see a wall of new security job postings from Uber the next day after that yes we we we did um Circle C who got affected by this who had stuff in production for this is anybody here a developer do anybody use Circle CI wow that's the that's a first that is literally a first for me giving this talk that or this bringing this up that

no one nobody got a hit um basically the kerning messed up and I'm really mad about that because that was fine 10 minutes ago anyhow um long story short attacker hits a remote developer remote developer's environment gets compromised attacker gets into the circle c internal Frameworks uh and internal platforms plants malware that starts stealing credentials that then takes those credentials and gets into customer accounts the same day that they come out and say hey we had to rotate all of the access Keys sorry if it took down your production instance but we had to rotate them uh same day a uh independent researcher said hey something's going on Circle C cuz all my honey tokens went

off and now we know why someone actually got a hold of their credentials and was attempting to exploit them I don't have fancy graphics for everything else uh one of my favorite stories uh not in a good way but just I think it's interesting uh anybody drive a Toyota do you have the tconnect app yeah that's the right answer at Toyota back uh in 2017 subcontracted someone to do maintenance on it good idea in general for some reason that subcontractor pushed a portion of the tconnect code onto a public GitHub repo why we'll never know it takes a security researcher until 2022 to figure out that that contains an actual data key that affected 296,000 customers uh

nothing super sensitive so email accounts and uh um some identifying information but no pet like credit cards uh so Toyota Japan uh puts out a really nice blog post and statement saying hey you're probably going to get fished be on the lookout it might not be us just be aware but for five years it took him five years to figure that out uh Astro zenica all we know are the very scant details that got released but it was The Perfect Storm Tempest in a teacup kind of situation where a developer pushes the test environment credentials into GitHub public no big deal it's test credential like what what on Earth can happen in a test environment well

somebody else inside Astro zenica pushed actual customer data into the test environment we don't know how big this was we just know what happened because it's Hippa they can't just come out and say this many customers um and they didn't unlike Airline crashes uh that come out immediately and say hey we had a crash we are going to fix it and here's exactly what I went wrong and here's exactly the steps in security we all just kind of get H we'll make best effort sorry and move on this one's kind of scary to me because I am an asentic customer we live in this world of of constant cat and mouse uh literally people trying to get in and get our data

and our Hardware resources or machine resources Alfred uh the creator of cybercop the first commercial Honeypot that ever got released um said this and I think it's a great a great very nice Summary of why we should take this stuff seriously I think it can be reduced to this attackers and this this is supported by like Verizon DB report if you haven't read the DB this year go read it it's a great chock full of things ransomware is kind of ebing and flowing and uh it turns out that log for J we got mitigated and it's kind of not it's still a problem but anyway dbr is full of of good info but 80 plus% of all

attacks are organized crime at this point of malware attacks um yeah there are nation state actors like we all know about Microsoft and their forged key that happened last month um that was the China backed hacking group attacked them and guy was trying to get in the state department Commerce Department that stuff happens there are activists that just want to make companies look bad like the kid from lapsis he was 19 so I say kid um young man from lapsis uh who did it for the lws Apparently and just to make Uber look bad publicly but most people are after the top two anybody know want to guess this isn't for the prize Hat by the way uh but anybody want

to guess what number one why they want your machine resources why do they want your data money but how do they get your money ransomware my favorite joke I heard all of last year was how did the hacker evade the FBI they ran somewhere yeah thank you thank you that's the exact right response thank you all um the one thing all of those attacks have in common that I went through and again I'm not trying to scare youall this is a reality is that there was a credential somewhere along the path that gave the attacker access to other things or a credential that was misplaced because it was hardcoded when it shouldn't have been hardcoded attacker is going back oh for

here for a second what we see them commonly do to get these things is lateral slides lateral movement through the environment trying any door they can get their hands on and then escalate privileges up let's take over the machines escalate privileges as high as we can and then go see what else we can get into and escalate those privileges lateral slide escalate up time and time and time and time again but because of that we kind of know how to defend for that or we should and that's what we get to in the later part of this but just a quick definition we I use the word secret and credential interchangeably because they mean the same thing to me

in my head but this is what I mean by it it is an API key a user name anything that gives you access to another system or decrypts data that's it like if if it's an EMV file CRT or what have you um an actual SSH key something that should be secret you should not be sharing that out in the world this is how it commonly ends out in the world and I do this all the time anybody want to guess why we do this it's easy it's easy and when do you need to do something easy because you're in the middle of a a rush te testing and debugging like I just need to make sure this credential

works is this the right password and if you do it locally and you take it right back out of your local file who cares that that's how we debug things that's what we do in life the problem is then we add commit and push and that's where the problem really escalates to the point of we put out this report every year feel free to download it if you want that's the first big third of this this talk is just going through these um so we as a platform since 2017 have looked at every single public G commit you can as well if you want api. github.com timeline uh or maybe events I always get the too confused but there's a public

API you can just go subscribe to it's in Json it's over six billion 60 billion commits deep it's it's insane um last year alone there were over a billion commits that we scanned again just public GitHub so not public replit or gitlab public or anything like that um and we're seeing a giant growth in GitHub year-over-year uh 27% increase in developers just in 2022 um at the time the fastest growing language we were seen was HCL uh anybody know what that goes to terraform yeah that's great you guys are awake I'm loving that this is just far enough after lunch that you're still awake and the coffee's kicked in from the 3:00 Coffee Rush all right first question

from the prize hat and probably heard this stat if you were in the last talk who uh who knows how many secrets were discovered last year in public GitHub by G Guardian who said that uh you get piece of candy there you go all right yeah 10 million uh that's a terrifying number and that's a fun way to deliver it uh yeah we found 10 million hard-coded credentials this was a 67% increase over the previous year and I have a slide that lays that out but this this isn't cumulative this isn't overall of time this is just new ones out of that how many billion 1.27 billion commits how we did it that's public as

well we built our own detector engines but we documented it all out there and that's you can go look at the actual how we did it so they're just saying it's all supported out there all right second question on average how many commits per thousand commits containing a secret any hands now because I can't just hear all the way in the back what 100 little high anybody else 10 a little bit little bit High still anybody else 569 56.9 that's a great specific but no anybody anybody else I will give it price is eight rules over there I'm not going to throw it but 5.5 you get an Apple uh you can you can come

up and get it later or if you want to pass this back here here I'll throw it to you and you you can pass that back to him you get an Apple uh 5.5 out of every thousand commits on GitHub public last year contained a credential a hardcoded credential that means one out of every 10 authors pushed one you can see the previous years like there was a little over six million we discovered in 2021 uh over almost 3 million we discovered in 2020 and again this isn't a cumulative number this is brand new ads and when we stop and think about like the fact that GitHub itself only grew 27% this is startling so who's doing this

stuff turns out it's everybody it's not can't just be new people yep that's the answer um so this is where they're coming from and I don't want to point fingers but it kind of makes sense when you think about General populations but does anyone see a weird missing Nation off of this list anybody any security researchers in here that investigate um where origins of attacks come from yeah Russ is all the way at the bottom it's on the list but there's one missing oh yeah they should definitely be on there as well but I think that's tied to the other answer uh oh so North Korea isn't doing public GitHub that's the other reason they're not on here um

Indonesia so go look at where attacks are coming from and you're going to surprised by your logs you start tracing IP addresses at least in my research I found like a shocking number of like Indonesians like why are they coming from Indonesia and I've talked to other people about this and there's no clear answer maybe they trampolining maybe North Korea is using Indonesia is an attack Vector all we know is that Indonesia isn't leaking their secrets onto public GitHub that's what this data tells us anyway moving on um so if we look at specific categories of oh wait that was supposed to be one of my question slides wasn't it oh no oh I moved it around sorry that was

supposed to be a question slide um so if we look at the how which specific Secrets got shared out there what types um others the biggest category because other is literally the bucket for all the other like 2% 5% 7% that weren't big enough to show up at 3.8 so let's let's ignore the other for a second and I should have just taken that out but data storage is number one cloud provider is number two which totally tracks to what we originally talked about attackers are only after two things and they're disgusting uh because they want your money messenging systems why do you think messenging systems uh are are important what off

otherss yeah um I think what I heard between those and the answer I'm looking for was was fled out um is there's a lot of companies that think okay I'm just going to pass this key between developers through the system it's better than email right slack yeah it's it's slightly better an email um anyway we look at generic detectors as well so we have specific detectors look at like specific providers and then General generic detectors that look for things that look like H passwords things that are b64 encoded um Bearer tokens things like that generic passwords is the vast majority of that now I'm just getting into stats and figures on it but all right we'll go back to the

specifics who thinks they know the most commonly leaked specific Secret in 2022 I I would have thought so too but it wasn't anybody else what oh sorry he said AWS any other guesses no no other any other big providers out there anybody want to guess Google who said Google you get you get a marker that if you spill something on yourself it'll white it out um on a tide pen um uh yeah Google API keys 99.7% of all the 10 billion or 10 million we detected were Google API Keys um you can see the rest of it splits out again other skews everything and it's weird and the colors choices were not mine uh when we built this but

you can see like that's how it spreads out all right this is I can talk about stats and figures all day but the V what I want to get to is later here uh what's the fastest growing leak secret we've seen in 2023 technically open API but I'll give it to you uh you get progress software kleenexes uh I didn't just clean out my bag to do this I swear I had candy in it and didn't eat all the candy um but yeah uh this so far this year we found over 50,000 incidents open AI Keys it's not the largest number but it's the fastest growing considering last year there were B basically none um it was not even

detectable last year I went and dug in the stats and I forget it but it's a very small number that's in the other category all right but what does this mean for your Enterprise these are all interesting stats sure and for you as developers to think about but what does this mean for the companies you work for well we went out and asked we asked 507 people that said they're decision makers in it and we put together a report what they said in the voice of the practitioners state of secret and absc it's a really long title um all right only got two more of these I promise um what percentage of it leaders said they experienced a secret

leak in the last 18 months5 a little bit lower price or right Rules by the way what 100 I I think that's more accurate no what did you any other guesses 26 higher 36 36 that's a good guess too 75 75 on the nose you get you get some notepads to say Adobe on on them um 75% of respondents said in the last 18 months they had experienced a secret leak 60% of those or 60% of all people said it somehow impacted their company and employees that's the reality and that's why we need to get on top of this and solve it because we're starting to really impact the bottom line for our companies um 94% said they're planning

to improve their secret practices the next 12 to 18 months this is my ultimate problem with survey based research is people can say a lot of interesting things and this is proof we this is select all that apply uh anybody see a problem with the math on this like immediately yeah because all these it leaders like yeah I'm probably going to fix it um yeah there's we're probably deal with this yeah but we're all we're all dealing with budget it's only so much time in the day but everybody's got this plan that one day we're going to fix it one day we're going to do something about it we know it's a problem and that's what I

keep harping on um for the people that did experience and I know I went through that real fast I'll leave this up here for a second longer um for the people that did experience that 27% admitted that they were relying only on manual code reviews who thinks that's a good idea yeah I've never had anyone ear a hand on that I love manual code reviews I think there's a serious value to the manual code review how else you teach Junior devs anti-patterns how else do you teach them this is not lining up with the business value of our company that's what code reviews are great at the human to human connection pair programming that's that's get overall

Better Together finding an individual string that's wrong man humans are terrible at that that's what machines are great at um so for people that did report they had a leak how many this is their self-reported number average number of occurrences and by occurrence I mean if it's the secret is shared within a repo how many places in the repo did it appear any other guesses Price is Right rules it is 3 3.95 last piece of candy right who's I thought I said eight you said three all right that's last piece of candy and that is the end of the prize hat for now uh I got one more thing but I don't have any more questions all right

um so uh done with the priz hat now we're done with the silly Part um I did that to kind of because this is all terrifying news right like know this should could sleep well at night um so what do we do about it and that's where I want to end with the last 10 15 minutes who here is familiar with Dora metrics Dora metrics the uh accelerate any here read accelerate okay there's a book called accelerate uh and the research they did in there uh formed the devops research um I remember what the a stands for not Association but um basically this is how you measure the success of a devops organization actually let's just real

quick dor metric Trix metric yeah how to measure software delivery so you can go to well this is not the one I was looking for because the Google has the official one there we go Google Cloud um but we can take how all devops organizations perform and break it down into these four leading metrics uh deployment frequency uh assessment that's what it is research devops research and assessment I never remember the as4 um deployment frequency how of of an organization successfully releases to production lead time for changes how fast they get a change out the door change failure rate the percentage of deployments that cause a failure and the time to restore a service if something breaks how long

does it take to fix it you can take just these four metrics and judge the overall health of a devops organization so says their research which is also survey based but half the book of accelerate explains why that works um Nicole Ferguson um and other people wrote this Jean Kim um a lot of really smart people so go read accelerate so we took that idea of like well how do we map out what a successful organization looks like and we said this is wrong this isn't good enough this is what Google's entire developer documentation says if you leak a secret simply just go rearch tecture app good luck they didn't even say good luck um I don't recall saying

good luck uh this isn't good this doesn't tell you what you need to know so we think that it starts with hey use a vault cool uh e either like something off the shelf like ashy Corp Vault or if you're using Azure Azure key Vault or it'll be a Secrets manager or what have you that's a good start but that's a start so how do we map out how we get to success and I think it comes down to really three pillars you got to have people invested in the process that trained and they know that hey this is a problem we need to fix we got to stay on top of this you got to

have clear processes that are documented that show this is what we should be doing and here is exactly how we do it and then you rely on the tools because if you just throw Developers key Vault you're not gonna have success key vault is good but it's not something you do on an individual basis if you're working in a team it's a team effort same thing with hashy court Vault yeah you can use Hatchy Court Vault completely on your own by yourself as a developer but if you don't Implement as a team you're not going to find the results you want and if you try to automate things completely on your own without having the processes in place everything's

going to fall apart it's the three- leg problem REM one of the pillars everything falls apart but anyway we think it boils down to five levels and of course computer science we're off by one so we start with zero uh and this is never meant to be an accusation so never feel like hey you have to be better this is our general guide of like here's how we think teams can improve over time and a general road map for you so just a place to gu gauge where you are as an org and then how do you have that conversation with your devops leads and your operations team and your cesos and your CTO say here's

where we think we need to get to based on where we're at at the moment so level one zero is a lot of organizations that's why we put at the base of the pyramid on the secret management side they're hardcoding credentials and we already went through all the riger moral in the first part of this talk of why that's bad just everything's un encrypted you're throwing things just willy-nilly into your code base throwing it through your CI environments and there's a password in place somewhere that's hoped for the best or the best answer you can have of what are you doing for security oh we have a firewall that's not good enough um and then the secret detection side

how do you know if people are leaking credentials here and there you don't you just don't you're taking the word for it you're not even doing manual code reviews at this point again not an accus just this is where we see like the very bottom level if you're starting a project and you've never thought about this stuff it's probably where you're at and then you get people that are starting to think hey maybe maybe I need to use an EMV file because if I store my EMV file outside of the repo that's what AWS tells me to do with theab uh AWS credentials file that totally works and it does problem is sometimes people move

that into the repo and then they forget to set get ignore or they modify get ignore that it allows it to be shared or they copy paste the entire repo or the entire folder and the EMV folder into a bucket last year there was a firm that found 1.5 million git repos that contain secrets in public S3 buckets people just threw them there because they could store them out there for cheap Dore does not work outside of get servers um but okay but but we're moving in the right direction let's take um our our config files uh and let's group them together let's make sure we're storing things externally from our repos and you're starting to think like okay maybe

we shouldn't hard code that's a good first step um oh yeah and log data we finally realized that PL Tex credentials are in our log data maybe we shouldn't do that maybe we should build sanitizers in there please um but secrets aren't really scoped like everything is just fully allowed to do everything because that's fast and easy and now you're starting to periodically look through and say like okay manually did we did we check for this and secrets are in fact rotated manually sometimes when you remember it when you think it's a good idea the alarm goes off hey time to rotate that AWS key from last year again better than where you were before you're on the

right path that's the conversations you should be having is like how can we do this better if you include one of these 1% every day you're going to get to the next level pretty quick so intermediate people now we start using vaults now we start using secret managers now we're starting to scope things correctly so if someone does get a hold of it the only thing it can do is a very specific limited thing that you cannot escalate it doesn't allow escalation uh stowing things dynamically uh or you're loading the secrets in dynamically from the vaults correctly everywhere so if someone does get to your code they see vault. project name. secret name. secret that's

programmatically calling it in that's that's awesome like if I was an attacker I'd think oh man I don't got time for this got to figure out some other way in you're starting to scan things at the developer level automatically and you're starting to uh you scan continuously at the pr process the merge process the pr the merge request process that every time the code's going to get merged into your main branch yeah we've we've we're we're scanning at that point and starting to look at your build outputs to see if any SC Secrets got jammed into there so scan your Docker images your piie packages and that rotation is still manual but uh they're rotated at least

at least periodically you kind of see where this is all going and I'm starting to run out of time and I do want to leave time for questions but uh level three is okay now we're going to start doing this for real everybody is going to use Vault all right there is a clearly defined process around how to use it and rotation is clearly defined and it's a regular scheduled event and we know when it's going to happen now we start actively monitoring logs cuz we know that hey this stuff's going to get leaked let's start actively looking for it on the regular with some tools that let us do that data Dog Log IO logs IO they're all there's not one

winner here there's a bunch of ways to do this uh and now developers starting to use tools to scan before they push uh this is where truffle hog comes in git leaks uh we have GG Shield um there's if you're just looking for Pure detectors there's a lot of solutions out there on the open source um AWS Secrets uh or AWS Labs get Secrets is another good one if you're all in on AWS um but you're starting to build that into your git um GI hooks system so every time you go to make a commit automatically check the remediation process involves the developer now you're getting asked like why is this here and you start getting to uh holding

a developers account for why they're doing that maybe it's a test credential maybe there's nothing to worry about maybe you're maybe they did it by accident maybe they didn't know maybe they weren't trained maybe they didn't get the notification maybe they didn't read the memo but we can start fixing the people and the processes and the tools alt together all right and then we get to our expert level no matter where you are here this is where we think you can get to because we know companies that are at this level we work with them day in day out there's a central Vault Storage or hardcore um not hardcore Hardware managed device storage that has complete clear access logs you

know exactly who's called what Secrets when who changed things who was in the system at all it's all clearly logged and actively monitored secrets are Dynamic meaning they change constantly the best passwords are the ones that do not exist and we've gone to a system that is relying on I am rules uh wherever possible and the only passwords are the ones that are mandatory but those are very shortlived so someone gets a hold of it they probably already expired the one that have the hold on um you're charting you're checking throughout the entire pipeline at every step uh that's what I'm saying the open ID connect you're starting to replace um passwords with tokens short live

tokens um yeah you're enforcing things so so that's the the process and the people part like hey you didn't do this right thank you um how how do we fix that how do we as a team make this better now you're scanning every single commit every single time it moves even before it's made consistently across the board everybody does it if not and it still shows up there are consequences to that uh not be heavy-handed but it's hey why are you pushing Secrets continually we told not to do that here's all the training here is exactly the tools now it's a you problem a specific person problem um and monitoring is ongoing continual and secret automation or

secret rotation is completely automatic has anyone ever flipped has anyone here flipped on AWS automatic rotation or I guess AWS users in the room how many of you use the secret rotation automation it's there in the documentation it's one config right on please be proud um with all the other tools you can do it there's systematic ways to go about this so again you can download this but this is what we would love for everybody to go have that conversation with their teams like hey we think we're at a level one but we're almost a level two why why don't we talk about processes to get to that to get to hear we want this to be a road map of

where you can find success not a judgment not a we should feel bad nobody should feel bad it's a journey nobody's born knowing this stuff not even lonus so in conclusion it's a constant cat Mouse game and the stakes are very real I was talking to a colleague who said their company leaked an AWS credential they caught it within 5 minutes they got a bill for $150,000 because they caught it they pulled it out of the code but it still took a while for them to rotate it that's the stakes we're playing with now uh open AI there's a dark reading article I didn't quote it in here but actually I did quote it in here uh but

it's uh reflet um reflet has does not have any automated checking for their code uh if you leak open AI secret onto GitHub public there's a deal between Hub and open AI that will invalidate that they'll send an invalidation request immediately and you're safe you leak that same credential on RL it it will live until someone tells takes it down like you or then send your own invalidation request uh yeah somebody got um a bill for $110,000 because a hacker group found it on repet and passed it around in Discord and everybody got a free AI that day but that's the stakes we're playing against so again don't want to make anybody feel bad about this stuff and I don't really

want to scare anyone but bring it to your attention have the conversations internally I'm Dwayne I live in Chicago for rids since about 2016 maybe up on Twitter about anything and if you want to sing some karaoke or talk about rock and roll I'll be around Oh wrong direction [Applause] thanks and with that open up any questions and we do have a mic because we're on the internet right

there Hi great talk um you mentioned you scan all of public GitHub and you find these credentials I imagine the vast majority of people putting secrets into public GitHub uh small developers small companies people that aren't going to have tremendous amounts of professional experience you you also scan private repos as well do the metrics line up uh right is there a difference in the like the ratio of the metrics from like professional developers and Enterprise versus what you find on public GitHub is it comparable or is it this actually really interesting to bring that up um because we just are talking about that internally what I can share publicly because again we're dealing with private

repos and customers and ndas and all that what I can share is uh we have severity scoring on it's automatic on the system so if it's valid and it's public um it automatically gets a critical score uh we do see a a curve um I can't say exactly the numbers but we do see a curve downward of um the more professional the organization the higher they are on that list the but I think that's the the more that that curve so it's um less severe less criticality from those teams but that's a result of their maturity in our opinion that's what we think is actually happening is like yeah they're consistently using hash cour Vault so they're leaking a lot

fewer secrets that hit that criticality curve um so it's a combination of those things so it's not like one thing I can point to it's like oh they're just Juniors or that's a brand new company um but yeah I can't I don't actually can't release those numbers on on the individual but that's a good question but yeah we do see people that are checking consistently are higher on the elevation scale and there's nothing controversial or secret about that any other questions I did have one other prize and since you asked the first question it's a cord minder sorry I almost hit you in the head didn't mean to do that there you go it's Cod

reminder so you mentioned about a manual code review and um who's it oh right there yep yep okay so mentioned about manual code review and it's kind of know okay but not not that good but uh any automated tools that can actually scan this and give us some more detailed U um uh input on the on the code um there's a number of providers out there full disclosure get Guardians one of them that looks for hard-coded Secrets uh truffle security is one another one I would name off the top of my head um if you are all in on GitHub GitHub advaned security is pretty darn good for what it is um but it only works

on GitHub is like the the on there thanks very much for coming um uh but yeah there are other providers out there and then there's the whole world of sa um St um static code analysis um static application code testing sact sat I'm getting acronyms because I'm really at the end of a talk U but yeah then there's the static code analysis tools out there and those range all over from sneak to um I just there's a ton of them out there sneak comes to mind first when I the ones I always think of when I think automatic code review um but yeah if it's a combination if you use test I believe if you use testing Tools in

conjunction with humans that's the best outcome because manual code reviews again how do we improve the business logic how do we teach Juniors not to code anti patterns manual code review is best for that but it's like uh back in my website building days uh regression testing like pixel to pixel my eyes ain't that good I can't tell you the difference between two shades of purple I just can't uh back stop JS could tell me every time and it was automatic and it took seconds and I would use it for that but for copyright yeah um for actual copy I would want the person that has an English degree on the team to actually read over what I wrote

not not just trust a machine okay um so an instant I've actually had a my place of work at one point was I'm sorry can an instant where I had at my work at one point um a while back was we had a h contractor come in from a you know consulting company uh to help us build up um and install a product and a few years later one of our guys actually went hey do we know about this git repository and found on public GitHub the guy had uploaded um you know a bunch of like infrastructures code type stuff uh including Secrets um in that thankfully by the time we discovered it that test environment was

gone now the passwords were active but theoretically you know at a point in time when that was uploaded you know there was potentially a test system where there were these passwords there and like you know you can have all your uh you we we actually use Vault um and you know we've got you know our own git server and you know even if we had scanning on that what's the sort of approach when you do have you know like what's stopping a you know a junior Dev or some external person or whatever just get push to you know a public repository like how how do you find that you know when it's not really in your it's not

your repository it's who knows right like it can be anywhere if it's pushed privately you're right that the the the greatest thing that g gives us is the ability that everybody that touches a code has the entire repository the most terrifying part of git is that everyone that touches the code has the entire repository this is the the double-edged sword uh if you actually type man get into a terminal it says get the stupid content tracker it's dumb it has no idea what it's doing um if they do it publicly there are tools that will look publicly again that's where we started from that was the first thing that our Founders built was let's look at every commit on GitHub

and find out what strings there are so that's our public monitoring product is literally look for your strings uh look for your vars look for your um secrets and let you know when they pop up out on GitHub public uh we're not the only people that do that and other solutions that pop to mind I don't have I don't have another solution Pops to mind immediately but I do there are other systems I believe crowd strike but if there's anybody from crowd strike in here let me know if I'm right about that but there are other tools that also can help you monitor public places but it's that's the Toyota problem is that their subcontractor pushed a repo

publicly and didn't tell anybody and it took a security researcher five years to figure it out um there are other systems that are constantly scanning and monior and that's only gotten better GitHub itself will probably tell you at this point depending on secret it is their secret detectors are again pretty good uh and if it's an AWS key AWS is going to find it immediately AWS is constantly scanning the entire internet constantly because they have to any other questions because we're right at time but there's nothing else after this so I got a couple more minutes oh yeah a question let get the mic up here I try to be loud it's for the internet you can be loud in the room

but the internet can only hear so much much so usually when we're talking about like credentials we talk about like MFA or two Factor authentication what level do you think that would come come to if we were say um you know you're adding a pass phrase to your SSH private keys or um maybe there's other uh Protections in place so even if in the code a uh you know it is leaked maybe there's like network controls so the server can't be reached from a external Source 100% 100 that's a great Point um and that's something I guess I don't really I don't talk about enough in any of this stuff CU I'm so focused on the credential end

of it but this is one piece in the larger equation um if you talk to us at the booth the first thing I say is we are a code security platform because that's what we're focused on network security um the biggest thing I think people should work on is egress like know exactly where that traffic should be going and if you don't know where that traffic is going go fix that right now fix that before you fix this don't tell my boss I said that um is on the internet damn um but go go go fix things like that um because yeah if the call shouldn't that's the the talk I saw earlier on um MFA issues with MFA if the

Call's coming from somewhere weird don't accept the call um but this is all part of it so yeah overall comprehensive security strategy yes you should have your network so locked down that if someone does get it it doesn't work but part of that strategy should also be we rotate so fast that they get the code from yesterday it doesn't matter that that um that line in Return of the Jedi uh their code checked out it's a little old but we about to let them through like that wouldn't even worked in ancient Rome like that's a dumb idea like a password should be immediately invalid as soon as it's used or it just shouldn't exist um but yeah that's a

good point um the more secure your network is the better your security yeah or like adding pass phrases test this HQ yeah or something like that yeah yeah password Keys should be a mandatory practice I 100% agree with that is that like level two level three like I don't know oh if I was going to if I was going to map it I know try go back to the talk and oh if I was going to map it um I would say it's somewhere between one and two yeah but definitely three are going to be using it consistently cuz three is where you go you start the developers are starting to do things consistently level four is where it's mandatory

across the board everybody does it and it's building the process but I think I think level one is where you start having that conversation of hey do we do we do that oh we do now okay that that's yeah that's go along that path because there's no real hard boundaries here it's like overlapping I like that if I have time to uh one further question is if you're doing all this scanning um how does disclosure work for you when you find like hundreds of repos and hundreds of companies that are leaking these secrets if it's public yeah we email the committer oh okay and say hey do you know this is here um if whoever signed

it so if you're signing it with an email that doesn't work know that we can't reach you and we can't tell you about it makes sense um so if you're some people really think that's the security thing like I I want I don't want to use a real email and there's no git has no way to check but that's the disservice you're doing like you as a committer can't be reached now so who could tell you about any issues they find so always sign it doesn't have to be a good email it could be like a proton mail but just make sure you're signing with something where we can actually reach you all right thank

you great talk thank you any other questions other than that oh one more one more and then we'll call you the last question and then we'll all go get some coffee uh I I just wanted to point out with the SSH thing like if you use a volt or any other SSH CA like you don't really have to encrypt your SSH thing like you can use a YDC or whatever to grab yourself an SSH key and then it rotates or whatever un signing an SSH key so if if if you use a s shca like so Vault can act as an S shca okay or or you can use S sh's own CA um but if if you use vault

as the ca then you can use like uh ad or any other authentication method to grab your grab the key from Vault that it will then use to log you into the server so then you don't really have to worry about password encrypting it you're mean at that point you're signing it with the certificate Authority or that that's taking at that point you're using some other mechanism to give yourself the key and then it's signed get for you I think we're all on the same page the more security you throw at it the better it's a matter of technical implementation at that point that no you're right if um if it's if you're that advanced on it that

you're using it to sign then yeah yeah I was just saying all right cool hopefully uh hopefully that helps some people at home all right thanks very much everybody give it up for Dwayne Dwayne McDaniel