Bsides Seattle: Secrets Security End-To-End - Dwayne McDaniel

I don't have a clicker, so apologies for having to look all the way over here. And they're going to try to fix this. This happened earlier to some another presenter. Uh so if you have epilepsy, I apologize uh profusely. It's not my fault. If you want the slides for this talk and you don't want to look at the screen uh or you can look at your own screen, the tiny URL up there takes you to where you're looking. Um, man, that bugs me and I'm not even looking at it. Sorry, everybody. All right, we'll just get going then. All right, so if you came in here to learn about secrets security, this is it. This is the whole

thing. Don't let your secrets leak. And that's it. That's that's the entire talk. They gave me an hour to tell you that. And if you could just promise And if you could Thank you. If you could honestly just promise me you could make that happen, then honestly that would be it. Uh things don't work like that in real life. Um here's a story. Uh Toyota once accidentally pushed part of the Tconnect repo into a public repo. They a subcontractor did this, but we don't know exactly why. Um but a security researcher eventually finds this key and said, "Hey, this works." And this got pushed in 2017 and he announced he told them that in 2022. So 296,000 customers

uh data affected. Um, maybe you saw this one uh on the news. Uh, if anybody went to the talk this morning on MFA and whatnot. Um, they covered this story as well. Oh, maybe it's stabilizing. Maybe, maybe. Oh, maybe. Maybe not. All right. Um, but basically attackers got in, stole some HAR files. HAR files are HP archive files. Those contained hard-coded credentials. Then they realized, hey, some of these belong to Cloudflare. Now, to Cloudflare and Octa's credit, they came forward with this pretty quick. there's an announcement. They said, "Hey, we're doing the right thing." And Cloudflare honestly thought they did the right thing. But um they didn't catch everything and all it took was well, one

of those keys worked and then once you're inside, well, we find more hard-coded credentials and we move around all over the place and uh 5,000 key rotated later and every system restarted. That wasn't a good day. So, it didn't technically destroy Cloudflare. Um in fact, I don't think it did any reputational damage to them because again, they did the right thing, but still things got stolen. Um, these are a little I don't have fancy slides for these. I just these are stories that happened. Um, if you get the show notes, the speaker notes uh in the slides, they take you to the links for all this, but um yeah, the common thing between these two, I'm not going to go through them in

depth, was hey, we stole credentials, we got in and did stuff we weren't supposed to do. That's the trend. If I'm an attacker, I don't really really need to be that fancy to use a key and get in and do stuff. And that's the heart of what this talk is really about because that's the dumpster fire we're living with. We are doing our best to patch. We're doing our best to train people. We're doing our best to make the world secure. And just then we're leaving keys out everywhere. And that sucks. And then we're fighting tech. And then we're writing reports. And then we're staying up all night and doing horrible things. When Oh. When we want to do is

this. We want to play Monopoly with our families. Um, I love Monopoly. I love games. I love love love Monopoly. Um, I would much rather be doing that than almost anything I do in the rest of my life. And anything I can do to make my job easier, to make the the work job just work, uh, so I can get back to doing what I want to do, I'm going to do. But more importantly, with this picture, not about monopoly, it's people. If we're not seeing this as the actual desired goal of tech, like what we're doing in security, then we're missing the whole point. If you're in security to win CTFs, good for you. If you're in security to learn how

to improve your skills in CTF to make the blue team better, to help people get back to this, then you're doing it right. I'll fight anybody on that. I'm Dwayne. I came from Chicago to give this talk. I help people figure stuff out. That's my entire goal in life. That's my purpose. It's what I do. Hopefully, I'll help you figure some stuff out today. Maybe not. I don't know. I host a security repo podcast. Uh it's called the Security Repo podcast. We have amazing guests. Everyone from Jason Hadex to Jason E Street to Tanyanko to wow about 60 80 other people. Um everything you would want to hear about security. It's a broad world. Go check

it out. I am mostly on LinkedIn these days, but I also love Blue Sky. That's why I put my blue sky down in the bottom. Um, it's a pretty good platform. Speaking of Blue Sky, uh, I don't know if y'all like karaoke. I love karaoke. And some of us are going to go over to Palmer's East tonight and take that over. Uh, they start about 8:30 karaoke there. So, 8:39ish. Uh, you want to join us for karaoke? Come on out. That's it. All right. That's the plug. It's going to be on the permanent record forever. And everyone will give a plug for a place here in Redmond. All right. Uh, obviously I love music. Karaoke. That

was Kim Gordon. I was playing in my intro. I work for these people. GitG Guardian, we do non-human identity security these days. I would love to tell you about that, but that's not why I'm here to tell you a pitch to my machine or to our our company. I'm here to talk about what what attackers really really want and how we can do this better to let them not get these things. They want our machine resources. Anybody? Why do they want our machine resources? Anybody? Bitcoin. The mine. Yes. Bitcoin mine. but also to sell it for pennies on the dollar on the black market. How much does an Azure instance go for per hour on the dark

web? Depends on how good of a price you get. A quarter, a dollar, an hour until you get caught and thrown off the network. Um, and why do they want your data? Starts with an R, ends with an Yeah, ransomware. That's it. Getting into it. Uh, that's it. like ransomware to sell it back to you to sell it to the highest bidder to prevent the company from actually using it. And that's the actual harm of ransomware when you get down to it. You stop the company from benefiting off your data. That's what they'll pay for. Oh, it got leaked. Oh, I'm sorry. That's what we get in the news. When I say secret and credential

and key and certificate and database connection string, in my brain, I've been doing this stuff for long enough that it's all the exact same word. So, I apologize cuz there are subtle differences between an encryption key and an API key. Semantically, I'm not going to be panic about this today. I'm going to say secret and credential as if they were the same word. Just going to get that out there for everybody. Related to all of this secrets, like why is this a problem? Because we keep making secrets really fast. When we think of passwords, we immediately think AM. We think human beings. Human beings only have so many passwords. We have a lot of them. But

for every single human ID that has to authenticate to something in 2022, Cyber said it was 45 to1 on average in their research of over a thousand companies. That's what that looks like. Uh I don't know about you, but I think the world sped up since 2022 and now the estimate is somewhere around 100 depending on the sources you read. uh a bunch of people doing studies on this but we live in the world of agenic AI and it's gonna get crazy real quick as soon as we enable those agents to deploy their own agents which is happening already I don't know how many things are on this screen and that's the point we don't know we don't

know what's about to explode and if we keep doing the same thing we've been doing with long lived credentials when this happens because this has already happened and we already know the terrible of this. I don't know how we fix it. So, we got to fix this yesterday. So, what do we do? What do we honestly do? Well, there's three big buckets I think we could talk about for fixing this. The first one is just eliminate credentials. This is possible. It's not easy. You have to there's a lot of determination to do it, but it is possible. IM rules. I'll talk about spiffy later, but we need something just to get rid of these long live

credentials. If humans are involved, we can MFA. We can pass key. Let's do that. Please, if you haven't started implementing Gaido and pass key, oh, everybody's going to love it because I hate MFA. I hate it. I hate it. I hate authenticator. I have like a hundred things on my authenticator. So, I have to scroll down and scroll down. It's like, oh, it's that one. Versus I just use my freaking thumb and wow, that unlocked a ridiculously long uh SSH key essentially that gets me into anything. I have to log into LinkedIn with that. Like, it's great. I think people will dig that. But the last one is we got to rotate credentials. Now for humans,

that's a problem. Humans hate rotating credentials. Machines, machines don't really care. So that's the fork in the road right now. Let's solve this problem of secret leaks for humans and for machines. For humans, I think the first two make the most sense. Let's go to roles. Let's go to pass keys when we absolutely absolutely need to prove something you have and something you are. I don't even know if something you know. Maybe I do, but that's going to get leaked. But something you are, that's hard to leak. something you have. I can clone you, but h then I don't have what you are machines. Oh, I can make new MAC addresses all day. I can do all

sorts of fun stuff. I can copy and all sorts of fun stuff. And then this happens a lot. And this isn't a human being's access. This is a machine's access to another thing. If for those of you who can't see the screen, uh sorry, technology, um that is a hard-coded credential on line 10. that is the database key. Um, this is a problem and it's a growing problem that this specifically is the problem I'm talking about. Uh, it's a growing problem. Um, because these things keep ending up on GitHub in public. I'm not picking on GitHub. I'm saying GitHub kind of won the that's where all the code end up ends up. I'm not saying it's malicious. I'm just

saying that's what happens. So, my company puts out a report every year, the state of secret sprawl. Now, if you look this up, do not answer, but I need I need five people to raise their hands because there's a prize involved. Just raise your hand because we're going to play a guessing game. All right. All right. One, two, three, four, five. You're just the first five people. Everybody else that didn't just point to put your hands down. How many secrets do you think we found added to GitHub in just the year 2024? So with how we looked on GitHub, we looked at all the commits that happened in 2024 and we said, "Hey, there's a

secret in here or not." How many secrets do you think were added to GitHub in 2024? 7,800. 7,800. 24,000. 24,000. 100,000. 100,000. 15,000. 15,000. 25,000. Who's Who's that? You got 100,000. You're the closest without going over because it's 23.7 million. Yeah. Um that's a little o that's an octopus I made. And that's that's why I do the giveaway cuz it's a fun laugh, right? That's horrifying. That's not cumulative. That is 4.6% of all repos in public contain a secret. Internal numbers, it's 38%. And those are our customers. We're trying to help them eliminate this problem. What does this actually look like? Um 15% of all authors on GitHub public did this. So this isn't just juniors. This isn't

just somebody new that doesn't know what they're doing. This isn't just students. The students do it a lot, not going to lie. We send out emails every time this happens to the committer and some people are really angry about that for some reason and some people like, "Oh, thank you. You saved my job." Um, we're trying to be the white hats here. That's literally why we do this. And we have products related to it, but that's, you know, that's how we stay in business. Um, fun fact, we went back to the fingerprinted ones we did in 2022 that were valid because we try to validate them if we can, non-intrusive calls to see, does this work? Does this return

anything other than a 404? And um in 2022 uh we found yeah millions of them but we tested 11,000 that we found to be valid and 70% still worked in January after we sent an email in 2022. That means the rotation policy doesn't exist. There's no governance policy in place around that. That's terrifying. Now what does that go to? What are those things? Well, you could have your own research on that. So we asked a bunch of practitioners like hey give us your thoughts on this like how's how good is your company with secret security and we partner with cyber arc over thousand IT decision makers director level or higher were asked and our get responses anyway um

and we said hey how strong confidence are you and they said 75% said oh we have a strong confidence and they said well how long does it take you to rotate a secret once you know it's leaked like oh on average the average came out to 27 days. We know for a fact that there's over 11,000 from 2022 that are still valid at least. 27 days is a gross under underestimate. This last one is like very interesting. Um, six distinct secret managers. We'll come back to that one. That's just kind of put that in the back of your head. Oh yeah, and 44% only 44% the 75% said we strong confidence. But the same people like 44% said, "Yeah,

our developers know what they're doing. They're not going to do that. What? I don't get that gap. That's crazy. So, what do we do to prepare for future leaks? I asked Google, and this is my all-time favorite Google doc that's still in existence. That's the whole thing. I'll let you read it if you can. Um, but basically, I I copied the interesting part. You should immediately change them and redesign your app to avoid including them. Yeah, thanks Google. That's it's not wrong, but it's like saying the only secret you need to know to learn to fly is don't fall down. Yeah, it's it's actually true. Just defy gravity and you're going to be fine. This is what I think actually

actually helps. And this is the solution not just to this. If you take one picture of this talk or download one slide, this is the one I want you to walk away with. This is the one I want you to go back to your company and have this conversation. It's not a fun conversation, but it's one I know you can have. It's like, hey, look, there are tools out there that can help us do this, but I'm not here to sell you a tool. I am here to sell you the dream of we can change our culture. We can change processes. That starts with awareness. Awareness of what? How you use your processes to not put secrets in the

wrong place. Well, where do you put them correctly? Well, that's where the tools come in. How do you get people to use tools? That's awareness around your processes. It's a virtuous cycle. The more we refine our processes, I have a deep deep deep-seated belief that processes are infinitely tunable. You can have a lot of bad processes, but you can fix those. If you have bad people in your org that won't cooperate with you and will not follow processes, that's a whole other issue I am not qualified to talk about. I could talk about tuning processes all day. Tuning tools, that's what we love talking about. We love talking about the tech. Let's skip over the people part

and just talk about the tech. So many so many conversations start like that. I I will talk about the tools. I love tools. Because if we don't do that, if we don't get people on board, then all we're going to do is swap in a different secret for the one we cleaned up. Maybe not in that exact file, but next time that same developer goes and pushes something again in a hurry. I love I'm a developer advocate. That's what my title is. And I love that title. It's like 10 Burners Lee is a web developer. I'm advocating as hard as I can because developers have hard lives. like they are doing amazing magical stuff under

pressure. People are yelling at them. Of course, they didn't do it the safest way because that's the slow way. So, who actually should fix this then? Who's going to who's going to actually take responsibility for running that conversation for getting up in front of people and like, hey, I think we're doing this wrong and we need to fix this. Who owns this? Is it the devs? Is the ops is devops? Is it the CISO? CISO is the one that gets fired. So ultimately I think they have to do it but the exec team. Yeah. Who went in front of who went in front of Congress or almost went to jail when Solar Winds happened? Uh it wasn't the CEO. Wasn't

their HR department?

It's true. For the camera, the audience member said he lied and that's 100%. Um I think maybe it should be the IM owner if that exists in your org. Might not. Um INS uh the research organization last year said actually a year ago because it was RSA last year um they their study said that until you hit the five billion revenue mark your company probably hasn't dealt with this person yet. They might be there. You might have someone's in charge of IM but go to it and like who owns active directory? your best hope is they say, "Oh, this person under this structure and that is what you're going for because then you have a

clear goal of like I own someone owns this thing is responsible and has a throat to choke along the way that reports to the CISO. They report to a CIO. I don't care who they report to, but they own identity." How they're structured is going to depend on how your org is structured. Uh for instance, GM and Ford came to different conclusions on how this is structured. They have this person and one or reports to the CISO, the other reports to the CTO. I don't remember which is which, so forgive me for that, but I've talked to both of those people. I think it's important to have that conversation. Should this person exist? Because if you go to it and like

who owns Active Director, you're like, I don't know. That's the most common answer. It's terrifying. So, that conversation of process and tools, I think we can break that down to getting everyone on the same freaking submarine. And I know what you're thinking. What submarine? Don't worry, that's the next slide after this one. One of my favorite books. I'm an old sales guy. Uh I spent the first 12 years of my career doing sales. And this is the most influential book I read in that period of my life. You can't teach kid to ride a bike at seminar. You can only learn things by doing them. I can tell you all day all day how to fix this

problem, but until you go actually try to fix this stuff, it's it's just noise. But inside this book, he gives us the Sandler selling submarine. And I have used this in so many sales processes over my life because it works. Not all the time, nothing that works all the time, but old submarines, not modern submarines, but I think up to the Nimtts class um they all were, uh, isolated by these compartments you could seal off. So if you get a hall breach in one place, you just seal off that component compartment and you're not going to sink, which is kind of important if you're a submarine. So the idea is as you move front to back through the submarine, you

need to seal off each of these things as you do them. I'm not going to go through the whole thing here because it's not really important. The idea is you go through this process and once you're at the end of the process, you don't have to go back and repeat part of that process. Once you discuss budget, you discuss budget budget. Then you got to get a decision. All right. So our plan should be something like this. Let's find all the secrets we actually have. Better easier said than done. Let's properly store those secrets somewhere. Let's adopt some better developer tooling. Continually scan for secrets, which I guess is just an extension of one. And then let's rotate

those secrets at scale.

That's what this looked like in a submarine. If we can do this successfully, we can get out of the hell of, hey, we leak secrets all the time and that's how attackers got in. Yes, fishing is the number one way that malware gets installed. If I'm an attacker, I am not going to hope an admin clicks a fish. I'm just going to start digging through data sources until I find a key that works and I'm just going to keep walking through your systems. Uber. One of my favorite parts of the Uber story was after they got in, however they got in, there's a lot of theories uh where they even got the key in the first place. Once they got in, uh

it was PowerShell scripts chock full of credentials. That's what let them walk through everything. That's how they taunted hacker one from within their own interface. First, we got to find all the secrets. Where are the secrets? Are they in PowerShell scripts? Are they in they in other places? Why is this the first one? Because, well, we can't protect them if we don't know they're there. This is the first step in any threat model. This is the first step in any good security team uh security process is know what you have. Know what you're protecting. I love OASP. Shout out to them. So, how do you do that? Hopefully the people that put those secrets out there in the world that

created those NHI, those non-human identities that have the secrets, we're keeping track somewhere, right? And hopefully they did that securely, right? Right. Hopefully. And hopefully they didn't like I don't know put them everywhere. Uh, how many people have ever put a database connection string directly into a Jira ticket? Yeah, you can be honest. That's fine. We're all friends here and you can't see it on camera. Um, yeah, that seems like a good idea when you're debugging and you're in a hurry, right? And like, uh, I about I I'm about to pass out and I need someone else to take up this ticket. Uh, all right, last thing. Here's the connection string you need to make this work.

Again, developers have hard lives. They're doing amazing work under pressure and sometimes these things happen a lot. This is where tools kick in. I obviously have a favorite on this list. I ain't going to lie. But I am here to tell you if you don't use tools today for secret scanning and try to find like through your logs through uh your Jira through your slack through your teams through your confluence through your code uh through random text files that you can get to on the network you need to uh again I have a favorite on this list for obvious reasons but you need to start somewhere. That's why I put GP on here. Get Grep is

awesome. Uh, happy birthday, Git, by the way. Talk to about that later. Um, they just turned 20 last week. But that's the first part. Let's let's get get the secrets in one place. Let's find out where they are. And all these tools will help you do that to some extent. Some with more work than others. And I'll stop about that part. So, okay, you found them. You know the scope of the issue. Now, what do you do? Well, that's let's go back to our submarine. That's the next part. Now, we got to properly store all those secrets. Some of them might be properly stored. So, it's important that these tools can also reconcile with the reporting out of your

other tools where you're properly storing them. So, when I say properly store, I mean use a secret manager. Uh the next slide has all the logos on it, but this is a very highly accurate architecture diagram of how secret managers work in a network. Um, this is the entire ability of my uh my this is my entire graphic design ability. So, so apologies about that. Uh, oh, and it's got my old Twitter handle down here because I didn't update the slide. Great. Uh, sorry about that. Don't go to Twitter, please. Um, I don't use that one anymore. Um, but Secret Manager sits in the middle of everything and it does these things. It allows you to encrypt

your secrets at rest and in transit. Maybe I'm using encrypt and transit wrong there, but in secure and transit over MTLS, uh, make this available all across environments. It needs to do that in order to work, right? Uh, it needs to have some way to centralize report of how the secret got in there, who's accessed it, where it goes, and then it needs to be easy enough for a developer to use because if a developer won't use it, then there's no point in having it. Now, good news. If you're all in on one cloud, you already win. Good success. You should jump for joy. Uh not that anyone ever is single cloud anymore. Maybe if you work at Google you

are but uh these all have this built in. Key vault secret manager. Secret manager. Actually AWS secret manager is extending out. They have secrets everywhere for the last couple years. It's a newer thing. Um but can go cross cloud now. And then you have your classic giant players. Hashcourt Vault, which used to be open source. Now they're BSL, but it's still free to use. So I would start there if you have never deployed a secret manager. Great tutorials online. I love that company. I love what they do. Other than the BSL thing, uh Cyber Arc, straight up money. A keyless Doppler straight up money. But uh our commercial products, not going to say one's better than the other. I'm going

to reference Cyber Arc more than the others simply because I did. Uh no other reason than that. And I like their docs. And of course this one from Vault. Um, so how do you use these secret managers? Well, you put them and you get them. You put the secrets in and you get them out. That that's it. You read the secret or you you put the secret. Uh, so this is the actual string you would do to get a secret and then the rest of this is just to parse it to show you that yep, we got the expected password. That's literally copy paste from um, Hash Corps documentation. That seems pretty straightforward to me.

Now, a developer might tell you otherwise because they're used to just throwing the secret directly in versus let's go log it properly and or short it properly and then let's pull it back out properly. Remember earlier I said Google's not wrong about redesign your app to handle this. The best talk I saw all of last year, maybe ever, about this subject, how do you rearchitect your app to do blue green deployments of secrets in a non-downtime way is literally this one from Kenton. Um, again, if you get the uh slides, which I'll put this link back up at the end, uh, there's a link directly to this because you do have to account for that in your app and your developers

have to account for it. Now, when you attack this problem, I would go green field first. It's easier to learn one a green field project and then go after your mission critical because if you start with mission critical, you're playing the high stakes game that you get it right versus this new system you just implemented, it's going to be a little bit lower stake unless you're deploying a new mission critical system. I don't know you. Uh and then go after your legacies and then find your zombies because when you do the first thing of that's find all the secrets. I guarantee you if you audit that, you're going to find API keys that go to something and

be like, "What on earth is this? When did we deploy this?" I have never met a company that didn't have that over a certain size, I should say. There's just zombies hanging out there. They're still up and running. They're still eating resources. Someone's paying for it. Speaking of paying for it, who's who's actually sponsoring this project? Again, this goes back to ownership. This goes back to that conversation of process. Is it you? is security the ones like, hey, I'm the one going to go to jail, so my budget goes to this to fix this for everyone. Or maybe this is one of the kind of things where, hey, you know what? This is so important that we

need to pull some resources to make sure that all of us don't go down with this ship when we blow up. Maybe I don't know. So, how do we get the developer to use it? Because that was the third one, third part of my submarine is better developer tooling. But how do we get them to do it? Because right now we know that the easy way is the insecure way because they're just going to hardcode a secret. We know that again because all the things I talked about earlier.

So is there any governance around this like getting them the right tooling? Sorry, actually I just realized that's in the wrong talk. I was meant to go right here. Apologies. I haven't given this talk in a minute. Um, the problem with most security tooling, actually if you ask a developer, all security tooling, is it makes you want to go slow. It's going to throw alerts up and warnings and block you. I have never met a developer who said, "Yes, I love sneak. I know it makes me better secure code, and I therefore I love it in my ID." I have never met a single developer who said that. I've had developers who say, "No, Sneak's

okay. I got I have to use it." So yeah, yeah, that's how they kind of see this world of security tools. What they want is tools that make them go faster. That's why they like IDE plugins. Uh helps them go faster and do things and scaffold. That's why machine assist or AI assist bots have become so friaking popular. It's not just because someone's shoving AI down your throat. It's because it saves you the time to go look up that specific way to format that string or that call. And I never want to build a 21 switch statement again. So, I'll just have it architect that for me and I'll just fill in the gaps. But we can start this world of

like, well, I want to go fast, but every time you go fast, we need to implement a new security tool to stop you from messing up with that fast tool. And we start getting into Zeno's paradox world of, hey, we're fighting the tools are starting to fight rather than we're fighting the same battle together. And that's where the superhero of any organization should come in. Security engineers are the real heroes of security. Oh, we can have a whole conversation about that tonight at karaoke. But these are the folks who are in the trenches who have been developers or engineering leads who know what it's like who work with developers day in day out and like how can we make your life

more secure and help you do your job faster? I've met many of these in my career and they're awesome people. They write unit tests for developers. They write tools for developers that help get out of the way and make their job easier. Because if the easy way is also the most secure way, developers are lazy. That's a virtue. Told you Git would come back up. There's this one magical framework protocol language. I don't even know what to call Git. It's not really a tool anymore. It's something else. It's something beyond that. But it turned 20 last or yeah, what's today? 16 18th. uh 11 days ago it turned 20. That was the first mention of git ever was by

Lionus Torval in an email on the Linux said, "Hey, here's a bunch of scripts I use for patches. Um don't think it's ever going to become a full source control management system." Three months later, they renamed the project git officially. Who knew gets put in charge? Who's still in charge today? And 97.8% of all code in the world relies on Git to get it around, to version control it, to do a lot of cool stuff. So why Git? Why why Git? Because we're again it's already in the workflow. Getting a developer to adopt a new tool is like pulling teeth. If they didn't want to choose it, guess what? They're already on board with Git. They already

love it or hate it, but they're already using it guaranteed. Maybe they're not. They're like 3% of the world's not, but most likely they are. So why can't you go to them and say, "Look, you've been committing secrets. We know you've been committing secrets. We have your logs. Um, how about you run this this uh git hook? If you just run this git hook, you can hardcode your secrets all day, but what's gonna happen is it's going to go see if that secret exists in a vault already. If it does, cool. It's just going to return the path to that, and then we're going to use said or something to replace it in the line. And then all you got to do is say,

"Hey, this works." And finish the commit. That's much much better than like, hey, you need to learn how to use Hashi Court Vault. One of those conversations gonna go a lot smoother. Now, will a developer always do that? Absolutely not. Dev's going to dev. And that's why you got to do continual secret scanning. That's where you go back to, hey, the same tools you use to find all the secrets in the first place. You got to go back and do it again and again and again. It's got to be continual. It can't just be a one-time exercise because we are launching new things faster every day and we're helping machine our machines are helping us do

it even faster. Again, agentic AI, chat bots, coding assistants are helping us do our jobs way faster but not always more securely. But if we could do that at the Git uh Git hook level, then we could also do that at the PR level. We could also do that at the get action, GitHub action level. Let's never forget GitHub actions is just fancy git hooks thrown on a server somewhere. That's all it is with different formatting. You can use YAML or is it YAML? Yeah. Um it looks a little different, but that's all it is. So if we could do that same thing of like, hey, the PR is going to happen. Wait a minute, we found a secret. Let's

go see if it's in the vault. It's not in the vault. Let's log it in the vault. What's the path it comes back with? Hey, let's change the code. Update the PR. Here you go. This works now. Upon finish, let's go rotate the secret. But it's not just code though. This is the problem ultimately is this is why continual scanning is not just a PR fix. We got to raise awareness like, hey, stop putting credentials into config files. But if devs are already on board with we're going to use this vault every time because it's safer and ultimately better and easier and I stop getting bugs and I stop getting tickets I need to fix then putting the string to call hash

hashcart vault or cyber or what have you into Jira uh that's not really a problem is it because I would still have to have access to get into that and there's a whole other world of that. If you want to see a reference architecture for those crazy things I talked about of, hey, how do you check in the vault, see if it's there? We built this with Cyber Arc, it's just open source. We sell this as well, but again, I'm not here to sell it to you, but if you want to go study how it's built, build a middleware layer that does this. We called it brimstone. They never did, but you can just go the

link to it is in the um again the show notes. To our last one, automatic rotation at scale. This is kind of the magic secret sauce. This talk is derivative of an earlier talk that is just about this circle right here called long live shortlive credentials where I look at this a little more in depth than I've got time to today but remember back on slide 21 and 22 but probably not because you don't know what slide numbers they are these are um but shortlived secrets rotate credentials much more often again machines don't care machine doesn't care if the j the jt lasted 04 seconds or it lasted a year. It's not going to complain it needed to update

its p polic it it it's just going to take whatever it's got and just run with it. So to automatically rotate a secret you need to do something like this. Create the new secret somehow test to see if it works. Maybe that's optional. Swap it in for the new secret. test to make sure nothing broke and clean up like in some internal labeling to say this was the old one, there's the new one if you're if you're keeping track in inventory. So, where did that logic come from? Am I making this up? No, I literally borrowed this from AWS. Well, yeah, you see you're scratching, you're shaking your head. This is 100%. AWS already built all these scripts for you.

Again, if you're already using AWS and you're all in on it, congratulations. You can just use what they built to do everything I'm talking about. Just do it. There's no reason not to. They have blog posts. They have awesome scripts. All of them do. If you're on like something weird I've never heard of, maybe they don't. But anyway, but if you're on prem, mix, hybrid, what have you. Now, the question you have to ask is, can I rotate that secret for that service through an API call? Some systems is absolutely not. We still need a human to plug a USB key into something. And I would challenge you in the year 2025 to ask why you're using

that system. Now, you might have a real reason. You might have an air gap system that controls nuclear codes. Please don't update the new codes with an API call. I would much prefer a human in the loop on that one myself. That's just me though. Good news though is most systems today, modern systems will allow you to do this. And good news is these enterprise security managers and these vault systems like Cyber Arc, like Hashy, like Doppler already know how to do this and have the scripts ready for you of like, hey, you want to update a secret from Slack, we have a script for you on that. Just okay, here's slap it in and let the

machine worry about it as fast as you want to go. And now you can rotate secrets every 30 seconds, every minute. There is network overhead on that. there is overhead of the middleware you need to run on that and that's normally in the co conversation when I have this conversation one-on-one with people there's like that just seems like a lot though right yeah there there's there's another method to this we just accept that secrets themselves are terrible and like we don't want long live credentials and if we just think hey the goal is to eliminate wrong live credentials then the rest of the problems kind of poof go away my argument is always this let's

Start with Spiffy and go out from there. There was a talk earlier today about Spiffy. Yeah, I got a thumbs up from Spiffy fan. Spiffy um secure secure production identity framework for everyone. I didn't have enough time to explain Spiffy in all its grand glorious detail. Spire is the reference implementation you can actually use in production to go implement this. But in a nutshell, when a process is born, when a Kubernetes worker is born, when a thing is born created, you know how it's created and you can hash against this trusted build step. These things are true. Therefore, we trust this thing is its thing. We're going to issue an API driven or an A we're through an API.

We're going to issue a 509, X59 or a Jot that expires before the expiry. We can swap in a new one because API or we can take that one verify that you are you the identity and just keep swapping that in all day as fast as you want to go. We can name space instead of having a secret that anyone ever knows or writes down. And it's secretless the same way that serverless is serverless. There is a secret somewhere. But just like your pass keys with your thumb, I have no idea what that string looks like. I have no idea how long that is. I hope it's 524 characters, but I don't know. I'll never know. I don't really care because

it just works from a developer standpoint. All I do is namespace stuff. Spiffy colon the path. Now I'm good. Uh we can even federate the certificate authority there and it could just run inside cars everywhere. There's a whole book about this. Again, I don't have time to get into it. It's one of the best tech books I've read in the last few years of my life. It's free. spiffy.io/book download. It's PDF. It's 198 pages. It's a great read. And if you're like, I don't want to read a book, there's one of the best talks I ever saw in my whole life last year. N cloud native security con was from Matias who was one of the artists on the

book or authors of the book and Tom who I don't think was in the book, but he did help with research on it. Again, that's where you can go watch it. Um, and they use AI generated not Disney characters. They specifically specified that it not be Disney. So guarantee nothing is copyright in there because they specifically spasked the AI did not use Disney or copyrighted materials. So none of that's Disney, but that's my great hope. Like we can get to a world of that, but you can't we can't just jump there on a green field project. You can. If you're going to start a new company tomorrow, maybe you can, but we have to start with what do

we have on the ground now? And that's that's the process I talked about earlier because again it's this is coming and we we might already be here and the number of secrets that are spilled every year leaked in public isn't going down and it's not going to go down until we do something about it. This is one plan you can take this slide and this deck to the rest of your team like here's a plan I got. How do we do this? But more importantly raising that awareness with them like, "Hey, we need to do something. What processes do we have to try to get out of this hell? How do we train people about this? What

tools can we put in place? I have my personal favorites, but you can do what you do." Because at the end of the day, I don't want to talk about tools. I don't want to talk about these processes. I want to be back with my family and doing what I want to do, hanging out, playing board games, because that's the goal. That's the goal of security. I'm Dwayne coming from Chicago. Check out the Security Repo podcast. It's pretty good. And uh hit me up on LinkedIn and uh yeah, let's go sing some karaoke tonight. Thank you very [Applause] much. So I have the room for next few minutes. So if you have a question, I

think I can take like one or two questions. I have 10 minutes to ask. Oh yeah, we got 10 minutes for questions. Got one way over there. I we could fill Donnie this. Hold on, hold on, hold on, hold on. I'll I'll do a Phil Donnu. Who is old enough to remember Phil Donnu doing this? Um, so more precautionary statement because not only are you deal with your hard-coded creds, but there is a trail of creds behind that that extend way beyond just the initial you have differentials things you just it is a living heck to um to get rid of those things. it it just becomes like this this monstrosity of spaghetti credentials just sprawling as far as I

can see. So yeah, it's really it's it's hard it's hard to nail down especially if you have a large infrastructure then you're looking at this at scale. It's not just one co it's not just one hardcoded thread. They're everywhere and it is it's tough. You probably can't do it alone. Thank you for that. Absolutely. Uh I love the shirt by the way. That's great. Uh any other questions? Okay, I'm back there. Uh, that's gonna be hard to get you the mic. If you could pass that back. I can just want to understand what is your recommendations or suggestions dealing with those legacy and zombie secrets because I know for the cloud secrets those those are the secrets which has

APIs and then secret management easily deal with that but for those still used by human and then where the API for rotation even not exist what is the best practices mic thank you if the I just realized that's not going to pick up on the my mic. Um, so for legacy and zombie, uh, what's the policy on that? Because those are probably not going to get put into this rotation policy is what you're saying. Yeah. Um, zombies, kill them. Kill zombies. Shoot them in the head. Um, that's a conversation of like why is this on? Who's paying for this? And as soon as someone that realizes they're paying for it and it's not needed, that's how you kill zombies. Cut it off

at the power source, which is the paycheck. Um, for legacy, that is a hard question. I wish I had a better answer for. And that is where fast rotation is the best thing you're going to be able to do. Fast rotation might mean a year. It might mean six months. It might you might get it down to a day. Don't know. It's going to depend on the system. Um like so much in security it depends on what it's actually doing. Now if human beings are interacting with these things uh because you brought that up as well. Um now you have a question of governance like why is a human being using the same API key as a machine?

That's a much wider conversation than just, hey, how do we rotate this thing? It's like, why why are we allowing a human to touch an API key in the first place and that's going to lead to a whole unraveling a sweater of like, oh yeah, that's just painful, isn't it? So that's unfortunately that's the best answer I got. Any other Oh, this guy. I'm not running. I was jogging. All right. uh as someone who looks at these uh pipeline scams and uh has found many secrets and gone to devs and said, "Hey, can you get these secrets out?" One of the things that constantly gets thrown back for many issues, including secrets, is always the concept of sprints. Oh,

yeah. And hitting release cycles and release targets. How do you deal with being the blocker for a product that isn't even yours when you're trying to help secure your organization? For Mike, for those at home, uh when developers say, "Hey, but I got this sprint. I got this deadline. I can't deal with this right now. That goes back to ownership. Again, this is not a quick easy I wish there was a quick easy answer on this. The best answer I've heard on that one is a bug is a bug is a bug. If you go to their manager and like how do you pay this person? This is a power move, but I've heard this works. Go to their like,

what's the acceptable bug rate? And if a developer is like, I don't care, and is pushing 40% of their code as bugs, there's another developer who's already applied for their job. That's the reality that again, dev's lives are hard. But honestly, don't be a jerk about this stuff. If they treat it if if they can willingly push bugs into production and their manager's okay with that, there's nothing you can do. But if they treat it like, whoa, you're saying that they have a 10% bug rate. Oh, we don't allow that. Now you have an ally. Not about security, about bugs. Translated as bugs because that's what it is. A bug's a bug's a bug. All right, one more back

there. Just for my running around sake, this will be the last one. I just want to understand what is your suggestion about the what is the best practices or frequency that we should rotate the students? Oh, I love this question. I love this question. The what what is the best frequency for rotation? The ultimate answer is just in time for the ephemeral life of the request. I'm not joking. That's what Spiffy can allow you to do with tenfoil hat mode. can say I need to get this one call as a workload from this database. Here's your jot for literally 30 seconds. Your JDT that lasts 30 seconds just to do the work. And then the next

time that M uh pod goes back to get more data, it's like I have no idea who you are and I'm not giving you anything. That is the ultimate ultimate dream that we get just in time ephemeral secrets that poof out of existence once they're used. Wow, that's really hard to get to. Um, unless you're dedicated as a goal. So, Spiffy, I I didn't say that, but um, that didn't pop out of just open source brilliance. Uh, Google's been doing that since 2010. Talk to a Google engineer about their internal API keys, and they're going to laugh at you like, "What do you mean?" They don't they don't use Spiffy. They use a pre precursor to it, but they namespace all

the processes and they have ephemeral just in time. You can do this at scale. If you can't do that, then you have the question of risk. What would happen if this was alive for six months and an attacker got it? What damage could they do? Are we willing to live with that risk? And if everyone that has the budget is willing to say, "Yeah, okay. We can just live with that for six months." Or they say, "You know what? You know what? If we can limit it to two days, I think we can do it in 48 hours." That's your answer. It's whatever your team's willing to put up with and the risk they're willing to eat. Because

that's at the end of the day, that's really all this is with security is like we're we're trading risks. There's no perfect. There's no you're never going to win security. That's the horrible truth that we live in. We can get better at it, but again, I just want to go back and play Monopoly. That's all I wanted to do. All right. Well, thanks very much everybody.