What to Tell Your Developers About NHI Secrets Security and Governance

Okay, welcome to our next talk at Pastor Con. On this track, we are going to be considering what to tell your developers about NHI security and governance from a a seasoned speaker up here, Dwayne McDaniel. Before we do that, just a reminder to thank our sponsors, diamond sponsors, Adobe and Aikido, and also our gold sponsors run zero and formal. A reminder, you can take pictures of the speaker and the his content, but no one else in the room, please, without their consent or anywhere else in in this venue without their without their consent. There's no need to record video as it is being recorded for us and streamed. So, there will be access to that afterwards. So,

we'd like to invite Dwayne McDaniel for What to Tell Your Developers About NHI Security and Governance. Thank you much everybody. Uh, thanks for being here. Um, so I got I gave a talk yesterday. Raise your hand if you were in the talk yesterday. Cool. Because I'm going to repeat myself uh on a few slides and if you weren't here yesterday, you won't know that. So, Kenton, don't give it away. Um, I know Kent was here yesterday. Um, if you want these slides, I highly recommend it. I tried to narrow this down to less than 100 slides and couldn't do it. Um, I revamped this slide yesterday based on a couple conversations and realizing what I actually wanted people to take away

wasn't what the slides exactly said. So, you're seeing a brand new version of this. It does match the abstract. But with that, let's get going. But that's the tiny URL. That's your last chance. Three, two, if you would really like to go see in their talk because there are awesome talks going on right now. Like I almost skipped this one to go to the proving ground one. Um, tell your developers if they see a plain text password, something's gone horribly wrong and you need to work with them to solve that and make that right. Quick survey of the room. Who here works in security quote unquote? I was besides I would be shocked if that wasn't the

answer. Who here is a developer? Right on. This talk is not aimed at you. This talk is aimed at everyone else to have a better conversation with you overall. Any DevOps people in the room deploying infrastructure? Right on. Right on. On. Same qual or same qualifier to y'all. Okay. So developers, DevOps people, I'm assuming the answer is yes to you. But raise your hand if you have ever deployed something that actually touched the internet that people you did not know actually used. Smaller room, but about half the hands gone up for people at home that can't see. Uh but something to think about because what I mean by that is you have built something that uses an

architecture of some form or another. It could be past, could be SAS, could be you built it all. You could download the whole thing. You could make Drupal and it's just one big monolith in a box. And you pushed it through a build process that used get underneath the covers and it went all the way through and it had multiple tests and multiple acceptance criteria and multiple points and eventually you got approved and it finally released. So raise your hand if you gotten through that and you've lived through all of that with a team. That's more than I anticipated. So I'm going to go through a lot faster through a couple parts here. But who's ever worked on a team where

they're like, "Get this done by this date." And you know the person behind you is uh that wants your job is going to take your job and they're going to replace you with AI very quickly if you don't do it. That is Hey, what's up? >> I believe you requested a whiskey Red Bull. >> I did. Not the Red Bull. Just the right >> I I definitely I don't drink Red Bull. The whiskey sounds right, but are we late? I apologize. >> No worry. I mean, I'll drink the whiskey, but I I I don't drink. >> Well, so I I was not going to put the whiskey and the Red Bull together because that's an abomination.

>> That's an abomination. >> I was like, I will provide both, but I will not mix them. >> My my buzz buzz was starting to wear, so this is perfect. >> What What What is this anyway? >> The benefit of someone else's speaker request. So, >> thank you very much. >> There is one other speaker.

A >> thank you. >> Thank you. >> All right. Please. Apologies for the disrupt. >> Hey, no worries. I don't even remember what I requested. LA I'll never forget last year though. Last year I put my request to please accept the talk and they gave me a certificate that said I was the only person to ever have done that and I have that framed on my wall and it's one of the greatest treasures I own. Um so nothing's going to top that. I don't even remember what I put. I think it was maybe donation to charity my name. I don't remember. All right. Uh again because I'm going to go fast because a lot of you are developers. I'm

going to go real fast on this. Um, but it's no wonder that when people that are getting yelled at that have to go through all the hoops and like they need to close tickets and they are being judged on how fast they get the production out the door that when they turn to AI, which is exactly where I got this code, and it tells them to do something stupid, which is like literally chat GBT told me to do that about three, four months ago when I asked for this code. It's only getting dumber. So, um, I don't think it's getting better. I don't think it's like CHP is going to tell me to do it the

right way. So, it's no surprise when security goes to a developer and says, "Why are you doing this? Why you knew this wrong?" They're like, "You know how I work, right? You've you've seen you've seen the queue, right? Like, you know that they want to replace me with AI, right? You know that they there's a queue of developers around the street that will do my job cheaper and faster or at least claim to. And I want to drive this home because I think this is the most important thing that I don't hear from stage enough. We got to remember that everything we do in security is to get to this. This is a family enjoying an impossible game of

Monopoly. I'm not going to get into the minutia of why that's an impossible game because I I'm a nerd about board games. But that that's it. That is what security looks like when we do it right. There's no alarms going off. No one's ransomed. No one's in a panic. They are doing something they love with people they love away from the abstraction of computer science that we live in and we think is real. That's real. Well, the game isn't, but there's too many hotels like that. No, kids that young would never have that many hotels. All right, I'm off that soap box. But this is what good security looks like. And this is what a developer wants to do at the end

of the day. They don't want to worry about CVS. They don't want to worry about the technical minutia of security. They just want to finish the ticket and get home. I'm Dwayne. I come from Chicago. Help people figure stuff out. That's my entire mission in life. I got a really cool podcast. I think it's cool. We got good guests. I've had a couple people in the room on this podcast. I'd love to get more of you on the podcast. Um I love rock and roll. The band we were listening to earlier that probably didn't make it on the screen. It's called Tel Novella. They're out of Texas. One of my favorite acts in the world. I told you that too long doesn't

read works. Uh so this talk is about humans versus non-human access. I'm going to assume we all know how to implement pass keys and phto 2 and all of that good stuff. Now, I'm just going to assume that then your developers have been told this and you've got guidance in this and you have AM plans around that. The non-human side scares the hell out of me. And that's what I want to talk about because if you go to developers say, "Hey, we're going to talk about nonhumans." They're going to be like, "What? What do you mean non-humans?" Because that's like saying there's only two things in the world, a potato and not a potato. Um humans and

non-humans. Like, what do you mean non-humans? Well, I really like workload as a as a term here because that's really what we're talking about when we're talking about development. A workload is a running instance of software executing for specific purpose. Oh, that's still very broad, but that's not a car. That's not a ceiling. That's not a network. That's a running piece of software. So, that's a workload. That's a non-human that needs to authenticate to another system in order to do something. OASP definition is a little bit different. It says it's part of a larger application because, of course, OAS thinks everything's an application, and it is. I'm I'm a believer in that myself. But they actually go on further

to say well they use secrets in order to do those authentications and actually have that communication. Why is this such a big deal? Well in 2022 cyber said it's about 45 to1 the number of non-human identities that we're giving authentication mechanisms to versus every human that we're authenticating. Uh here in 2024 2025 or 25 what year 2025? Yeah it's probably closer to 100 according to research. Some people say it's higher some people lower. uh Cyberarch's own own research that it was closer to 78 whatever. This is about 100 to1 on this screen and now we live in aic AI and we're deploying APIs and things faster than ever and it's terrifying how fast we're deploying

them. I've never seen this in my life like how fast we're jamming out vibecoded nonsense. Uh I had a buddy who builds applications like weekly just for funds. It's like dude what are you doing? Um this isn't free by the way. Is I'm causing this as an energy crisis? Like that's another entirely different matter. Go nuclear. Um so why does this really matter? Because this is how crime happens. Now I didn't put the Verizon DVR numbers in here, but 22% of initial breaches are credential misuse. Uh that's a over 20 up uh that supersedes the 20% that is vulnerabilities and 18% is fishing for initial breach access. uh cyber again last year said 93% of orgs

had two or more identity related breaches in the past year. When we say identity related, yeah, it could be logging in as a human. It could be those fishing things. But now based on the numbers and how hard web applications are getting hit and how hard CI/CD pipelines are getting hit and just the raw number of breaches that are abusive of API keys that got leaked, stolen or somehow other misused because unlike people who have a sense of hey that doesn't feel right or fingerprints where you can like biometric or multiffactor authenticate in computers just say hey there's token I'm going to take it run with it. That was my entire talk yesterday. So, I'm

not going to go down that rabbit hole a lot. So, how do we get out of this mess? Well, the whole rest of this is about how we're going to put this plan in place. I'm not going to get into a lot of minutia about the actual governance itself. We're going to go line by line through the top 10 risks for OASP here in a second, but it starts with raising awareness. So, that's the beginning of the conversation I want you to have. go back to your developers and say, "Look, we got a have a conversation about what's really going on in the big picture." That's nothing to do with your work. This is about the general trend

I'm seeing. We need to be aware of the stuff I'm about to talk about, including the fact that you're building faster than ever. And we recognize that. And then we got to figure out a better plan with you. We got to document and create policies that make sense and meet you where you are. And only then are we going to put tools in place that make sense. Now, that's green field. That's perfect. truly you're going to have tools you're going to have to mold into this and you're going to say all right we have policies around these tools already you're not starting fresh but unless we raise that awareness and it's not just their eyes glaze over because

you're saying here's a new term for you and you're saying here's what it matters to you and here's why I need to get involved because we want to be safe alto together and I am not saying anybody should print this and go to their developers and say here's a top 10 list for you please do not do that please please please do not do that because you will lose them immediately because you would lose me immediately. I read 90page PDFs all the time and I hate that hate it. Uh I like top 10 lists but again I'm a nerd that likes top 10 list. Some developers might like this. What I'm suggesting is we go

and structure the conversation in basically three big buckets and have this conversation with people over time and collectively with individuals, not just team leads, the actual impleers, the actual devs and say, "Hey, we need to have this conversation." I'm steering this directly at the dev and the DevOps people. There's a larger conversation that needs to have with security. There's a larger conversation that needs with the rest of the team. We'll get to that. And it starts with ownership. That larger conversation can't just depend on your developers. It has to be everybody in the world that touches stuff, including the person who outright owns NHIS. And that's a big question mark. Again, that was talk yesterday was recorded.

Does this person, the AM person actually exist in your org? Stats say no. Uh, unless you're five billion revenue or above. You might it might be a small company that figured that out that someone directly outright owns AM across the board including your non-human identities and governance around that. Right now every company I know is scrambling to figure that out other than a handful like two of them are the largest car companies in the world have like they got that person and I've had that conversation with their um with their team their CISOs. But most companies I talked to are struggling with like who owns this? In fact, someone in this room was saying uh at

one point call out who but was telling me like their team right now is trying to figure out who actually owns active directory like is it security is it is it Greg who who owns this thing and that's a larger conversation but again back on point today because I could talk about that as a whole other talk. In fact, a giant chunk of that talk yesterday when we talk to devs, it's like look, you're the one introducing these to the world and we know that ultimately you are not the risk holder. You are part of the team that makes this risk happen, but we need you to take as much responsibility as possible to make sure

we got it right going in. And I know that's hard, but we'll talk about making it easy as we go through this. But tell them ownership comes down to basically adhering to governance policies. If you do that from your perspective, when you put these things in, know that you're thinking this through. If you open it, shut it. If you turn it on, turn it off. Does Ann Landers um sheep stole it from somewhere, but I don't remember where. But go look up an blame Anne Landers for that because if you Google that, what happened? If you Google that and Anne Landers, it'll be like the first thing that comes up if you go that list. All

right, I'm unplug it. Plug it back in. That always tends to do things. Maybe Ann Landers ghost like really doesn't like me using this. Like she is dead. Her daughter took over. I still read Dear Abby every day. All right. Um, but it is Annlanders. So, um, but it's basically that. I'm not read the whole list here, but it's like basic common sense. And if we go back and look at the OASP top 10, how I've bundled this together. Now you might argue that like I put them in the wrong buckets for some of these. That's fine. This is my ideas, not OASP officially. OASP is the full top 10. But that first one's like on

proper opboarding. So when you go talk to them about their code and the NHI is the non-UN identities they've introduced that have access to things like hey so when does this go out of service? Like what does that look like? What's the offboarding process for this thing? And right now there's no good answers. They might have one. They might be like, "Oh, well this was tied to this system and when that goes out of service planned obsolescence in two years, we'll drop it and this is how we de decommission that thing." But I'm guessing if like anybody I talked to in development, they're like, "What do you mean? It's going to run forever. That's why we built it." It's like,

"Yeah okay cool cool cool cool cool, cool, cool. So, what what does it look like if we did take it out of commission? Like, what is who's who would sign off on that?" And you're going to have some interesting conversation out of that. There's no good answer to when you sunset a thing other than when you run out of budget or time. But that's reactive. Proactively, do you have a plan around that? A good governance model will. Good governance model says every 18 months you're going to review this and see if it needs to shut down the next two. That's one governance model. There's a lot of writing about this right now. So there's no one true answer. So don't

think I'm the ultimate ultimate authority on NHIS over priv. Hey, are we scoping this as tight as we can? When you're scoping these, how are you doing scoping? And they just sit there and listen. And you're going to be horrified by what you hear. But then you're going to watch the process they go through and it's going to look something like this. Who here has ever visited the DockHub authentication permission page before? It goes on and on and on and on and on. There are like 30,000 permutations for just GitHub. uh AWS combine all like 300 services times all there it's like 90,000 permissions that you could possibly do. So, it's not a real big surprise that

when like my company did research into all the leaks, I'll talk more about like the scope or the size of the amount of leaks a little bit later, but when we look at the secrets we find out there, like 99% of GitLab APIs or 58% I'm sorry, have full access and 96% of all GitHub tokens we find had right access with 95 offering full access to the repos. Here's the ones we find in public. We know the private ones are even worse than this. About eight times worse. Um, not the scope ring permission, but the number that we find in code bases. It's not really a shocker because it's really hard. How do you help them narrow that

down? And I say you in security, not the DevOps people because it's already hard enough on you. H how how are we helping here? This is an accusation. This is this is a call to action. This is if you've never stopped to think about why are you over permissioning, sit with them and go through the exercise with them. And at some point, you're going to pull your hair out and be like, "Yeah, just set it wide because it will work that way." And I understand you have tickets to close and I we're wasting your other's time going through micro picking this out. But but but if you help them with tooling to figure out like, hey, how do we figure out what

you're allowed and not to do and build scripting around that? Not saying it's going to be easy, but it's possible deterministically. Maybe maybe this is a good use for a uh AI with some tooling. Maybe. I don't know. These are a bunch of questions I have. There's a bunch of conversations I've had. These aren't rock hard solutions. And reuse. Hey, is this NHI out being used outside of the one use case we just talked about? Probably. If it's a running piece of software like a database, absolutely it is. If it's a container that's supposed to come into existence to do one little thing, absolutely not. It's only supposed to do that one little thing. The bigger concern is like, are we

reusing the same NHI across environments? And this is just good old classic OASP top 10 like are we allowing prod and prepro to hit the same database? Let's hope not. But until you have that conversation with them and you they understand that like look you aren't yelling at me. This isn't something I've done wrong. You're just raising a they're not going to say you're raising awareness, but they're going to like no one's ever asked me that before. That's a good question. Huh? And that's how we're going to raise that awareness is by leading them down thatic path of like what are you doing? Not in a fusiontory manner like trying to understand. Guess what? The more notes

you take on that, as my friend Sean says, if you write it down, you document everything, you're gonna have a pretty clear list of like, oh, this is stuff I need to clean up. This is stuff I need to get on top of. This is where the training I need to help give people and help them through. This is a good one. Should a human ever be able to use this? Can you can set up rules. You can set up all sorts of detection rules to tell like allow lists and agents like did BTO3 call this or did a human over a Linux box called this in? Did did Ayda Love Lace the mother of all

coding? Did she curl her way in to get to your user list or like why is JQ Q being thrown? A machine would never throw that. They don't need to. They can already read JSON. But if you don't know that they're setting up a service count with the idea of like, well, I need to log in occasionally. There's a talk here earlier where they somebody did a screen test on a server they had no idea what they was doing. They just unplugged it, put a note on it with date stamp and like took them nine months to come. Somebody finally come in and pull their hair out and be like, "Look, you've destroyed a mission critical system."

Like, read the note. It's like from nine months ago because they were logging in with a service account to make sure a crown job was still running on an unrelated service using that service account that was tied to that box. Crazy. But if you don't have that knowledge of like you know a human should never ever ever ever ever go down the same path and make this call like that this application workload should do to them. It's just another call. It's just another thing that you can hit with an API endpoint. That all goes under ownership and governance. Maybe I should have called it governance. I don't know. Ownership makes it feel more personal. Governance

is like something we adhere to. ownership is like, "All right, I have stake in this." And again, developer might say, "I don't care. I'm just checking boxes and getting the ticket done." And that's how some people are going to act. Some people are going to raise questions and be like, "I never really thought about this stuff before. How do I do that better?" And if you don't have answers to that, and all you're going to do is accuse them of doing it wrong, time to rethink what you're doing. How are we helping them? How do we go from the department of no that everybody hates to the department of how do we do this properly and

faster? Well, that's where the next piece comes in because the next piece is technically very achievable and it's stuff that you just need to raise awareness on first and then give them technical solutions on second. And then there's that technical complexity bit which is only going to get worse as we go along. But that's something you're going to have to partner with them on no matter what. But we'll get to that. So, who's ever asked one of their developers like, "Hey, what's your time to live on something? What's your time to live on this this thing you set up?" like what's the time to live on that that API a key? What's the shortest amount of time we

could possibly grant it and still make sense? And in their brain almost guaranteed unless they're already using jotss or already using PKI and be like what do you mean expire? It just it's a key. It just works until it doesn't work till till you changed it or you made me change it like Yep. Yep. Yep. Yep. Yep. Yep.

Yeah, let's move on. Sake of time. Um, I was going to say more there, but yeah, I only have so much time today. Um, and then back to secret leakage. Again, these are all coming out directly off of the NHI top 10 for OASP. U maybe the most important ones, like if you see a plain text credential, say something. this is the heart and I put it in the middle of the talk like pretty much the dead middle because that's the pinnacle of the conversation and if they say well I'm doing it because that's how we do it you have an organizational problem there is something that needs to be addressed much wider than just that person

if they say oh you mean like in Jira like when we occasionally write things in Jira it's like yes that's exactly what I mean we shouldn't be doing that There are a bunch of better solutions. Let's pass it around through one password. If we have to pass it through, let's figure out how to properly use vaults very quickly. This goes real quick. I say secrets, credentials, keys, certificates um tokens connection database, connection URLs, all in the same word in my head. And I know there's fine detailed differences between all these things, but I'm just going to say secret. We're going to move on. Everybody's going to be happy that I mean all of this laundry list of thing.

You can disagree with me later, but that's what I mean. Sorry. So when I say secret, that's what I mean. When I most of the time when a developer thinks plain text credential, this is what they actually are thinking about. And again, this is a Python example that uh chatbt gave me a couple months ago. I don't think it's gotten better since that says here's your access token. So, it's like, of course, that's how they're going to do it. Especially if they're in a hurry. It just needs to work, right? Cool. It works. Good. Move next ticket. We'll worry about the security aspect later. You're going to get yelled at months down the line, but maybe you're not even

going to be there anymore. Maybe a take your job already. It works. It introduced risk. Ah, but risk you can live with. It's a business after all. We know that that's an ongoing concern and something that we even though we've been saying it for years and years and years and years and years in the industry to don't do that. We know it's getting worse and people keep doing it. Just from looking at public code on GitHub and then for the first time ever we did the research on our internal customers put out this report. It's not gated. It's just go download the thing do it through tour. We don't care. Just just go read it because it's

got a lot of great info. So I have a really terrible gift. Actually, I got a couple. One's kind of cool. The other's terrible. Um, we're going to play a little guessing game. If you've did this yesterday or you know the numbers, don't play. But as close as going over, how many secrets do you think we found hardcoded in GitHub public just in the year 2024 that were added just in the year 2024 on GitHub and public? And we to preface this, we look at every new commit that hits GitHub and everything that becomes public on GitHub. We look at all of those events. And you can too. api.github.comvents. github.com/events. So, we look at every single commit, we

scan it for a secret, email the committer and say, "Hey, you did this. Don't please don't do that." And then we move on with our lives. Um, but then we save all of that data and we put out this report. That's where the numbers come from. So, how many how many secrets were added to GitHub public just in the year 2024? >> Who I need hands because I can't see you. 10,000 >> 10,000 >> million >> 5 million >> 100,000 >> 100,000 >> million >> 10 million >> what >> 100 million >> 100 million no >> two and a half million >> I think 10 10 they say after over 10 million was the so 10 23.77 million is

the the answer u so 10 million is that you is that you that was you all right here's a terribly crocheted hacky sack and I have a flashlight that I'm not going to throw at your your head. Oh my god. >> You sneaky dude. You did ask for something. >> Oh, I did. Oh my god. [clears throat] You requested a tiny [sighs and gasps] trophy uh presented to me just for asking. So there you go. Hey, I have a tiny trophy I got just for asking for it. I will treasure this award. It will go I'll put a little shelf right above my award from last year that I got just for uh asking for them to take thing and

they wrote on it in gold Sharpie. [laughter] >> Only the finest. >> Only the finest and they I think everything spelled right [laughter] presented. Yeah. Yeah. I think they corrected it but yeah spelled right. All right. Yeah. So I do this for again something else for you a little sec later but um I do this in a fun little like jokey way but that's a terrifying number to me because that's 4.61% 61% of all repos someone posted to that's 15% of all committers did this last year. Uh this is a 25% growth from the previous year. We changed our methodologies a little bit but we've seen consistent 25 to 50% growth year-over-year since we started doing this report in 2020.

So that's 70% valid number. Um so we do a quick validity check of every secret we find to see hey does this work? the system come back and say this works and if it does uh we say that's valid. Um we took a subset of what we found in 2022 11,070 or something like that and reran the same validity checks in because it's still public. They didn't take them down. Um ran it again in 20 uh January of this year 2025 and 70% came back is still valid. Yeah, people ain't rotating. So that's that is the crux of the previous two. So when I combine them and like hey how long is this supposed to

live that's different than how long is the NHI supposed to live but permission set like how long is it supposed to expire? Uh should have a plan around that. How fast do you rotate these things? Is it in plain text? Is it in plain text? Is it going to be almost impossible to rotate? So, can we move to a better authentication system to deal with all of this? Because in sure authentication is something we're going to be dogged by forever. And the answer is yes. And I'm going to go real quick. So, whiplash on this one. There are standards coming like spiffy and this reference implementation spire that are coming that are going to give the developer a

way through an API to inject a name space uh spiffy ID into workload upon birth and then let it rotate automatically from the back end what that certificate actually says and then allow list it through the system to say you have the spiffy ID you're allowed to go in here and then we're going to separate out the concern of authentication from authoriz ization hopefully at scale because Google's been doing this for over a decade. You can do this through Spire. There's an awesome book. If you read nothing else because of me, read this book, spiffy.io/book. And if you're like, I ain't got time for 198 page free book, which is really entertaining. By the way, uh there's one

of the best presentations I've ever seen in my life. Uh it's from Matias and uh Tom back at Cloud Native Security Con that talk about this in depth with cute pictures. Uh I just actually told Cyber today like you really need to make a comic or um a coloring book out of this and help people understand this concept because it's so fundamental but it's coming really quickly. It's not just cloudnative people that are saying this uh the internet uh engineering task force you know the people that maintain the standards that make the internet happen like HTTP and IP and TCP those standards same people are working on workload identity and multi system environments we're on draft 4 right now

it's the same idea of spiffy of namespacing and allow listing for processes at a very fine grain level that separate out authentication and authorization that's where my workload definition came from earlier is from their actual documents. You can go join the mailing list. It's pretty pretty active right now. There's some big disagreements on protocol right now, but that's what we make these standards. It's we're all having conversations. Not going to get into the details of it, but you come up and say, "Hey, I am me. Okay, here's your application DNS or your process DNS record. That's you. Cool. Well, now we'll let you in the gateway and we'll worry about certificates and certificate what you're actually authorized to do

once we're in the trust boundary. And you're like, well, that sounds futuristic. Nah, it's in core of Kubernetes now. So, it has been since May. You can go read it. great article from uh Anish. Um but basically you issue these shortlived automatically rotated tokens for your service accounts and when it needs to do an image pool it trades that for a token lets it do it expires and that's how it works now it's one of the options you have since May same idea that we're going to move to these ephemeral federated or federated identities with ephemeral secrets can't really leak these things because if you leak a jot that expired 15 minutes after it was issued it's a very short window

for an attacker to back. So, they're going to need to know that's all coming. Right now, they're like, I have no idea what you're talking about with Spiffy and this stuff. That's a team meeting. That is a architecture meeting but overall, it's going to save them time. The spiffy book spells out in a giant chart. It's like five hours a week you can save your developers by getting away from they have to hand set permissions and worry about all of the minutia of how to deal with API calls and all of the credentiing. You just take that off their plate and say here issue it a credential when it's born or not credential a yeah um um yeah

credential a naming a name space when it's born a spiffy ID when it's born and the system kind of flows after that. There's a lot of minutia to that, but that's the future. It's coming three to five years. I think that's going to be the standard way we do everything across all systems. But right now, how are you authenticating to SAS and other third party systems that are way lagging behind? You could roll your own spiffy right now for your internal systems you're you should be for Kubernetes. How do you do it for these systems? I don't know because right now they're hard coding it. So, how do we get away from that? Well, we use secret managers. If

they're not already using secret managers, this is the time to get them on board. This is as important as MFA in my opinion, maybe more so because of the scale of the NHI problem. Again, 100 to one versus every one person has MFA. Machines can't MFA, but you can fault in secrets at runtime through canure secure connections from places where they're being stored at rest. And that's exactly what a cloud secrets provider is. If you're not all in one cloud or you got a bunch of legacy laying around, there are solutions. Are they free? No. Well, one of them has a free version on the screen vault. Um, but it's free as in puppies. You're going to have to

maintain it. You're going to have to actually deal with it. Uh, but the trade-off is very, very much worth it. And instead of hard coding a secret, you hardcode a path. And if I'm an attacker and I see a path into a vault, it's like, now I got to get into the vault. Man, now I got to get into the vault. I'll figure out another way in. Maybe I'll look around for the vault key. It might be in a secrets.txt file. Who knows? But now I know I have to get into a vault. And that's that's a hard much harder problem than just Hey, here's an APK key. And it just works. Speaking of just working, I don't know

what's going on here. I'm not stepping on stuff. I promise. Unplug it. Plug it back in. Oh, there we go. Wiggled. It worked. Sometimes that works too. So the benefit of this and this is what you can tell the developer directly like I know this is a pain and it sounds like this is way harder than hard coding a credential and it might be a little bit of a learning curve on it. Thank you very much. Um but once you do that we will handle the rotation for you. I will never come back and say you need to rotate this secret. You're never going to remember what permissions you set because we'll have access to that from the back end because

that will be logged. we will be able to do analysis on that based on the fact it's in a vault if it's a good vault based on our tooling. We can use these pre-written things in our system uh pre-written scripts that exist with all these cloud providers and all of these other tools that will let us rotate that. And the developers like what do you mean? So like I don't hard code it. I never have to replace it. You're never going to bug me about it ever again. Like yep. And if they still say that sounds like a terrible idea, say all right, let's keep opening bugs, man. Like, let's get your bug count as high as possible so when

you get reviewed, you'll know that you refuse to do this safely. And if they still refuse, that's between them and their manager. That's risk a risks risk acceptance right there. And of course, this all works through third parties as well. If you can't rotate an API key with an API call, it's 2025. Ask why that's pos that's not a thing. Move to a system that you can do that with. And all of the providers like Cyber Arva, Doppler, all of them, all the enterprise have the same scripts. They will help you with those scripts. Figure out how to do that at scale because they want you to rotate these a lot more often than they are. Again, the

shorter window that a machine token lives, a machine secret lives, the shorter window the attacker has to actually do anything. But Dev's got a lot going on. How do we get them to adopt adopt these quickly? Well, first you explain the significant effort that they're going to have to make of continually maintaining and updating these secrets and they're going to have to remember what they did and you're going to bother them and it's going to take them weeks of their lives or they can do it right the first time and you'll never bother them again. But they're still going to say, "But it's still slower." In the short term, it's going to be until you figure out

how to meet them where they are. I'm going to go real blazing fast through the next pieces. But I think Git is the greatest thing in the world. I've worked for two companies with Git in the name. I've taught well over a thousand people Git basics. And git is amazing, but you have to understand basically what it's doing in order to understand the real magic of it. And understand all your developers under have a very different understanding of git than the other developers on your team. No one ever sits down and says, "Hey, you have to learn git from first principles." That might happen. No, there's an awesome git book and I highly recommend everybody read it. Even though it's

outdated, core principles are still all there. Um, but most people know eight commands, maybe 16. And then people use gueies, which I'm not against. I love guies. who get cracking. I love those people. But uh it's automagical to them. They don't really understand what's happening. Short primer on git. Promise 2 minutes. Git is this awesome way to track what happens in the file system over time. You can take snapshots at any given time. Those you're checking in and saying, I want you to track these things. I made a change. Cool. Now we have a different version of that thing in my in my history. It does this by compressing the actual objects themselves that were changed. If

it didn't change, it just points back to the last time it was changed to say, "Hey, down the chain, that's the version of this thing I last changed." Otherwise, takes and makes a little compression using the zib library and throws that into an index, a single file that we call the index. Um, so when you do a git ad, that's actually what it adds to that compression of a blob into this one piece of list. basically uh it's in binary and then we say get commit and it makes a permanent record of that in the git tree. If you're like that sounds like blockchain. Yes, blockchain all came from this. It's this story paper actually talks about git and

this exact phenomenon that we are chaining together these blocks of blobs and these points of proof. Now the awesome part is you have a permanent memory of everything. The downside is you have permanent memory of everything. Meaning, if you committed a hard-coded secret and you didn't and then you took it back out in the next commit, guess what? The commit where you put it in is still there. Unless you did some kind of weird surgical maneuver where you destroyed that commit and you got everyone that's ever touched that repo ever to agree to your changes. Good luck with that. And then hope there's no shadow cache of it. Guess what? There's a shadow cache of it on GitHub. Um, and

no one else that ever saw the code made a copy of it that you don't know about. That's also the power of Git. Everyone can have a copy of all the code. Buried in Git, though, is this thing called hooks. Uh, hooks is awesome because hooks lets you do anything you can damn well figure out. Uh, it's just a scripting trigger mechanism. You pass a threshold in the workload and it triggers the script to run. So, pre-commit, it's probably the most popular one. Uh, post commit or not post commit, pre-receive, um, you can do on the server side and the user side. Uh, there are 17 and all. Go giths.com. Great explanation deeper than I get into

today. this is what all of GitHub actions is based on. Like they're like, "What if we took GitHubs and made it available on the server and they called it GitHub actions?" That's that's the history. It's a little more detailed than that, but that's all I got time for today because I'm oversimplifying. But you can make these scripts do anything you damn well please. Like hacking as dad joke. I still run this. So when I do a manual uh successful um commit, it tells me a dad joke because why not? So why don't we build good hooks that say hey you tried to plain text your credential developer let's not make that commit happen. Let's stop the commit. You can halt it. And

now let's go check on your vault to see hey does this exist secret already exists in the vault. It does. Cool. Here's the here's the path into the vault that you need to hardcode. Well let's just let's just use some said or x or whatever o maybe and just swap that out in the exact line that you did that on. Bring it up for review and say hey would you like to commit this instead? This isn't sci-fi. I've seen this work. Uh, if it's not there, let's store it in the vault and make up a path. That's a little trickier because of paths are weird, but again, I've seen it work. It's just weird looking paths, trust me.

But we can get in more of that later after after this talk. Uh, and then your system should be if we ever commit a new path, we should rotate whatever's on the end of that path once it's hits uh the once it hits the um the shared repo. But that should be a GitHub action you have set up for our PR check run. But dev's going to be devs and they're like, I don't want to run that locally. It's like, well, of course that would save you time and everyone a lot of time and a lot of effort, but they're going to get around it. they're going to ignore their git hook sometimes. Uh so

why not do the exact same thing at the PR level with some scripting with GitHub actions or whatever runner task runners you have on your git system. Again, exact same thing but at the PR level. So was it exposed in that PR? Yeah. Did we rotate that secret automatically on the acceptance of the PR and we replace that hard-coded secret with the vault call? Yeah. this is how we're going to help them do their job a lot faster and safer once we convince them that hey we'll build you an IDE but if you don't know how to get works and you don't know how to script and you don't know how to build an IDE

it's a lot harder to meet them where they are so that's my other challenge and call to action for everybody is learn the basics learn how git basically works learn how hooks work and learn a little scripting bash ain't hard it's confusing it's really deep but basic bash scripting you can pick up an afternoon oh yeah and there's a reference implementation for everything I just talked about. It's called we called it brimstone. Cyber called it something else, but it's it's out there on GitHub. The notes are in the um speaker notes or the links in the speaker notes. So the last piece of technical complexity on this list, there's no easy answer on here, but tooling is going to

help us. And it's that ongoing conversation, and this is something you should already be having with them, like how can we help you not deploy insecure code? And it's not just here's a tool. It's like, can we help you understand that you might expose some serious dangers here? Not just risks, dangers if we do this in an insecure way. And how can we help you stay in compliant? We're going to be monitoring for policy violations. They're going to become bugs. We're going to come back and we don't want to yell at you. How can we help you stay in compliance? I think part of that is explaining the way we view the world for posture

management. I have a whole other talk about security posture management but at the heart it's just answering these basic questions like are we checking for this stuff are we securing it now some stuff anomalous traffic there's no way they can deal with but we can help them with tools and let them know that this is what we're checking for and if there is a local tool that will help them if you know how to get hook work with Git hooks you can start tying it in and help them automate the process and again tie it into your GitHub automations or GitHub action automations if they miss And back to the virtuous cycle, help them raise awareness, help put processes

in place that they can follow, give them the right tools, but tools come last in my opinion. And then help train them on those tools to follow the processes, to follow the documentation. Again, there's a whole other talk we could have about human authentication, but I've really been focused on nonhumans. I'm just going to skip. There's um there's a whole world of tooling that is emerging right now and how you look at this problem of non-human identity. I think it come down to those four things. This is from the well the non-human identity management group. Those four like questions in the bigger picture that you need to ask and your org needs to ask and you need to have the

conversation globally with all stakeholders and developers and everybody. And there's a ton of tooling. Again, if you're not having the ownership conversation about how you do this and what you think needs to happen and the policies you need to set, none of these tools are going to solve this for you. There's no magic wand here. This is a giant organizational problem. But together, we can solve it. But that's the three buckets I put it in and know that if we don't deal with this now, we're never getting another chance. AI building AI is coming. And if we don't put the governance models in place right now and have the conversations with the devs that do exist, the humans who can

control this, maybe we just need another profession because the breaches are only going to get worse and they're logging in. They ain't breaking in. So if they see a plain text secret, have them say something. Then work with them to figure out how to not make that happen. I'm Dwayne. I got opinions on things. Obviously try to help people figure stuff out. Hopefully I didn't confuse anybody. Don't worry about that. That's just where I work. And there's the links again. So, thank you very much.