Long Live Short Lived Credentials — Auto-rotating Secrets At Scale

so right now uh you're in the right room at the right time you're going to get to hear from Dwayne McDaniel on his uh session topic Long Live shortlived credentials autor rotating Secrets at scale so with no further Ado uh Dwayne McDaniel give him a round of applause folks and we'll give him our time and attention thanks everybody I have way too many slides so we're going to go really fast hopefully we'll have time for questions but I doubt it but I'm happy to talk to anybody afterward so really quick uh again don't have a lot of time who in this room considers working in devops I see a few hands anybody here in security I think everybody's hand should

go up cuz we're all on the security team whether you know it or not is anybody here actually on a call like they're waiting for the page something good job yeah seriously like give yourself a round of applause everybody um and who when they got that call had to go rotate at least one secret during an incident anybody oh I see a couple hands okay uh anybody do this ever or the to rotate 5,000 Secrets at once because uh they worked at Cloud flare and this happened to them and I'm not picking on cloud flare here just to be clear this was a very specific complex situation uh but still at the end of the

day they said hey okay because attackers got into OCTA and because they used those credentials to break into our stuff at Cloud flare we did the right thing and to protect everybody we rotated all of our secrets that doesn't sound like a good day to me that sounds like a pretty bad day at the office uh before I go any further I'm Dwayne um I live in Chicago I've been doing developer advocacy since 2016 these days I work for G Guardian um I have stickers if you didn't come to our booth and get stickers uh I have them uh we also had a booth here so I'm not going to do a whole sales pitch about

what we do at G Guardian but uh I will pitch the security repo podcast it's a podcast about security we've had Jason hadock on Tanya Jenka uh all sorts of amazing guests and we have a lot more coming um find me out there in social media and if you want to talk about Rock and Roll I Love Rock and Roll um the good news in all of security is even though there are a lot of attacks there are a lot of there's not a lot of new novel paths people are following we know how attackers generally behave and it looks something like this they even wrote a whole framework around it um it's one of those chicken egg I think at

this point in time where if you're a new attacker are you just following this because this is how you get in are you just following the lapsis Playbook at this point anyway uh the point is credentials are baked into this every pen testing course I've taken every Workshop I've ever done is like we're going to try to get an active directory but along the way we're going to get there by looking for just credentials laying around and unfortunately this is a really bad problem last year we crossed the Rubicon and now the number one root cause of breaches according to sofos and IBM and cesa and other people is leaked credentials uh pford credentials um Uber bought super admin

credentials for $20 on the dark web and that's how the Uber breach started um this is something that's a growing problem not a shrinking problem uh G guardian we look at every new commit that hits GitHub public every year we write our whole report about it you can go download this off the website no sign up needed no email taken from you go in there in an anonymous browser download it over tour if you want we will not bother you about it um but we look at every new commit there's like 1.13 billion commits last year and there's a public API feed if you've never looked at the Google or not the Google the GitHub API uh it's it's

overwhelming that somebody built a library doio for it and it's just a fire hose and you're watching like every second there's multiple commits it's crazy anyway last year we found that many hardcoded Secrets just out there on GitHub public um that's a lot and it's growing uh these aren't cumulative numbers uh that's 2022 we found a little over 10 million that were added in 2022 and 2024 I guess we found 2023 at 12.8 million or round up a little bit but uh that many unique um I will say we do try to send emails and tell you you did this has anybody ever got an email from us about this in here anybody I know we sent it though um but

that kind of a side quest here please sign your your get commits with actual emails that you actually look at because there's people like us in the world who are trying to tell you hey you did this thing we can be better than this um the scariest stat out of all of this that we found this year was 90% of all the secrets that we did find to be valid which is a much smaller number uh but it's still Millions um 90% of those remained valid 5 days later when we went back to check I think that's because it's really really hard to rotate Secrets otherwise we do it a lot more often but attackers know this as well

just like they know the amiter and the lapsis playbooks they're depending on the fact that you didn't rotate that secret that they find an AWS key they find a um a database string with an encoded token and they just assume it's going to work or they hope it's going to work cuz that's how they're going to keep getting in but how do we get in this mess I think about this a lot actually the history of computer science and like open source and how we ended up where we are today and I think it's important to step back from time to time and not just look at the new fancy technical Frameworks and the new way to approach

to things and think about how did we get here maybe it's time to step back and rethink this because we're still living off this model uh back in the old days there were paper tapes that you fed into machines and it would do a batch function and you'd spit out numbers how do you protect that you arm uh you would Nancy gr uh uh uh Grace Hopper would say you just put a guard with a gun at the door and they can't get in they can't mess with the paper they can't mess with the hardware you're good to go and it worked and then this whole thing where we connected these big machines together and suddenly kids were playing Zork and

wasting Computer Resources and how do we stop that well you lock the door obviously but let's put username and password to know who is playing zorc and who's wasting the Machinery sources and who's sending those crazy emails from SPO Alto uh and then this thing happened where everyone in the entire world has a microm Compu in their house and they can get on this thing called the worldwide web and they can use HTML to build wonderful wonderful beautiful websites or not um but how do we secure that well username and password is worked before let's use that again how do we stop the people from intercepting and messing with those messages well we can lock those kind of like we lock the

doors that's TLS SSL and it works and then this happen and I'm not picking on click it I just shout out to them for uh making this picture so I didn't have to recreate this picture and they were just one of the one of thousands of devops pictures I found online but we got to this stage and said okay what worked before um cool we use keys keys worked that's Keys worked and then we got to here uh 2022 uh I think this is out this is definitely an outdated number now but at 2022 cyber Arch looked at into it and they're like yeah it's 45 to1 Machinery resources to humans and we keep having

this conversation about authorization and in the same breath trying to address the human being and the Machine and it's clear that we need to step back and think well we're going to authenticate humans we're going to authorize humans machines are something different though what got us to this point in history is not going to take us further but I also know that we can't just rip and replace there's a medium ground that we're going to have to live through and waddle through through the muck before we get to that golden promised land where it's short L certificates all the way so first we've been doing this and keep doing it if you are working in your

company I talked to someone here uh who's been slowly eliminating their hard-coded credentials for humans and replacing with IM rules and and that's the right approach if you can eliminate a credential eliminate a credential use roles responsibilities if a is involved it's time to MFA if you don't it's time uh I got two of these and if you can't see or for the camera uh I got a UV key and a Titan key they go to different things but I rely on these this is how I log into things like this machine uh and do my work um I think it's a good idea they're not that expensive and you can do a lot of really cool things with them

you can get into your house with them if you really want um and then there's this last one of like well we have all these credentials laying around and we can't rep place the fact that the credential exists maybe it's time we just auto rotate them and it's really simple all you got to do is gather all your secrets into one cohesive secret manager uh secret manager and then set an auto rotation policy that's all you got to do just it's just as easy as flying you just don't fall down just pick your feet off the ground and you're flying um this is the only AI in the entire talk I asked chat GPT to make me a picture of the

secret of fly is just not falling down and I love this picture so much the longer you stare at it the weirder it gets all these slides are going to be available by the way I'm going to mention some links actually all the links that I reference material that I've been talking about is all in the speaker notes I think it clutters the screen to put it on screen and again you'll get the slides and look at it so how do we actually do this caution before we go any further I didn't know where to put this slide cuz at the end it's too late don't get the idea like hey yes let's go do this and then go burn

yourself out this is true of any technology you ever adopt whether it's Secrets management Secrets rotation secret Discovery uh remediation anything just don't boil the ocean you will burn yourself out the best advice I ever ever ever got in security was you will drown if you try to understand all of security focus on one thing and slowly everything will reveal itself and that's been very true in my life so how do we do it we do it Bird by bird uh if you've never read this book this is your reading assign for me go read this book it's beautiful it's wonderful it's magical um how do you write a book about birds you go Bird by

bird it's it's that simple so how do we do our handle our secrets Secret by secret and if we take it step by step we're going to get somewhere and it's going to be better than where we're at now can't guarantee it's going to solve all your problems I'm not here selling you a dream I'm selling you there's an improvement path we can take but you're going to have to make a plan like anything else in life you're going to have to make a plan if you're going to make it happen so first you got to figure out who owns it and if you can't answer the question who owns active directory in your organization then you can't answer this

question because no one can own this approach because no one owns identity in your company who owns identity in your company if you have a solid answer then that's the person to have the next conversation with are you mature enough to actually do this whatever it is you're going to do whatever we're talking about here what are other plans sometimes you got to crawl before you can walk and you got walk before you can Sprint just know that um that link goes to a secret uh secret management maturity model that we built at G Guardian that shows like this path that we've seen companies go through of from no security to everything's automated and it's beautiful and auto scanning

happens everywhere all at once um it's possible so figure out where you're at in that path and figure out your next steps even if it's not what I'm talking about for the rest of this talk you really have to uh talk to your developers so Dev pops people in the room you already know what I'm talking about you can't just throw a new approach for the entire team uh devs work the way they work because it works going and say you got to do this isn't their traditional way of doing what worked and what works gets them paid keep that in mind and lastly you have to have that conversation seriously how do you find

all your existing Secrets but we'll get back to that one in a minute that's the easiest one to solve because if you don't have the first one you can't get executive Buy in to do any of this stuff so let's go back to my two-part plan again very simple plan gather all your secrets into one Secrets manager what do I mean by that well here is a very specific true architecture diagram of how these things work forgive my Photoshop skills or my slide Google slide skills uh basically it's a place where people can put their secrets specifically developers and devop folks can put their secrets by secret I mean credentials anything that unlocks or locks data

Crypts or encrypts data or gives access to another system that's how I'm defining secret here so that's what you stuff in a secret manager you can stuff other things in a secret manager I want want went to a talk about why can't we store our website config inside of Vault and he made a really compelling argument and then he got to the end and said this is a joke but still you can store anything in Vault um but instead of hardcoding your credentials and your code or using EnV files or config files that are in plain text the secret manager stores and very importantly encrypts the data gives you a way to reach that at

runtime in a safe way transport it safely across the wire and only run it when you absolutely need it in whatever service it is oh I have a slide that says all that very importantly it also works across your environments if your secret manager doesn't work across all your environments you don't have a secret manager you have a secret management idea going on if you have five versions of Vault systems you're going to struggle with the rest of this so that's before anything else walk before you can crawl or crawl before you can walk um centralized reporting is key if you don't have logs you ain't got nothing uh logs are the secr to life I think uh

boring as they are you can grp them and grp is Magic um and it's got to be easy enough for a developer to not fight back against you and say I can't do it this way this is terrible that is going to be very much dependent on your situation and what they have a stomach for but it's getting easier I think it's never been easier to implement a secret manager especially if you were all in on one cloud provider if you're a monoc cloud shop congratulations you won uh you did the right thing at some point in your history I believe personally this is me talking um because all of the big three providers give you a way to do this it's

secret manager secret manager and Azure key vault I like Atri keal's documentation better than I like the other two but that's not because it's better it just I think it's me personally anyway just go use that one and you're done and you can go home and feel good about yourself um if you are like 99.999% of the rest of humanity your stuff is all over the place you're on Prem you're in a mix you're using every service that ever came out in the last 20 years because you have 15 CIO or yeah CIO over that time uh what real the lifespan of a CIO is like 3.4 years which means every 3.4 years you're getting new infrastructure whether you

like it or not because they got hired to make changes um but if you're AR in that boat like most of us well Vault used to be open source it's BSL now well now it's IBM but that's a whole other conversation we can have in the hallway um Achilles cyber Arch Doppler these are the big four that I talk about a lot there are plenty others um these are just four out of a hat um we work I have worked with these four anyway personally um they all work the same way you feed into this very encrypted system a secret that you can never see in plain text again unless you do something like this which is directly

out of the hash Court Vault scripts um where you actually call the secret using some kind of secret like this or some kind of path like this is this easier than hard coding a secret absolutely not is this way more secure absolutely it is if I was an attacker and I saw this I would think I got to break into Vault now I'm going to move on to something easier but that's me maybe somebody out there knows how to break into Vault um so what do we add what what Secrets do we attack what's our plan of attack here and the last wow I'm already got like less than 15 minutes left um so Green Field is always going to be

easiest always no matter what you're doing new Secrets start there hey we got this new idea for a project hey I got this new idea for security let's see if can marry this together and be beautiful together um follow after that it's the keys to the kingdom um if you haven't collected those if those are if the keys to your uh crown jewels are sitting in an EMV file on someone's laptop it's time to actually figure out a secret management solution for that then work down the list Legacy applications less valuable than move down the list and then there's zombie secrets um I bring it up here because of what I'm going to talk about next I didn't know which

order to talk about it is but how many people think they know where all their secrets are and for their entire company not a single hand yeah because nobody does uh we live in a world of zombie of of Shadow it has been plugging the system since that term was invented in the 80s all of that shadow it has Shadow Secrets how many of them are real that can still Grant access that you have no idea they exist that's what I'm talking about zombies they're just wandering around out there waiting to get be beheaded so how do you find your secrets well you send it a company all email and you say send me all of your secrets um cuz this

is clearly not not insecure uh I'm kidding uh so how do we actually go about doing it well if you're starting with a brand new project there's nothing to find you're going to make them as you go along and that's you're just going to add them into the system for the king crown jewels you better know where those are for everything else well that's why these exist uh these are ways to go find hard coded credentials um they all work to find the credentials um I have very strong opinions about which one I like personally but I'm not going to get into that in this talk cuz they all work uh to find the secrets so if you want a CSV

full of secrets or spreadsheet go go nuts uh text file is going to be your output from uh all of you can get a text output from all of these so that's step one go find what's out there or start from Greenfield a second now now we're to the meat of the talk and you're like you've been talking for 20 minutes and you didn't get to the title of the talk yet yeah now we talk now we are so set your autor rotation policies how do you do that you can't teach a kid to bik ride a bike at a seminar is one of my favorite books of all time uh because it's true I could

have spent this entire time walking you through scripts and real life examples of how to do this but my hope is for the people on YouTube and the people in the room that the rest of this talk will be like oh I didn't even know that existed now I can go actually have this conversation internally so if you are already on one cloud provider again I thank you one because all of them have scripts already pre-written for you that can do this these blogs exist again the links are all in the show uh the speaker notes and you just go set this script up set a runner and you're good to go every system is going to have its own in and

outs and that's why I didn't want to talk about anyone specifically um but they all basically do this create a new secret some of the scripts I see say test the secret some say don't I think it's a good idea to say test it um two swap the new secret for the existing one in your secret manager if you don't have the secret manager this ain't going to work you can't do this in an EnV file I mean you technically can with said but woo that's a weird day um three test to make sure you didn't break anything in production because everyone's going to be mad if you did uh and four do a cleanup step

this is where enumeration comes in because you don't want the old secret to just disappear cuz it's not going to once it's published somewhere it's always going to be somewhere on someone's machine and some branch and something you forgot about in Shadow it so you need it for logging reasons like hey someone tried to use this where did this even come from how old is this oh we retired that two years ago according to this enumerated file we keep of all our old secrets so AWS secret manager rotation lambdas one of the many many many many many places I read I just really like the L layout of this one and I decided I'll put this one on the slide but it's

one of the many places you can go and read the free awesome open source that isws and quick just shout out to open source the Four Freedoms got us here besides wouldn't exist without open source I don't believe because one of the Four Freedoms is to go look at the code and understand how it works and we can build off of that so this directly comes from all of the scripts that I read so thank you very much everybody that wrote those and give a round of applause but we don't have time for that all right so what about all the rest of people existing in the world that have multiple clouds and multiple PR on Prem

and a giant mix of things well you still gather everything in one place and then you find out what's rotatable not every API allows you to do this and this is where the rubber meets the road and why I'm not saying you can do this tomorrow with everything you own it's a journey start with the big guys uh the big companies Circle CI slack they doc how to do this pretty well cyber Arch actually has some really cool auto rotation abilities of their own they know how to talk to all these systems and again I'm not promoting cyber Arch here specifically I'm just saying I liked their documentation when I was reading this and uh I read a lot of it

because of other reasons but um find out how you can rotate that other systems key probably just an API call in most cases almost everything can do that if if you're working with a system that can't that's a that's an um a feature request right there uh that feature request probably is already open this needs another up vote so go do that again how do you build that script the exact same way except instead of just calling ads internal or Google Cloud internal now you're calling whatever service internal if you want to get really fancy this is one of the reasons I've been reading cyber Arch documentation so hard we actually built a collaboration with

them uh it's open Source Brimstone as we call it uh the again the full URL for the GitHub repo is in uh speaker notes which I will share um but basically let's marry the finding of the secret with the auto rotation of the secret and just make that one big old system with a bunch of hops I personally think that's a lot of hops doable 100% people run this in production absolutely are there is there a bunch of points of failure that could happen with this yeah but it's better than not rotating your secret but it's better than manually spending days of your life trying to put this in place or maybe it's time to admit that using

credentials for our machine resources and and workload identities is kind of a terrible idea we only implemented it really because there wasn't another option and because computer science led us to this point that said everything else we did with sec security up to this point was a password so let make passwords on all these machines and now we're overwhelmed with them it was 46 to12 years ago what do you think it is today I don't know but it's got to be higher than that and that's where things like this come in spiffy if you don't know it is a beautiful framework inspires the actual implementation you would put into production and spiffy stands for the

secure production identity framework for everyone which means it's open source and free and you can go study it but it's like um ooth uh it's an idea it's a set of protoc not protocols but a set of how you would do this it's an opinionated set of like here's how you should do it Spire is the actual API you can put in place actually the servers and it's basically um that's give microservices their own sht or jot just at runtime just when it's needed and it only Liv for as long as the workload needs it it's very easy to authenticate this because you have a centralized server that manages this but it's also a Federated server system so you can set

up as many servers that handle these search as you want that can all talk to each other so no matter what system you're on no matter what you're communicating with between they can all say yeah that search real I can identify it with everything else on the Chain not really a blockchain but you start seeing uh diagrams look like this and my brain automatically starts thinking blockchain which is a good idea just crypto is weird um but yeah that's that's let's bridge that Gap and get all of our kubernetes all of our other machine resources CU it started life as kubernetes but it's for everyone so you can run anything with this idea and it's

not the only way to do this if anybody saw the talk earlier today from uh uh so or so security or po security uh it was literally the same concept of Long Live credentials suck that's replace them with a better way to do that and there are a lot of ways up this tree if you can eliminate the credential let's eliminate their credential so to wrap up here I want everybody to sleep well at night that is my goal working in Security is to give people ideas that they can take home Implement and you have one less alarm going off than two in the morning because that's when this is 2 in the morning it doesn't say that anywhere but

that's where I think it is where you should be getting a good night's sleep and not answering a call because everything's on fire so it's this simple gather all your secrets into one place a secure place that you can address from a programmatically addressed to get the secrets out of it at runtime in a secure way that's called a vault and then set up an auto rotation policy and there's tons of examples again my talk isn't to this is a new way to approach how to do auto rotation is to hopefully inspire you to take a look and have this conversation with the rest of your team and if you do go down this path

remember go slow go Bird by bird just focus on the new stuff first then your crown jewels and then everything else should fall into place if it works out for you or just stop using credentials for workloads um or uh manual credentials keys for workloads I should say um I'm Dwayne Chicago uh check out the security repo podcast we have awesome awesome guests and we talk about awesome stuff and with that uh I do actually have like one minute for for questions I believe we got a couple for you yeah do okay uh Aaron in the audience here perhaps still Ain oh there we go now you got a name in a face Erin is wondering I can't

see anything this is like blinding light I didn't see who raised their hand as aside from not knowing these tools exist is part of the reason developers don't do this is the overhead to the code I mean how much effort is it for developers it's like anything other system at first it's a wall learning learning curves of everything is a wall I I firmly believe this and we just approached everything as we're going to climb that mountain we'll climb the mountain if we approach it like it's not as easy as hard coding Secrets therefore we shouldn't do it we're in the wrong industry cuz wow we're never going to fix anything if we have that attitude

we'll never put MFA we'll never start using UB Keys um but you need to make an easier path so the example and I'll go back sorry for the people that in the audience that sensitive to this flipping okay um if you can build the pr for them to show them the way wow that's easy because they will say oh that's all I have to do building this and sharing this in a noncured way oh that's fine that's you can do that all day and give them the tools again your mileage may vary with that but that's what I firmly believe uh and one last question here and you just mentioned it again uh Ubbi key yeah you actually utilizing it to

unlock your Mac there in front of uh in front of you is this a feature that's natively supported by Mac OS does this decrypt storage aka the file Vault that's a great question you can set it up too I actually on this M2 I said I get into my machine I meant I get into my home machine my Linux box with it um this actually use the biometric which is a hardcoded our um Hardware based solution that I firmly trust but it's how I get into my Google accounts it's how I get into other accounts I'm not going to say on a YouTube video uh CU I don't want people to know all the services I use but in my Google accounts

I can't get into without that and yeah some other stuff so okay well thank you folks Dwayne McDaniel thanks everybody