Cutting through the Static: Dynamic Credentials for AWS & Terraform Cloud - Adam Mikolajczyk

I'm not sure where the room host is but that's okay no worries um thanks all for coming today I definitely do appreciate you being here thanks to the sponsors uh for helping making this happen and thanks for the selection committee for letting me come talk to you I definitely do appreciate it I know it's late in the day I know we just had some cookies I know we've had lunch so I'm not going to get anything too weedy this is a relatively short uh presentation uh my goal in this is just to give you guys something that you can take home take with you something you can Implement uh probably on Monday if you really wanted to but before we get

into that all I just wanted to make a quick introduction my name is Adam mikolai chick I'm not going to ask you to spell that although if you email me you'll need to know how to do that Nikolai chick sounds a lot easier than it looks I'm a systems engineer at the Red Canary at Red Canary we're sort of part of an engineering team that does network engineering for all different parts of the company wherever they might need it if there's a need we fill it I kind of fancy myself as a cloud architect engineer with a strong focus on security um a little thing about me I live I'm just about 100 miles east of here those

of you guys who know what Blue Knob is I live on top of Blue Knob uh second highest peak in Pennsylvania by about 70 or 80 feet so we kind of consider ourselves the highest I don't think anybody lives at the actual highest one uh so yeah I had an early morning driving over here this morning I lived there my my wife my kid in a very recently adopted shelter pup that's him right now named Manny great dog but man he's kept me busy as I get older I forget how uh how much puppies can really just take a light and take a lot out of you but he's been fun anyway moving along uh so we're gonna talk

about today again real short real quick uh what is terraform Cloud if you don't know that you probably aren't here or you probably didn't read the the the description uh what are static credentials what are Dynamic credentials why do I want one over the other why is one better than the other uh and how do I get there again pretty short and sweet I'll try to save a couple minutes then for questions uh so let's just jump into it uh again if you don't know what terraform cloud is you probably aren't here but just as a real quick rundown terraform cloud is a software as a service platform it's kind of a collaborative repository of all your

infrastructure is code your States your resources your policies security access controls things like that it's the axis of sort of rotation around which all your API driven infrastructures code gets deployed gets turned from code into things out there in the world uh it talks to any number of API driven Services uh we make heavy use of it we also make heavy use of AWS among other providers but AWS is kind of our big one so this is where I'm going to focus on here so yeah that's what terraform cloud is but again I hope everybody here knows uh what that means uh statically defined credentials so terraform Cloud what does that mean in terms of that well AWS we're familiar

with has AI IIM Keys you get your access key you get your secret key uh you've got terraform you plug those in either at the code level bad idea or at the environment variable level better idea you plug those into the you plug those into the workspace and then every time terraform initiates a run it reads those variables it sends them to AWS as its credential but they're static they're they're there they're staved as part of that workspace uh again they're secure as as terraform can make them but we've seen issues in the past where you know third-party vendors do experience data leaks anytime something gets written down anything time something lives in a place it could leak

so yeah that's static defense um you can mark them sensitive obviously within terraform and that's typically what folks do as you can see that's kind of what it looks like in the terraform UI you've got your access key you've got your secret key your Market is sensitive and then in theory no human can ever read that but it's still stored somewhere anything that's stored somewhere is immutable it's written down it's saved it's it's there it it can be read it might be protected but it could be silly red uh so with short-lived credentials what are short-lived credentials well they're generated on demand rather than an IAM key that could live indefinitely we've seen IM keys in over hundreds and

hundreds of days uh sometimes over a thousand never a great thing um short-lived credentials though they're they're literally just that uh they're generated by the STS service the security token service on Amazon I think by default they live for about an hour although that's configurable anywhere from an hour to 12. um short-lived rentals utilize oidc standard it permits in a nutshell we could spend all afternoon on what does oidc in a nutshell it permits a third party to do ID verification and figure out who you are and what you're allowed to do there uh utilizes Json web tokens don't really need to know what they mean necessarily just part of the oauth 2.0 specification which is another thing we

could probably spend all afternoon talking about if we really wanted to but again short short session here so we won't devolve into that too much again specifically within AWS the security token service the STS service it's a global Service which means it's not beholden to Regional outages so availability zones regions within AWS going down shouldn't be an issue STS is kind of important if STS stopped working bad things could happen IIM essentially would fall apart so we want that to be Global and that's a great thing the credentials are valid in terms of Terror as I mentioned earlier credentials are valid for anywhere for an hour to 12 hours but as far as terraform is

concerned one and done uh whether they're whether that set of keys works with an AWS ecosystem immaterial to terraform once it's used them once it throws them out it doesn't actually store them it doesn't write them down it doesn't keep them in a database they're ephemeral they're gone foreign and that's every single run so you do a plan and apply the plan the plan run generates its own set of keys it finishes you do your apply it generates yet another set of keys so never too successive runs will use the same authentication to get us there terraform service accounts and IAM tend to have really powerful privileges most folks just sort of stick admin admin

credentials on them because in essence you want terraform to be able to do just about anything so kind of for expedient's sake most folks use admin regardless they're generally really beefy really powerful if they walked out the door that's not great um and again everything anything that's static has that opportunity you could slap it on the back of a monitor on a Post-It note you can stick it in your password manager you can stick in your secrets manager employees leave all the time they walk out the doors for good and bad reasons you know are you rotating those keys every time an employee leaves maybe maybe not we have lots of workspaces so rotating those

keys independently every time we have employee turnover it would be a real a real challenge and it has been it has been um let's see that administrative overhead though just goes away if you don't have to rotate the keys that's something less for your it Department have to worry about uh I mentioned earlier um if those if those Secrets ever leaked those environment variables again terraform protects them pretty well and there's no human way to go read those that we know of but they're still stored somewhere anything that's stored somewhere has a greater than zero percent chance of leaking that's exactly what happened with circle CI in January this year they leaked a whole bunch of Secrets

um if that happened to terraform Cloud that could be really devastating that'd be a really bad day for a whole lot of us um you'd have a whole bunch of really powerful keys out there in the world and yeah that could that could be a bad day uh and then last but not least forensic isolation since every single run is marked with a specific set of credentials if something malicious happened you always have a single instance a single run to go back to to say ah that was where it happened ideally that leads you to who did it or how it happened or when more information the better though from a forensic perspective uh so again okay so how do we do this

what's our bill of materials not much really not much at all um when I started researching this and kind of read through I was like that's that's it why haven't we been doing this you know since day one and hopefully you'll feel similarly when you hear this so you need a public signing key from terraform Cloud they publish a key public signing key every customer of theirs gets the same key it just proves that when you receive communication from them it number one came from them and it wasn't tampered with along the way you need an open ID provider that's under the umbrella of IM on the AWS side so you create an open ID provider it

references that signing key and it also references an audience which I'll can show you later the audience just basically says who is this meant to talk to in our in this instance it's it's IIM itself uh we need an IAM role this is really basic you create an IAM role standard typical everyday IM role whatever privileges you assign that role that's what this is allowed to do again that's typically administrator but not always I mean it's probably not a great idea um but for you know again expediency's sake a lot of folks just use administrator you need a role assumption policy the role assumption policy allows the Federated ID provider in this case terraform Cloud to operate as that IAM

role and then in essence whatever privileges I am has um has been granted and then you need two variables within the workspace that's it you need a variable that says true or false are we using Dynamic credentials yes or no pretty simple any second uh second environment variable that is just the arm value of that role that's it that is that is all there is to it it's actually just that easy I think the first time I implemented this it took me a little more than an hour and I was shocked it's kind of all there was to it what does that look like in practice let's take a quick quick look at the workflow

so a user would initiate a terraform run plan or applied terraform is going to talk to IAM and say hey I need to get trust information and permission grants do I trust that individual do I trust that user context do I trust where it's coming from and what is it allowed to do IAM goes to STS and says okay we trust that person we know they're allowed to do X Y and Z send the security token back it sends a brand new security token back to terraform Cloud which terraform Cloud then uses to do whatever you've told it to do at that point the key is valid again within the AWS ecosystem the key is

valid but for terraform's perspective doesn't care it's gone it will never use that key ever ever again again if it's a plan and apply the first run will get a set of keys it'll do the plan when you click off the apply or if it automatically just rolls into apply it'll get a second set of keys and that again those keys will never be used ever again okay how can we make this better still because that's pretty good uh well this is IAM so IAM allows us to place conditionals uh conditional language within I am I'm not going to get into all that there's a link in the presentation that can can take you to

all the global conditions but they're all they all still hold sway here uh so you can place conditionals like I Source IP address you could go get the list of all the public IP addresses that terraform has you could make sure that anything coming from them is coming from one of their IP addresses uh you can constrain it by the organization name which is a good idea a minimum uh the project if you use projects within terraform Cloud workspace names you can constrain it to the to single or individual workspaces Or List subsets of workspaces and finally you can you can constrain it to the Run phase you might we don't but you might have a set of

Junior Engineers who are only allowed to run plans you might have a set of senior Engineers who are only allowed to run or are able to run applies and plans you may have a approvals pipeline that kind of thing so you can segment it that way um and again IAM Global condition keys are your friend you can do all sorts of interesting stuff with them that is actually it uh there's not much more to it than that I I really hope and oops sorry one last picture of the pooch uh let me show you and this is this is this is all in the um this is all in the see if I can't do the mirror thing here

maybe not ah okay yeah we're not going to step through that's not it sorry attached to the attached to the artifacts for the presentation I've got a coded example it's got the terraform code in it's been fairly sanitized so you know keep an eye on it look at it there's a couple of values in there that you'd have to replace you know your individual environments with but that's about it it just it just it works really well like I said I was I was kind of shocked we've been using static keys for a very long time implementing this was not only easy but again it just works and it vastly improves your security posture and it

really diminishes your attack surface area and this is something that I really mean this you know we come to conferences a lot and we get big ideas and we get good ideas but we have to go home we have to sell them up to Shane we have to sell them to our colleagues we have to talk about them this is something you can go do on Monday before the end of the day you could you could have this up and running it's it's pretty slick I I recommend it highly and it's working well for us but again that's it it's a short session I appreciate everyone's attention I hope you got something good out of this are

there any questions or anything we could certainly uh fellow and white first

I don't know I we we don't use those so it's oidc compliant so I would assume any any vendor or any service provider that you know supports oidc this should work just fine um you'd have to you know consult with a vendor to make sure but I'd be surprised if they didn't so it's definitely worth looking for I know uh Azure you know you could do the same thing with Azure there's an implementation plan for Azure as well some of our folks have been tinkering with that so I have to imagine it's it if it's not there yet it will be coming I'm not I'm not positive I'm not an Azure guy myself so I can't speak to

that but I know it's it's in the Azure ecosystem oh yeah certainly uh if you look at the yeah if you look at the schedule if you click on my presentation uh there are there's a coded example and my slide Decker both in here so feel free to grab them um and you can reach out to me send me an email if you have any questions I'd be happy to talk about it I'm not an expert at this I didn't you know I didn't invent it I just kind of pieced it together and implemented it

yep yep yep yep yep exactly exactly 100 and you mitigate the issue of an employee walking out the door with those keys you can memorize it you can photograph it you can take it with you right right even yeah even if someone leaves under guspices it's still you know it's still due diligence he had to take care of that just to just to yeah any other questions or okay so go ahead

as our what

no no our identity provider we actually use OCTA as our identity Rider so we I personally as an individual authenticate through OCTA that's how I get into terraform Cloud this is how terraform Cloud then interfaces with AWS when terraform Cloud takes my code and turns it into API calls and starts to build stuff what user context does it use how does it obtain those keys and then in this case it's getting a fresh set every single time it runs sure no no it's not it wouldn't be authenticating so I could have a general vanilla user let's say I have a general vanelli user account with an AWS my IAM credentials give me minimum access to do

things um that IIM context is not in play here when terraform runs traditional old school static implementation terraform has the AWS access code access key and the secret key not of me of an IAM role we defined that has XYZ privileges typically admin right that is what that is the user context that's in play here that had been a static set of keys in this case it's like a brand new ad admin or whatever set of privileges you ascribe to it are getting cut every single time it's like you go into IM delete the key make a new ones I go into I am delete the key making every single but automatically on the Fly does that help

awesome awesome certainly good

right right yeah this is specific absolutely yeah yeah I mean I guess I have to assume other providers we'll do something well yeah

anything anything even ball so vault's pretty good we use Vault for some things but it's still written down in my mind I go back to if it's written down anywhere at all in any fashion it's it can be leaked it can be exfiltrated accidentally on purpose or otherwise if it's stored somewhere it can flutter out the window you know where whereas this that's not even an option it's a zero zero exactly exactly any other questions awesome well I hope again I really hope you guys found this useful I hope you take it back and implement it because it's it's pretty great I'm a big fan as you can see now thank you though for your time and attention really

appreciate it have a great rest your day [Applause]