Federating AWS CLI

I'm going to go ahead and just get a quick show of hands how many people are actually an Amazon customer or have been in the past all right so you guys have probably had a little bit of experience with I am roles or I am in general as well as CLI um and I know everybody's really thirsty I heard the liquor liquor cabinet was locked up with some crypto ransomware so hopefully we're going to decrypt it after this we can all have some beers all right introductions uh my name is Paul Marino and I the security engineering lead for Pinterest um I'm focused on Federation of all the things and I like unicorns and cats hi I'm aan elwa I am a veteran

consultant and architect and I do a little bit of everything so I'm not focused on being focused and I like long walks on the beach thanks all right so here's the agenda we're going to go over why we're doing this um how we can fix this uh how ays Works which ays is a solution that you guys will all see um a live demo of or sorry a recorded demo of uh problems encountered when we were building it uh some potential future problems we imagine we'll see down the line uh some pre-release improvements uh some future improvements as well as uh some time for Q&A so why are we doing this so here's the problem many organizations have IM

IM accounts with poorly controlled credentials um that have potentially destructive access to production so that means you have a developer that's got some slot practice and hygiene with the way they handle key data and that ends up on the Internet or or out of their control out of your control and then it has damaging impact on your environment uh so despite tofa enabled accounts uh accessing the Amazon CLI still only requires an access ID and a secret key to utilize it so a little bit more on why we're doing this uh this this product will reduce the plain text stored keys in repos and on Hardware uh I consider everybody body's laptop and every developer that we've got at Pinterest

kind of uh a risk especially considering it has Sensi of key data such as this um this will allow Federation and directory Services uh this automates the onboarding and termination which is very important uh some it departments have Sloppy practices about terminating all the access uh and this also enables the use of IM roles for users and service accounts uh I don't know if youve got a lot of familiarity with it but uh roles are the way to go because you've got a larger policy size you can you can app so you add cool things like uh the ability to restrict what source IPS uh the Amazon command line can be accessed from uh and this product will also allow

the centralized use of existing tofa uh such as through your VPN uh so that you don't have yet another OTP on your phone uh I think I have 12 just for my current role so here's some examples of the problem that we've seen uh you can see there's a security week article here about attackers that were scraping GitHub uh as as well as other locations for plane Tex access IDs and secret keys out in the wild U and they basically use them to stand up Bitcoin mining operations uh and sadly I found there's a lot more open source solutions to actually scraping credentials than there are to solve this problem so another example uh I'd say

this was uh also pretty lucky uh these people got popped with a $2,300 $2400 bill um because some hackers decided to use their account to launch a bit coin mining operation as well uh it's lucky because it was only $2,400 approximately the larger example would be these guys so this is an actual quote by Anonymous somewhere uh so code spaces if you guys don't know uh was dosed uh with some Ransom uh and they before they realized they actually had somebody that had unauthorized access to their Amazon account uh what they did is they ultimately for their infrastructure all the way down wiped out all of their backups to the point where they were no

longer to do B no longer able to do business uh that's really sad for a company that's actually touted uh their ability to protect their customer data from catastrophic events such as such as this so how hard is it to find credentials out on the Internet it's actually really [ __ ] easy uh it took me about 60 seconds on paast Ben uh and I found just a whole set uh actually did have the full link here but I felt it would be embarrassing to actually leave it up so I decided to blur everything out um another quick example uh also found on pin uh someone that was also uh gratuitous enough to actually tell us what region they're

running their Amazon instance in to let's see here so let's say I had the credentials what could you do um well the easy thing I would probably do is just go ahead and get a quick list of the Route 53 uh and see what hosted zones they have uh as you can see uh it'd be pretty easy to get a quick list and then all I had to do is run that one liner there to delete their their entire hostage zone so that probably wouldn't completely eradicate them off the net but cause a lot of complications for them for a set amount of time so or we could try to do something like this where I actually just want to

go and create my own user uh we're just going to go and create a pony and then add that user to the admins group um so after that I can pretty much do whatever I want from the web console to um beyond the CLI so how can we fix this so we had a couple goals in mind when we decided to start uh development of arys uh we wanted to get the IM keys out of the hands of the developers um and out of the users and even some service accounts in some cases um and allow them to still do business as usual some pretty basic goals um from here I'm going to hand it over to

amen so if only there was a credential service that would give us temporary credentials oh wait that's Amazon STS but so we need a middleman that can also talk saml and elap that will do some like you know some glue in between there we found shth which is an identity provider out there that does SSO and such sorry we're supposed to have two mics no no no second mic so shth um it's an SSO as I said um it allows for a tribute based uh exchange framework it also does Ro based but I'm not going to get into detail um it can as sle metalware uh it can also act as an auth authentication and authorization

metalware and has a soap and HTTP binder there's tons of documentation out there on this and we chose this because of that um also there's actually Amazon specific documentation on how to Federate it on the web console um but not actually anything about getting tokens that was what became a pain in the ass so I'm going to go ah and for this over to aan and he's going to tell you a little bit more okay so now we have um shith STS and out of that AIS was born so putting it all together so coming into the project first of all uh we wanted to establish what are our knowns unknowns and what are some success factors so

some of our knowns we knew that um shth can talk both ldap and saml okay great STS also talked saml okay another good thing um also however we did find out unfortunately that STS only supports 1 hour credential temporary credential tokens okay uh that's fine but at least we know that ahead of time we also know that developers don't like change and are very very whiny okay so I think everybody here most people here have experienced if you're a developer I'm sorry um anyway so some of the unknowns um what are we going to do with these 1hour credentials how are we going to make these work and we didn't know that STS was not really ready for prime time

so um there was buggy there a lot of bugs a lot of documentation issues um I got to know a lot of AWS support guys by name so that was interesting um things that were success factors coming in was that um we wanted to make sure that business as usual happen for the devs so they wanted to no change for them as much as possible and AWS support was really good uh they were really helpful on this so what's STS well STS is a uh credential service you give it auth ication it gives you credentials that's it um there's three items up there out of the I think five total uh one takes saml the other one takes oath um it's um

but some of the limitations is that you can only have one role so if you have users that have multiple roles when they go sign in they'll be asked to do multiple roles and for our purposes where we wanted to automate it that wasn't useful for us so we had to limit users to only having one role um also the 1hour credential thing was really a big limitation so what do we do well um first uh Paul was thinking let's get a PC together so we we brainstormed and said okay let's get a PC for the flow and actually figure out how to if it's even possible not necessarily automating it not necessarily making it clean for

the developers but uh just making it work so was a lot of development a lot of pre-work some of the irns had to be added as attributes for an ldap uh we have to make sure that shth can consume those so making shth work with ldap and uh making sure that ads liked our stal assertions so a lot of trust policy a lot of um finicky details that STS was very particular about rightly so um in making sure that it's accepting our assertions so and then at the end get temporary credentials then once a PC is established okay how do we make this nice and pretty and and clean for the Developers thus Aris was born so what is

arys aris is essentially a broker that works between shith STS and um and the user themselves it broker credentials the user would come in um enter their credentials and it goes out does all the leg work and at the end writes credentials for them to use for their normal day-to-day aw CLI what you get at the end is the Federation of aw CLI um you get reauthorization every hour talk more about that later and um this the same work can be performed the same way they did it before um just a little pre-work ahead of time um but it is is a slight change to how they do business so you do still end up with um a little bit

less whiny devs but skeptical nonetheless in fact actually um when we went to deploy a PC to try it out with some of some of our developers and admins um I got crickets um I had I had to get Paul to chase him and say Hey listen can somebody try this out so um they didn't want to change to status quo so how does it work well um you have an interactive shell login uh the user enters their username password uh actually they only have to enter the password because it'll take it from the shell and um Aris will go to shi's web login enter the credentials come back from shith with a SLE response it will

then take that s response and send it over to stf's service H send an assertion over there SCS checks it makees sure everything's fine that it came from a trusted source which is the shth which is I talked about earlier and then um you get credentials and you profit everybody wins the idea here is that AIS does all the L work so shth doesn't talk directly to STS it's you have this man in this this agent doing the work um but wait a minute what about these 1our credentials what happened to that how did you you fix that well luckily we have Python's request Library uh which is really awesome so in request you um

when you make a request it will maintain the session cookie and all the session State information for you so perfect so before the temporary credentials would expire we actually make another web login request to shith shib is like oh I know you already I authenticated you already like an hour ago so okay fine it gives you back the sound response we do the whole thing all over again and uh we just update the ials wherever either in an environment variable or locally for the developer and um they don't have to do anything so it'll do this all throughout the day now we set a 12h hour time limit because we do want to make sure we reauthenticate them at least

once a day so I think 12 hours is enough maybe in Silicon Valley maybe you should be 15 or 18 hours but I don't yeah so I I'll send this over to PO so I'm basically going to reiterate what he just said but I've got a beautiful Brad Pit as a white Andy developer so uh mechanics of this again are uh you've got shith and Amazon web services have to have a trust established uh so there was a lot of leg work uh setting that up uh there was some documentation it just wasn't super clear uh I gave some feedback on it uh again I this was also done on a a Linux system they have a lot

of documentation on how to do it in a Windows environment which has proved to be a lot easier um but we went the hard route um shet is configured to go and fetch some attributes based on the user and then do some magic to map it to a role so uh shith ends up uh taking input from the user on behalf of Aris uh shth um has that trust established with Amazon already it generates a SLE assertion um based on the attributes of the user and then it passes it back to Aris Aris writes it to your operating system environments environment variables um and then it allows you to uh use the Amazon STS to perform CLI

actions now time for a

demo I did tell you I like cats and unicorns so as you can see I'm uh going to say hello bides um and then I'm going to launch into

errors sors fetches my uh my shell username and and then I enter my password in which is just password by the way and then as you see magic happens and as you see there we've got an access key secret key and session token and we're now able to run commands if we like um and I'm actually comfortable with having that up on the screen because those are invalid after an hour so you can take a snapshot of that if you want they won't work all right back to Aon so some of the issues that we ran into uh with with AWS or at least the STS Service uh documentation was one thing um documentation was lacking uh it

was you could hardly find anything in the other assume um calls that you can make there was a lot of documentation for errors uh examples what you would get but when you get an error with assume R samle nobody had any clue um until like you reach a level three engineer at Amazon so um documentation was really sparse um there was also undocumented like so um items in Botto that were just not documented at all and and also when you do a temporary credential you get three things instead of two uh you get a session token as well and there there was no documentation where what am I supposed to do with this session token so I got

these temporary credentials and I'm trying to use them um but I couldn't use them and they're like oh you actually need that I'm like well can you just update your documentation to say so so you know that would have been nice um but more importantly and this was just this was me for literally like a week or two um bodo's library was hardcoded with something called Anon equals false so Anan is a parameter that basically DCT it's an anonymous connection so um if Anon is false that means you're coming in from inside the AWS infrastructure if Anon is true that means you're coming in from externally and so um assum roll with SEL was not allowed to make calls

externally but I didn't know that um and so sometimes I would be at the office making a call and it would work but then I get home and it doesn't work I'm like okay what's going on so I hop on the VPN and it works Magic and so it was really intermittent and it was hard to figure out and and I was trying to think okay maybe there something wrong with my code I I spend ages trying to change it and I realized no my code is fine uh it's them and so um we uh finally got to a level three engineer level four the guy actually who wrote the stuff and he's like oh yeah

that you're right uh we shouldn't be doing that and so he fixed it in like half a day so that was awesome um kudos to Amazon for fixing it quickly um but the idea is that as STS was not really ready for prime time um there were also issues with um encoding issues with saml where it was supposed to be a64 encoded but it wasn't and so I had to make changes to my code to just redo it which is fine um and just some general other bug so um you're welcome everybody um I did the leg work for making it available to all of you so um you're welcome and uh that's it and we'll send

it over the PA all right all right so I'm going to talk a little bit about what we plan to do next with this um there are uh some potential future problems that we've seen down the pipe and one that we've actually experienced um I'm worried that maybe the shth Earls or if we use a different IDP that maybe that those might change and that would totally screw up this this uh uh Aris um and that would just stem from package updates or just general code updates Etc um there could be some other STS related changes um as you heard and uh reiterate how much pain he had to go through with Amazon I also experienced a lot of that

on a day-to-day basis and I can tell you that they do do a lot of things without telling anybody U so that could be something that would happen that would impact the function of this of erys uh changes in the XML response provided by Amazon it's not very likely um but could also happen U then something that did happen um but I decided to leave it in a potential future problems because it could happen again uh certificate trust configuration between IDP and Amazon not a problem right uh actually part of the configuration process um insists that your IDP go and fetch a publicly available XML file to do some validation on their end so back last month U my it

Department that has been using this product calls me up and it's like your shit's busted I'm like okay so go look at my logs start digging around and then I see my all signs point to this XML file as you can see as I've highlighted that actually expired that day uh Amazon was down for SLE integrated products for six hours because of this there a big [ __ ] up on their end is kind of Basics if you ask me so what we want to do before we decide to uh open source eras uh we're going to go ahead and do some mac and a bonto support it does work on both right now uh it needs a little bit more polish

uh we want to demonize it so that it doesn't have to be a user interactive all the time um improved logging and of course exception howling some other feature improvements I've got a an awesome guy in my it Department that's been willing to step up to the plate and help us build a Mac app which would be very useful uh if we decide to open source it as a user you would input your Amazon account username password and then the IDP Earl and that should be it um and then that would actually let this right to your your o OS environments um as well as maybe a potential local file if you absolutely need it for something like a browser

extension um oath maybe I've actually talked to one of my co-workers about this I haven't really explored it too much uh it may or may not happen uh and then something other than usernames and passwords because those are bad uh I think everything's moving towards aert world or if we can we can get move it to aert or key world where we don't have to deal with passwords anymore open source date ETA is July this year uh hold tight I hope to get it done we're going to make sure that we drill this drill drill out all the bugs so that you guys don't see any of them and now I'd like to open up for Q if

anybody's got any questions for those of us that aren really familiar with sh [Music] sh correct but it can be done for other sorry so the question was shth is using is like the crucial element to this working uh that's correct it's the IDP we chose to use for this um I would imagine that you can do this for other things like an OCTA or One login uh the other or Ping Identity the caveat there is all of those uh present an MFA or a two-factor login so we would have to modify the erors product actually ingest that and then pass it over to them as an IDP so this product is using shth in a

non tofa mode behind a VPN and firewall so that would be something that we would have to add on on top of that again I would rather rely on having um other tof uh than adding another one to the mix but for now yes to answer your question this is tied to shth anybody you know what I've heard about this it's a good question oh sorry he asked if I played with hologram so hologram is actually something I was I heard at a devops Meetup at my my office actually um if you want to put me in touch I'd actually love to talk to them I actually did mean to follow up with them yeah anybody

all right oh one

more so the question was um if we had to serve as account that we needed an access ID and secret key um would we try to move it over to this the answer is yes I would like to but of course this needs a lot of Polish and we need to make sure that it's not going to become an availability impacting you know Cog in the the wheel there um but I would like to see it used for all of our service

accounts cral uh it's a good question uh there's a little backstory there too uh the question was um why we decided to go with samel to do this instead of any of the other options uh to be honest I didn't really see there was a whole lot of other options maybe aan can answer this one yeah it's more so because um because we're using our Lop our Lup is a source of Truth so that was the issue there um so s was the most compatible at the time um you can use ooth um if you want to um but to get that to work with shth I think would take a lot more like work shth kind of did s out of the box

some that was part of the decision something's Chang any more questions okay well thanks [Applause] everybody