Josh Stella - A Live Simulation of Advanced Cloud Misconfiguration Attacks

screen i am excellent uh so i'm josh stella i am the uh ceo and co-founder of fugue i have also been the cto today i'm going to do a wide simulation of an advanced cloud misconfiguration attack uh so what we're going to do is a few slides uh if you're like me you don't like slides very much and uh and then we're gonna jump straight into uh actually uh i'm gonna breach my own uh cloud infrastructure uh using the same techniques that um i learned from a doj filing in a real world cloud breach a couple of years ago so we're going to go through what hackers really do um at least have done in in some cases all right

well i just told you what we're gonna do here so we can skip this it says q a here i'm not i'm not used to doing things on go to webinar uh if the organizers uh want to help in any way i love it when um i get interrupted and questions asked and you know so however you folks want to handle it not sure how to how to do that in the in go to meeting but hopefully we can get some q a going and you don't have to wait till the end um you know if you uh if you have a question about something i'm doing just let me know okay so cloud misconfiguration

is by far the number one risk in terms of uh what hackers are going to exploit in your cloud environments um you know i've been in this business for for a very long time and the old attack surface which uh still exists but with much less uh completeness was really the network and the operating system the application in the cloud uh those things do get exploited but only as typically only as means to an end the end being getting access to the control plane apis of the cloud to do things like steel data out of s3 today we're going to steal some data out of s3 and we're not going to do that via traditional kinds of hacks we're going

to do that via a cloud misconfiguration hacks exploiting misconfigured resources and many of those misconfigurations are not seen as compliance violations they're not seen as vulnerabilities so for example um having s3 list permissions on a running ec2 instance is insanely dangerous uh but most people think that's fine so it's i'll show you why later it's so dangerous but it truly is so so they're not seen as vulnerabilities and that makes them uh really easy to miss and really hard to find many of these misconfigurations aren't things you can catch you know we at fugue do uh infrastructure as code scanning for uh misconfigurations and and that's fine and well you want to do that it's important to do that

but many of the things that the hackers are exploiting can only be detected in the full context of the running environment and can't be caught in small isolated you know templates and so on so you have to be aware and your security tooling has to be aware of the relationships between these things you know what iam permissions are associated with a compute instance and uh you know what does that compute instance have access to as a public ip address etc so um this combination means that these things are really common in uh in cloud environments says enterprise cloud environments but it's really all cloud environments at scale every single high-profile breach i'm aware of every

cloud breach i've ever researched is is largely due if not completely due to getting configuration wrong all right so every year uh we if you do a big survey we're just this is this is from last year we're just finishing up our one from this year where we get uh 300 uh organizations that are operating at scale on the cloud uh to answer some questions and these are uh these are not of course uh uh necessarily a few customers or uh these these are just organizations operating at scale on cloud and so these are some questions we asked last year 84 of those respondents were concerned they had been hacked and didn't know it um i'm glad the number is that high it

should be should be everyone should have this concern um a lot of times breaches never get noticed and part of the reason for that is uh very often the data exfiltration is happening not over customer viewable ip address ranges so you can't even see the stuff on the network it is common to read in the news about those hacks that later were discovered i suspect that that means there are far more that have never been discovered by the people who were in fact hacked 92 percent are concerned they're vulnerable to a cloud breach i mean the only reason to not be the only reason to be in the eight percent that aren't is if you're not

doing anything in the cloud i've been living this for over a decade now purely focused on cloud security uh the hackers are clever they are doing things that um that evolve over time uh you know hacker used to mean it started out the the word meant a clever programmer um now it means somebody who breaks in and steals stuff and so on um but they're still clever they're really clever and i'll show you some of that later so you should be concerned that you're vulnerable to a cloud breach all right 76 said 76 percent said that misconfiguration risk will increase or stay the same it will either you know stay the same or get bigger this year 76 percent

uh it is axiomatically true that misconfiguration risk will increase this year and and the reason for this is every time a feature is added to any cloud provider and i'm not talking about necessarily one particular customer you know one particular company's implementation of cloud i'm talking about the attack surface of the cloud grows every time features are added and i'm actually going to exploit a feature that was added years into ec2 this ec2 services uh life cycle uh two to steals from s3 so so this is a growing problem and it will always be a growing problem this is just the new reality um so uh dave olympic i'm saying here i'm seeing a lot of cloud misconfiguration

errors in the real world and it's scaring the hell out of you yes the blast radius of these things is often quite ugly too all right um you know exploit strategy has changed this is with cloud it's certainly alongside cloud it's hard to determine causality the coincidence here um i suspect there is some causality because the nature of how cloud works uh versus data centers um but largely what we've seen is a shift in behavior among hackers where in kind of the old days um hacking was much more like a hollywood movie version of hacking you know you would target an organization and then try to find ways in you know that might be purely technical ways that

might be special engineering or phishing attacks or whatever but you would target uh an organization and then find or create vulnerabilities um examples of this are you know the north korean hack into sony motion pictures they did not like a movie that they had made and they went after sony and uh they were very very successful another one would be uh stuxnet well i think it's broadly accepted now that a combination of the nsa and israelis uh uh introduced a malware to uh uranium centrifuges in the iranian nuclear program to slow that program down um that still happens okay that kind of targeting an intentional hack does happen far more often though what we're seeing

now is the automation of finding vulnerabilities so so instead of for example we know from the social media postings of the capital one hacker uh that that she didn't go after capital one she just kind of woke up one morning and her automation had been running all night looking for vulnerabilities and saw capital one in the list okay so so this automation uh allows hackers to kind of develop uh shopping lists if you will of organizations that they can go hack uh without you know they can find the vulnerability itself if a friend of mine or an associate of mine i would rather who studies pretty much this topic exclusively told me that the average amount of time that he's

seeing between an ip address being on the public internet for obviously a dns record and it being scanned for vulnerabilities is under seven minutes so here john bredon is saying uh you know uh they'll discover uh misconfigured cloud assets within hours of their deployment as bad as that sounds it's worse the reality is worse it's minutes and so as you're thinking about your uh your cloud security strategy it a lot of it has to be preventative and to be because as you'll see when um when i uh when i do this hack it's really easy to get a whole lot of data if you've just done a few things that seem innocent uh wrong so you need to be preventative

uh and you need to try to limit blast radius and we're gonna talk more about that as i go through this hack okay um a little off my game because i have all these technical problems getting in but i think we're ready to go let me see uh what have i got in my [Music]

all right we're going to get a aws window open and i'm also going to use the diagramming part of my of our product feud to show you uh what i'm trying to do here as a hacker okay so we make security software here too and i'm going to use our diagramming to show you what i'm going to do come on load up

all right what i want is this guy

okay so what i'm what i'm going to do here um and this this diagramming is just a part of our product what i'm going to do here is i am a hacker who wants to steal data out of s3 okay and we know from uh doj filings and other reports uh in a pretty detailed way how this particular breach was done uh what was done so just to explain this diagram a little bit over here on the left is the internet okay and then we're going through a gateway internet gateway into a vpc network here is a ec2 instance of virtual machine running and that ec2 instance is sitting in this vpc so the way this real world breach

happened is there was a misconfigured application running on the ec2 instance and that gave the hacker a foot in the door now very typically when we see this at fugue it is uh forgotten infrastructure very often uh people will have orphaned resources in the cloud stuff they've forgotten about and it accumulates cves because it's not being patched and that gives the hacker a toehold so you do need to be concerned about that because it gives hackers a way to get into your account and get that toehold but really honestly uh nobody is going after typically the contents of an ec2 instance they're going after s3 they're going after an rds instance or particularly an rds snapshot is a really

popular way to steal data because you only need s3 permissions for that not rds permissions um little known facts so your again f3 permissions are dangerous in your running environment so what i'm going to simulate here is i'm out here on the internet and i'm going to hack into this ec2 instance now i'm not actually going to hack into the ec2 instance i'm just going to shell into it and the reason for that is i aws are having work there i can tell you are very good at monitoring activity in their networks and if they were to see me as for example trying to use an exploit against a cve and they noticed it they would shut me

out and then our session here would be over so i'm just going to shell into it and then i'm going to try to get to data so my my job here my goal here right is to um exfiltrate data out of s3 all right i'll just go to a shuttle here i'm going to make this font a little bigger that looks a little small

all right so i'm just here this is my local machine my laptop and i'm going to um as i said i'm just going to ssh in to an ec2 instance because i i don't want to get kicked out of aws while i'm doing this demo or for this session okay so i've gotten here to an amazon linux ii uh ami based um uh instance now i i pers i i purposefully leave this out of maintenance to show you what's going to happen so this thing's been running i don't know for maybe probably two years now um and and very often we'll see orphaned infrastructure too that's been around longer than two years uh well here you

can see 72 packages are needed updating for security reasons in a year so that's two years so that's you know three a month something like this out of 152 updates available well if you have anything running in the cloud that you have forgotten about this is probably true of it no matter what and by this i mean operating systems we at fugue choose to use almost purely lambda functions for our application because uh we don't have secure operating systems because we don't run operating systems amazon does okay so i've gotten into this box right i've gotten into uh let's see this guy right here uh this where it says legit app instance i can go and uh infuse i can drill into

all the details and um i can see uh exactly the the ami id that is the machine i'm in okay um so i want to steal data out of s3 well it's amazon linux it's perfectly good linux i know as an experienced cloud hacker that amazon linux comes with the aws command line tools which you get to by typing aws if you're not used to spending time at a terminal don't get scared off okay i'm going to explain every single command in a lot of detail all right so the first thing i'm going to try to do is go take a look at s3 right okay so i got denied access that's good right it's bad for the

hacker uh but it's it's good that the uh maintainers of the system uh were thoughtful enough to to deny that access all right so what that means is when you are in this this is where people a lot of times get confused about cloud in terms of where the networks are and and what what surfaces the hackers are really exploiting because it's typically not the tcp network it's typically the identity network and what i mean by that is this ec2 instance is trying to go talk to s3 right we've got s3 buckets over here like maybe i'd want whatever's in this important records bucket that looks that looks tasty but i can't even see that this ec2

instance can't even see that it can't see s3 it can't even do a listing of s3 and i've drum ib and will continue to is s3 listing is very dangerous to have running in your production environment okay so why can't this ec2 instance see s3 the reason is it it is operating under an iam instance profile so when you think about identity and access management in the cloud you're not really just thinking about the identity of users the the actual components of the cloud use identity to form uh essentially networks can ec2 talk to s3 in this case no all right i need to open my text editor real quick here which one is this

and it's um [Music] i had to switch machines to do this uh presentation so i apologize for uh for uh stuff that would have been set up um right in front of you okay i've got some commands here i want to run so um i want to change uh pace a little bit here so okay so as a hacker i can't see s3 you might think well you're it's game over no it's not i'll show you how we can we can do some other stuff but uh in the news in in for a couple of years now has has often been uh the uh oh i don't have this one set up to paste i don't even have this machine set

up with a proper terminal so okay so uh the metadata service

go away so the metadata service the metadata service is a feature of aws that used properly is excellent uh used improperly is dangerous and it hits the news a lot of people say oh it's because of the metadata service that this hack that hack happened no it's no it's really not it's it's because of misuse of it so what the metadata service is is when you have your uh your ec2 instance running here this is a virtual machine okay and a virtual machine runs underneath a hypervisor right we all we all know this the metadata service is running on the hypervisor and this special ip address one six nine two five four one six nine two five

four is always the hypervisor okay so from any ec2 instance you call that and you'll get that ec2 instances hypervisor um and then there's a service running on that where you can see here i'm looking for latest metadata curl is just a command if you're not used to seeing this kind of thing it's like typing this url into a web browser's location page our location field uh just here we're doing it at the command line and then we're gonna we're gonna print out what we get back okay so i'll hit enter there and here you can see i can list stuff uh you know related to this instance here's iam so if i do the same

thing and i keep drilling in i can see oh look at this security credentials that looks interesting and here i see a thing called maintenance role all right so what i've now done as a hacker is i know what iam role i'm running under so if i go back over to my diagram you see here maintenance role well the hacker did not have access to a nice visualizer they're having to go do discovery and and this is another point that i really want to drill home um 90 of hacking at least is doing discovery it's learning and what you want to do is prevent hackers from being able to learn stuff easily you you want to deny them information

meanwhile you want yourself and your organization to have as perfect information as possible okay so here we've discovered the maintenance role from the ec2 instance well there's some stuff i can do with that oh right before we get into come on computer stuff okay um let's go look inside the security credentials for that maintenance role okay let's paste in this terminal uh before we get too far deeper into this uh we did have a question come in of how would the hacker have gotten access to the ec2 yeah so in this case they uh they would have been scanning any uh things with public ip addresses and in this in this real world hack it

was a misconfigured web application firewall it was a misconfigured whack but if you remember back to that uh earlier part this when we were in the slides right when we were in the slides and i said uh this um uh you know hack is is is based on getting a toehold through a cve and then launching from there and that's done with automation so the ec2 instance was noticed by the automated scripting that the hacker was using to uh to uh it's weird the hacker was using to just scan the whole internet okay and it got noticed because it had a known vulnerability this is really giving me a hard time let me it will not let me paste anything let

me just open up a terminal here

okay so i'm just going to use the terminal in code here instead of the other one all right so let's take a look at what happens when i actually dig into those security credentials all right so here we're curling we're getting the uh the metadata service all the way in security credentials for the maintenance role like what's in there okay let's make this bigger well this looks really scary right i've got an access key id i've got a secret access key and a token so didn't i just get from the metadata service uh kind of the veritable keys to the kingdom no i didn't i just got the credentials that aren't letting me see s3 okay so uh

who cares because and this is what i meant by properly used the metadata service is actually an awesome feature it does automated key rotation um and and people who try to roll their own key rotation typically end up not doing it well and that means keys are long-lived and that's super dangerous too so with with the metadata service and there's a more secure the metadata service v2 i think they call it uh has some more security built in than v1 but even v1 is fine if you kind of know how to use the tool okay um a lot of this game of cloud security and cloud breaching is being uh really up on your identity and access management uh

policies and making sure that they are not overly permissive and i if we're if there's time i will show you this window i will show you why you probably have overly permissive overly privileged iam policies okay but this one is this one was done the right way i can't see s3 all right but i'm a hacker and i know what i'm doing and so i know i can't talk to s3 but i know that i'm running under a iam role and that's where i'm getting my permissions so i'm going to go searching around once again discovery searching around to see if i can have access to um some iam policies okay so i want what i'm doing here is i'm shopping for

better permissions so here i'm using the aws command line tool again this time i'm not targeting s3 i'm targeting iam and i'm asking for a list of policies okay and then i'm going to do a graph which is a search for things that have a policy name that contains s3 okay so now i'm going and interrogating like the world of policies that i uh might be able to use all right well this s3 read write uh looks pretty cool for my use case that's what i want right i want to read and write s3 although honestly i don't even need write not at all i just need read but actually even more than read what i need is list

and i'll come back to why that's true uh you're probably sick of me saying that i'm gonna keep saying it because it's extremely important okay so here i'm running another command um here i'm asking i'm using the aws command line to uh talk to ec2 i'm not talking to iam a lot of people think the stuff i'm about to do is in iam but it's not it's in ec2 this is kind of a verbose one describe iam instance profile associations in ec2 describe commands are what are typically like list commands and other things but describe means uh what it says means describe this so here i'm going to get a list of i am instance profile associations all right

what is an iam instance profile association it is the connecting configuration that makes this ec2 instance have the maintenance role so that the the association is associating the maintenance role to the ec2 instance okay and that's going to have its own identifier so i just want to list them all let's just describe them all okay so here i've got some now you'll notice that i uh there there's more than one you can see right here this instance id and just to to prove to my cunning audience that i am telling the truth that is the same id this is really that compute instance i'm i'm operating on um right here and you can see that it has

maintenance role okay so that that makes sense i've gotten back this description what i'm doing here as a hacker is i'm trying to get this id this is what i want okay so i'm going to copy that guy i normally have my text editor on my other screen but i don't have another screen today because um oh actually that's the wrong id i want the association id this guy all right so we're going to grab that and we're going to drop it into this big long command that i'm going to walk through every bit of so you understand it before i run it note that so far all i have done is discovery i haven't touched anything

i'm just looking around all right so this is the first time i'm going to try to actually make a change in this environment to do anything detectable in other words all right so um or detectable using traditional methods so we're aws command line again and here we're calling ec2 a lot of people think the command i'm about to run is part of iam permissions it's not it's in ec2 permissions and and therefore this is actually fairly commonly out there in the world uh running on pc2 instances in uh you know dev stage or prod environments for hackers to go fine and that's really really dangerous a feud will of course throw alarms and screen if you have this configuration out there

but i think we're the only ones who've noticed this um so what i want to do here is with ec2 i want to replace the ian instance profile association that's a mouthful but it is replacing that connector between maintenance role and my ec2 instance so i have to give it the association id i want to replace this association id and i want to replace it with an iam instance profile whose name is s3 read write okay okay that looks like victory here's my same right that's the same instance id i'm on the same machine and now it says it has the read write permissions now i've given this talk before and inevitably somebody will say

to me well there's why in the world with that permission uh be out there running on an ec2 instance and here i'm going to show you it is very easy to uh say and aws says this and a lot of folks when i'm giving this talk will kind of allude to this well you just have to get your iam permissions right okay that's non-trivial and actually the things like the access analyzer don't really help you with knowing if the uh i am access analyzer is um a feature of aws it will tell you if you're over permissioned uh in that you're you're using things that are uh uh not needed but it it won't always

trigger um you know all of the uh things that are actually dangerous in fact i would argue a trick it doesn't notice most of them all right so why do you end up with overly privileged overly permissive iam policies so here i'm creating an iam policy to talk to ec2 that's when if you remember i typed aws ec2 so these are lists of actions and one of those actions is the what what was it um replace i am instance profile association okay so the way iam permissions work allowances work really is i have to turn on access to the action all right so why do we end up with this action that obviously we don't

really want able to be run out there uh well let's see how many right actions there are in the ec2 service 289 uh let's look at lists 124. read is going to be shorter 23 tag so we're talking about you know 400 plus of these permissions and by the way these often have to perform any reasonable business function you often need to have a number of these things that may not be obvious to you uh turned on and across services in many cases so what very commonly happens is people get overly permissive and what they'll do is they'll say oh god i'll just turn on you know how bad can the list be how bad

can reed be and they'll turn on a lot of that or in this case we've got some right permissions we don't need but um it is uh not a a glass house that i would throw stones from to say uh oh just do i am right it's hard and you probably have you know but there's one of my favorite reviews of our product on g2 uh the review it's a very positive review but the reviewer starts out that fugue is going to hurt your feelings uh because it's going to tell you things that you're doing wrong and that are dangerous and and that's that's we all we all live in that situation we run fugue on food and it tells us

when we screw up okay so now i've changed that association so let's take a look at s3 right we want aws s3 ls is list it's the unix command linux command starnix command for list and that's also in the aws cli so let's take a look ah now the last time you remember what we got we got access denied this time we're getting an enumeration of our s3 buckets right so i'll show you these in the diagram here before when we were in this instance we couldn't see this stuff over here but now i get a complete list of it i just say give me a list and here's a list of all the names

so what i'm going to do next if we know from the doj filing is uh exactly what the hacker did in this case it's really great when there is a detailed dij filing so how many commands have i done i've done maybe a dozen or so commands only one of them was not discovery all the rest are discovered so you really want to blind hackers if you want to prevent them okay uh we know that the hacker used a an aws cli command called s3 sync because it's in the dij filing sync is not part of the f3 api it is only a utility feature that is baked into the command line interface so we know that the hacker in this case

was using the aws command line interface to perform a sync and she did something like this i want to sync everything in important records to all your base all your base is my um is my my hacker bucket okay so an s3 bucket if you're not super familiar with this this has files in it this is a an object store like a giant web server a global web server so i'm going to suck everything out of important records and synchronize it into all your base now in this case i've got both of those in my one aws account in the real world this would be traversing to probably another aws account controlled by the hacker

all right i just stole all your data and none of it traverse your network boundaries that you can monitor um i just pulled all this data from your s3 bucket and put it in mine now we know in this breach that 700 f3 buckets were uh reached in this exact way using a sync command okay so let's think about discovery here uh first of all sync itself when you see here where um it's finding all the files in this case these are some photos i took in iceland not you know important date well important to me but not uh not stuff i mind is on the internet it has to use list permissions to get

the list of files so you actually can't use sync if list is turned off um this is a tiny bucket with you know a handful of photographs in it but in the real world these breaches usually happen to very very large quantities of data in these buckets and buckets can be absolutely huge and the hacker stole data from 700 of them well if if i can't list i don't know what files to get even if i can get right and i can't list i have to know the names of the objects so by turning lists on you've given them a map to and what i mean by turning it on is having it out in the production environment in a

running instance okay um that is the i would argue that the principal misconfiguration in this there are really three or four total uh the worst one is definitely the iam role assumption uh this should never never be allowed for anything running in production under any circumstances it's just a absolute black hole of risk um but the next bad the next worst thing was the s3 list permissions the the misconfigured laugh with the cve who cares because if i mean you want to prevent that but but that is a minor minor piece of this hack um patching or operating systems is important you should do it but had i properly configured my iam ec2 and s3 resources

um it wouldn't have been a problem okay and so that's what i mean about limiting blast radius blast radius is defined by what the hacker i mean at its at its outside right the limit of blast radius is what the hacker can find so if they can't see things they can't take things generally speaking so what you want is anywhere in your infrastructure you want a limited visibility to the rest of what's going on uh for for the hackers but you yourself need to have full visibility into the entire environment and track it at all times so you don't end up with things like out-of-date machines or i can't tell you how many times you see

or people will you know will run for you and in about 10 minutes we'll we'll show uh you can you can use this for free by the way at a developer scale uh we'll show you know this kind of google maps for cloud diagram we do and somebody will say what is that network and in one case i had somebody at a big enterprise software company say oh an intern built that last year and i had compute instances running in it well there's there's hackers in that network right because they're sitting there uh exposed okay i think we're coming out of time here uh so any uh q a for me yeah so why don't

we get uh probably the closest one um if they couldn't use ls to get a listing of resources could they use sync star dot star or something similar to that uh actually no because think in order to um no in order to to do the individual get commands that it's going to do right it needs the file names and so sync is just a utility running on the local machine that's using the the aws api uh behind the scenes so no you can't you can't get around it that much okay yeah but by the way the the way to not use lists is to store the keys you need you know the object names in some other

resource like a database so that your application isn't having to do lists on f3 it knows what files it needs right one of the others not a very specific question but um someone wanted to know your thoughts on the recent ubiquity breach given their cloud service provider for you know their devices you know i haven't read much on that um i don't have uh i'm too ignorant of it to have an opinion uh but now i will uh i will take a look oh uh is this um trying to remember which one this is yeah i i don't know i mean i i saw it happen i don't know enough about it to have an

opinion there's all kinds of reasons that could happen however if you're not using multi-factor authentication everywhere just do that um leaked passwords shouldn't be cause for alarm leaked api keys should so but leak passwords if you have mfa i mean you can uh and the ubiquity hack was was passwords or that was some of the league data um mfa gives you a nice a nice buffer i mean doing what i do my social media accounts somebody attempts to hack them like daily and um i have multi-factor authentication turned on so my phone says hey somebody's trying to do this and they don't have my phone so they can't do it now they could there are ways if they really

wanted to target me they could but it is awfully secure compared to just a password awesome uh i'm not seeing any other questions so that maybe cool well uh i had fun uh thanks for bearing with me as i worked out my tech issues at the beginning and uh i hope you all enjoyed this no worries i know i learned a lot thank you very much for taking your time you bet have a good one bye