Security Lessons Learnt From The Cloud Frontline - Ben Fletcher

primarily for their data and that doesn't change and that's really important to understand that you need to understand as Security Professionals what your responsibilities are when you uh when you take a bit of the cloud and use it I wish it was a flatline it's not you can buy you can get infrastructure of service which is the most basic thing so uh a compute the storage the hardware the electricity that's provided is all provided by the cloud provider but as you buy more containers of service platforms of service softwares of service this responsibility changes and again you need to understand it it's in the documentation for each of the services it's really important that you

understand this and understand what you're responsible for in terms of that like everything there's good and bad you know there's pros and cons and it's your business use case so you know while you may want full responsibility as an example when log 4J came out we had 14,000 people all Circle onto that problem you know that's quite a good thing to Outsource but again it's your business cases it's what you want a service provider to be doing what you trust them to do etc those are the things that you need to consider so I asked you to vote uh I'll let you vote once more if you've if you want um okay so so the QR

code there so what we've been doing is we have essentially created a the equivalent of the MIT framework within within AWS and we've been categorizing all the instances that we deal with one of the major things that we look at is the initial access method most often used by threat actors so you got cross permissions vulnerable apps uh web apps uh Brute Force locked on Le uh leaked access Keys credentials open bucket and dos let's see how people voted okay at number one is vable web apps at 42% loss and Le Keys 38% open buckets cross account and uh DS was a bit of a trick question it's not an initial access so it's annoyance but

it's not actually how you get inside the cloud let's find out what the answers are

so leaked access keys and credentials and a whopping 2/3 okay so this is uh the threat actor not really trying that hard you know and and a lot of this so there's two ways of getting really there is console access and uh access Keys when it's console access can you guess how many times I have seen MFA enabled and that has been bypassed zero okay um access Keys incredibly powerful but not in the wrong hands you know really consider do you need those access keys and there's plenty ways of obse secting them okay too often they're in GitHub and we tell tell people they're in GitHub we might be emailing the wrong person because a threat actor might have

changed the contact email um yes and also when the threat okay this is a thing that brings a tear to my eye every day okay a third of those access credentials are root 20% of everything we're dealing with is the customer has lost rout access that's very very bad um um you know please get off rout you can close your account with root it's gone forever goodbye there are active all the activities on Route don't need to be done by Route so your first activity is get off rout hide away the key don't create Keys hide away your password MFA enabled guess what the threat actor does if they get root enable MFA because they

believe in strong security [Music] okay the others are the uh are the the public facing so so the services and unsurprising ec2 is one of the largest at 133% we are seeing a spiking API gateways and again this is just the validation of the input so the person confirming who they are you know or do you have an open API Gateway or are you confirming the transaction is correct what are the uh threat actor patterns that we're seeing uh resource hijacking that's you know great get the cloud to do it for you Ransom events now in terms of Ransom events this isn't your typical encryption pay the money decryption a lot of the time this is just deletion

and ex and basically extortion so I've deleted it I kept this I'm going to release it give me the money we it's difficult we'll talk about the questions about has the data been exfiltrated it's difficult to tell but we have some indicators but most of the time this is just trying to extort the money they have no intention of giving you the data back the other one we've seen a little bit of a rise in is sort of scorched Earth policy the disgruntled employee or and this maybe it's just my anecdotal uh experience but developers that people have got may be quite cheap and haven't been paid and and so on and you know so

be very careful who you employ but all of these you know could have been could have been the blast radius could have been minimized by really considering what permissions everybody has usual stuff we know this but as identity is the boundary in the cloud it is really important understanding what permissions you're giving people and also I showed you earlier what permissions you're giving a service and let's get on to the permissions I gave to a service so my ec2 um unfortunately in my rush I also put a vulnerable web app on it or I developed a vulnerable web app on it as well in addition this ec2 is running something called instance metadata ver service

version one which uh you'll see in a bit uh meta the metadata Services is AB vital to the ec2 in terms of you uh troubleshooting being able to do different things it tells you about the ec2 information you may need in terms of programmatic access all of these things I've talked about in terms of how we're going to um execute this is cannot be attributed to one misconfiguration failure and I really like the analogy the Swiss cheese analogy in um in industrial access accidents where you know each slice of cheese is a security measure but they all have holes in them if all of those holes line up then you have these accidents and this is exactly saying you

look at some of the high-profile security incidents I mean the Microsoft one is absolutely classic I mean all the things that had to go wrong for that to happen and again this is exactly the same so there were tiny tweaks here that could have could have prevented this but you know this is a perfect perfect storm of all of these misconfigurations lining up so a crafted HTTP request uh you get the you get which uses so you get onto the imds Fe version one you steal the security token and STS security token service is a sort of Workhorse within within AWS which is providing roles and permissions as you go along so making sure that something

has permission for a certain time and so on and so now the bad actor has admin because we gave the ec2 admin we didn't have to give it ec2 but hey why not give them ec2 live change over how's that oh no better can you hear me at the back thank you okay so what does bad actor do I don't know maybe uh deploy a cloud formation temp plate spin up lots of metal inst ec2 instances and crypto mine and look at all that money oops that's went really quick anyway a lot of money went to the bad actor there now I know what you're thinking oh bit complicated how would I possibly write a cloud

information template do not fear you can go on GitHub um I'm watching you don't um you can go on GitHub there are forums there are service desk there's help towards this okay so what I'm trying to say is this threat is real there's Industries behind this just looking for those exposed access Keys you know credentials being fished and so on um and you read these GitHub you hear you read the introductions these GI it's very it's very um not very believable their reason for writing these when you consider it's a 57 to1 ratio in terms of compute power cost to actually return not really much Point running this yourself okay shall we see this for real

so here we and by the way pay attention because I may or may not have set up a uh website at the end where you can where you can practice this okay so uh um and I'll explain a bit more but if you want to oh sorry let me go back okay so I have my website it's amazing what you do is you upload your image there you actually upload the URL of that image and what you get in return is this strange uh URL but you see that demo. PHP question mark so I wonder if you put equal https I wonder what you do what you could do if you could actually craft craft this slightly

different so this is what I put at the end of that does anybody recognize that few smiling faces that is that is the metadata server so if you're inside your your ec2 that is that is how you call the metadata service on your ec2 I apologize that you might not be able to see that but what I'm doing there is curling it and then what you're seeing is all the list of metadata stuff there pretty cool well I think it's cool okay so now what I want to do sorry that's the same one isn't it y apologies there we go okay so I'm inside the metadata service then I go to I am and I look at security credentials what

this does is this tells me the role that is attached the name of the role that is attached to the ec2 I run that and it's called webdev o that's exciting what does webdev tell me so I can now go inside webdev and here you go so this is the access key the security access key and the token and this is called ssrf this is serverside forry request so the ec2 believes this request is coming from inside the ec2 it's not it's coming from the web app outside of that so so I now have information there to now assume this role hopefully it's admin all I do is export those I also had to find the region if in doubt Us

East one but uh it was you can see that within

this and I'm not foolish so this first four confirms you get call or identity so it's basically saying you know who am I um and you'll see there the Arn there you can see it says webdev and then i-0 so if you know if you know AWS you'll know that's an instance um an instance ID which kind of weird you know you can see console login from that you're going how is an ec2 which has no thumbs able to log into console so I've tried then to create an user this one didn't have admin I can't I'm not allowed to do that um but it's demonstration there if you look up AWS cert workshops there's a

whole workshop on this where you can run this yourself and and and practice it Okay so it's just a demonstration of all of those things that line up that facilitated this oops

apologies and at all of these stages you know so you know crafted HTTP request that could have been prevented by having you know a vulnerability scanner against your web app uh using imds V1 they defaulting to 2 now um you will get lots of warnings telling you that your old your ec2s are using imds V one and you may want to upgrade them to two um security token returned a very privileged security token and then you know obviously the the deployment of the cloud formation template the Swiss Cheese model all these things that could have gone wrong and what I'm trying to I'm not trying to show you how to hack here I'm trying to

show you um and I had to this had to go through PR so I definitely not um that for you to understand this better and understand these how these exploitations work you can better approach those problems within your Cloud environment and you know I reiterate this isn't unique to AWS

it really is at the moment from what we're seeing eat your vegetable security like basic stuff if you um if you do the basics right you won't be speaking to our team because you know you're thinking about that the lwh hanging fruit is being picked off at the moment don't get me wrong we're seeing more and more sophisticated threat actors but um you know just please put yourself in that security posture um as high as you can so inaccurate aw's account information once you spin up loads of ec2s and so on and once your keys are on GitHub Etc we send you emails to tell you this and we will put you into fraud status and so on so it's really useful

if we're emailing you know your team and not just one person who's left the company and so on you know basic stuff like that unintended disclosure of key is is Paramount have you really looked at all of your users I can tell you great cases I can't because I'm not allowed to talk about them where somebody has had keys stolen and they were able to do nothing you know because and and give those security people a pay rise they looked at it and they restricted the permissions of that user and again doesn't users aren't just humans they are things without thumbs um you know you could have your your guard Duty whatever whatever um detect detecting service you have

flashing red lights but if it's sat in the corner there you know what are you doing about it you can roleplay these you can practice these it's a moving landscape okay we are seeing latest Technologies come out and and the threat actors looking how to exploit them so unfortunately it's a continuous learning game and you know you need to think about you know I rushed my dragon cat believe it or not okay you really need to think take that step back and be that voice you know all this you know but these developers are be you know the developers being pushed hard to deliver be that voice of reason saying that's great however have you thought about

this have you thought about that okay I know what you're here you want to hear the stories you don't want to hear all that okay I'll see what I can do let's first of all talk about how do we approach a security investigation so that if you need to you're going to um do that so let's take the example of uh crypto M and with your account you get 90 days of free event management logs they are in each region okay and be very careful global global services are in Us East one and the rest may be in the services you need to go down to each of those regions and look at those cloud trail

logs okay but you'll be looking for run instances what that will then tell you is possibly the access key the username and the IP address so what you're looking for straight away is what is the indicator of compromise we do have people saying can you show if we've been hacked you know we had we had we were looking through 80 million logs a few weeks ago can't do it unless you know how do we know one cloud trail um call is correct and one isn't we need to use a context and again for yourselves you understand your environment what is the user context did somebody mean to run that instance or is that absolutely you didn't mean to run

that instance great we know that's in we know that's an indicator of compromise then we can work backwards from there and and continue so we've got this run instance we can we can uh relate it to the access Keys uses names the IP addresses and we can see if that user with the user was created or the access key was created and if that user was created well that probably means the user that created it is actually the threat actor and we have this we have multiple chains of access Keys being created and so on and you're working them all back to that single point of entry and hopefully it's within the last 90 days unless you've enabled your own

logs so again as a company is 90 days sufficient or do you want to have your own log and do you want to put them all into one location and we will circle around this until we find that point of entry and then what we have is the initial access which tells us uh which access key which user it is and and so on we can rotate those and uh then we get a lot a lot of times we're asked how was that access key stolen don't know it's like asking the police my house of burgled they use a key where did I drop my key you know it's an impossible question all we can

tell you is when it was first used within the AWS environment got to check all regions um you know what the first thing I'll ask do you have any logs enabled and you hear me slightly cry when I hear no please to enable logs is it important for you to enable logs you know and again retention and protection appropriately one of the other questions we always get asked and it's particularly relevant in the Amir region is was my data stolen now as I said you get 90 days event logs they only tell you at a bucket Bucket Level what's gone on created bucket delete bucket if you want to know if objects have been uh put or get and

normally it's get then you need to have enabled uh S3 cloud trail logs and excuse me and you know if you're holding pii those are the sort of questions you need what's your data what's your critical data and is it uh held there one other thing I'd like to emphasize here how many of you think if your S3 bucket is deleted AWS can recover it we can't okay when it's gone it's gone and a lot of people think okay I moved my critical data into the cloud that's great we do we have a resilience plan we have it replicated in six different regions but that's the latest copy so if it's if there's nothing in there we've got six

copies of it nothing in there okay you again you need to look is your data in the cloud the single data and have you chosen a backup policy you know what are the protections against that there is a lot of assumption that you know and and there's very good legal reasons why that's the case in terms of you know you control that data it may be a legal requirement that you have to get that rid of that data at that time so uh that's sometimes not understood okay so here's an example of uh it's a while back now so uh the overly successful shutout I called it so an exposed access key and it had limited

permissions but they were able to see which used users were admin and they were also able to reset their passwords fortunately two of the users had MFA enabled unfortunately the third user didn't have MFA enabled so they're able to reset the password set the password and they then uh privileged escalated themselves to user number three and were admin and started resource hijacking they then did something which could have been quite good but it wasn't great they applied which is an AWS managed policy deny all but they applied it to everything including themselves so not only did it stop crypto Mining and notify the customer because they couldn't log in that somebody was in there they locked themselves out good

news story here of course they had root they could recover all of this and it was a fantastic lessons identified and and you know I'm calling the is a multicloud storm and a lot of companies now and and rightfully so are going for the multicloud solution um which is great but obviously the crown jewels are now the identity provider and that becomes a real challenge for us so we can only see in AWS so that means somebody could be coming in at all sorts of angles within within the within the AWS and there's no way of determining if they're valid or they're not unless we see some obvious things like create users delete you know

run instances and so on but even if we find that they may have Federated into another user so it becomes really complicated and really what you need to do is be working with the identity provider to provide those logs then you'll be able to see where they're coming in to uh the various Cloud

providers so this this was a multicloud example where a user had Federated in and they were trying to apply this policy now it's actually quite you know to me it's quite quite a detailed policy it's basically allowing you to delete Trails stop cloud trail um and also mess around with config config is a really powerful um tool which allows you to configure and set configurations they tried about 20 times to apply apply this policy and it just kept denying and what then happened is they they went back out and then they Federated as another user now this was a gang or you know state level because we were you know multiple IP addresses or like their time traveling or so you know

um but what they then came in they Federated as a user with those comms and managed to apply these policies and and our sort of uh sort of conclusion is this was a you know a junior or somebody learning their trade but a really noisy activity and obviously the supervisor came on went come on you must know that you don't have permissions to do this because I don't think somebody who'd be able to write that policy would would not realize the basic mistake that they didn't have the permissions to apply that policy so it was quite um yeah I enjoyed it um okay so the key takeaways of this you know we are making this far too easy for

the threat actors and you know really basic stuff you know basic security will protect you so and it's is complicated I absolutely get that but the more you understand the more you get into the you know the the understanding of the cloud providers the different security measures on why they're there and why they're important the better you know threat modeling is absolutely you know the way to go these things the tabletop exercises the amount of times it's it's a process thing in terms of actually it's not a technological thing it is you know what happens with this who do I need to inform who's the expert of this actually understands this you know what if my S3 bucket is deleted uh or you

know people have left a ransom note saying I've got all your data how would we know that do we have the logs enabled those sort of things and really you know never stop learning it's it's a constant you know every day I'm learning and I'm sure it's exactly the same with yourselves so bit of a challenge for you you've seen how it's done that will be up till 5:00 I actually had the security team page me and say you know you left E2 um so I can't I can't hold it open any longer than 5:00 okay so the challenge is put uh inside the bside spell fast S3 bucket uh put a text document uh with your details like uh

LinkedIn or whatever and we'll do a little photo shoot after World you I did have some stickers but I left in Dublin sorry about I'll post them to you um cool couple things firstly a Shameless plug um um we run a couple of twitch shows um every Thursday I do one at 11:00 a.m. I'm the one without the bucket on my head um and you know there's the the the the North American Crew Wonder to r on uh Friday at 7 p.m. so aw .tv twitch I'm doing the very you know basic you know security startup the Friday one is is higher level so you know do please come along it's very interactive um and the more

better the final thing I'd like to say is this I went to my first bsides in 2011 um I always walk away from these things being absolutely humbled reminded how much I don't know so if you feel like that don't worry about it it's absolutely a normal thing and you know secondly this is you know a volunteer event run by amazing people and you know please do thank them they're taking time out of their day you know and a lot of time out their day and and and to do this so you know if you have a chance please thank the people because these we only get strong we only get better as a community stronger together okay thank

you very [Music] much