Breaking free from the chains of fate Bypassing AWSCompromisedKeyQuarantineV2 Policy - Andrew Kraut

um Andrew andon are security researchers with permissive La focus on Cor research in the class please welcome [Applause]

Andrew sorry give me a second to set up here

I have piles and piles of research articles that I use to reference like

cool um okay uh so this is a little different than a normal presentation this is uh got some multiv elements I want to do a sound check real quick to make

sure okay cool uh yeah so so uh hi I'm uh this is this is breaking free from the chains of Fate uh we're going to talk about bypassing the AWS compromise key quarantine V2 policy that's a mouthful um I'm Andrew my co-researcher Bon was supposed to be here and not crowd strike related uh was not able to make it it turns out you can't be in two different places at once so uh he's giving this presentation somewhere else because we managed to double look ourselves this weekend um I'm Bon has been uh in the in the tech industry for about four years now he's done uh it work and cyber security stuff for telecoms for banks uh as well as working

for some msps before he joined our research team for me so uh myself I've been in the tech sector since the dialup era um and I've been doing some security work since around 2002 uh I've been a redeemer uh instant responder forensic Sky infrastructure security appc I'm also one of those weirdos of Life GRC so um yeah um I realized that there's a lot of presentations here for practitioners uh so I decided I would try to put something together for all the threat actors in the audience I'm going to teach you all of the fun things that you can do when you discover elite AWS key and it has been quarantined uh so quick history um

rhinos SEC uh in 2018 they released uh AWS escalate along with a very long blog post outlining about 28 different methods for privilege escalation in AWS um of those 28 methods when your key gets quarantined only about eight are left uh and four of those are only kind of partially allowed so we're going to talk about all the different things that you can do with that now um folks that have used AWS who has had a key quarantine anybody have you leaked a key I see one okay um so the AWS quarantine policy is interesting it's a deny policy it gets automatically attached when Amazon detects that your key and its secret component have been leaked um the

question about how they do that is a little bit fuzzy I know that if you post it on GitHub and pastebin both uh e either those places they will pick it up pretty quickly um but it's a whole slew of different things that they specifically prevent you from doing with that key once the quarantine has been uh applied to it um so uh we're going to pretend today for the purposes of this that you have like admin PRS so maybe this is your it admin's laptop uh and the inos Steeler trojen has just pulled it into into somebody's uh somebody else's hands or maybe this is your terraform key and your infrastructure is code uh system has been has been

breached and that key is out there um so the first thing that's interesting is uh it only blocks right access so any readon policies that you have attached to your I users continue to function just fine you can effectively enumerate anything in the entire Cloud environment so that makes a lot of the rest of this really interesting um um reads gets and describes or what Amazon classifies as that read only access so when we get into privilege escalation uh now that that becomes interesting because now we know what's out there so to start with uh you can assume roles so um because it doesn't prevent Ro assumption uh if the identity that that the key was tied to is allowed

to assume a privileged role uh that doesn't go away which means you can assume that role you get all the Privileges there in and then you get to go and have some [Music] fun um next one uh these are two that you probably won't see very often but data pipeline uh data pipeline allows you to reliably process and move data between different AWS computer storage services as well as on premise data sources at specified intervals that's Amazon's definition it's basically a crown job plus rsync with a little bit of cloud flavor mixed into it um quarantine policy doesn't prevent you from defining pipelines or uh pipeline jobs or from creating pipelines but it does prevent you from passing

roles so if you've done any advanced AWS stuff you can pass roles to other services to allow them to take over temporarily that's not permitted which means create pipeline isn't very useful uh but we can uh put a new job into an existing pipeline which means we can give an existing pipeline a new location to copy all of this data that it's processing so maybe that's my S3 bucket that I own or maybe that's some external service that I wanted to just post all of that stuff to um another one that you probably won't have seen before is codar codar is kind of their like Co coding pair coding thing um you can associate team members to the

uh to the codar and you can specify a new project role if you have the permission to do that so if you make them owner then you can upload an SSH key and with that SSH key you can get access to the instance that's running the Cod star now those are transient so you don't get a lot of persistence with that but it's still a little bit interesting because you can usually get access to the code repos with that stuff um my favorite or one of my favorites is Sage maker though um Jupiter notebooks has anybody played Jupiter notebooks I've seen quite a few people cool Jupiter notebooks you might know more uh as a python remote code

execution as a service um this one you can create a brand new Jupiter notebook and uh and then with that Jupiter notebook you can access the metadata service so that means that you can get additional credentials uh that that that Jupiter notebook has uh and with those you can have all kinds of fun [Music] um another another cool environment that Amazon uh has is called glue uh glue is for ETL work um so this is a lot of of uh structured data manipulation and storage um effectively you define your jobs and then glue will just go and monitor those things run them regularly uh log everything and it's kind of a turnkey serves once it's set up um but

that it does have a Dev environment that you can uh use to to uh troubleshoot your glue environments so when you do that you can ask for an instance um and when you ask for the instance you get an SSH key that allows you to log in and have all all kinds of fun in that I'm

[Music] uh um I want to I want to talk about glue for Just Another Second so you get uh you get some really interesting permissions with this one because this is tied to so many different things um it does allow pass roll uh which means in some limited circumstances you can pass uh specially defined roles to either ec2 instances or other blue resources uh so you can kind of assemble a chain of data flow where you can start to move this data around or Grant yourself additional access with very very small steps um generally speaking the amount of additional roles you're going to get beyond what you had to start with isn't going to be significant

um but one thing that it does offer is uh enumeration of S3 buckets um it offers enumeration of all of the I environment so users roles groups and policies and ec2 um and then there's an optional permission that some people Grant which is S3 full access and you get a lot more with that as well um so blue is a really interesting thing to explore uh lambas are the next one I wanted to talk about uh with lambas there's there's a bunch of different moving pieces a lot of folks use it pretty simply where you upload some code it gets triggered and runs uh this is the server code execution piece uh if you want to add some kind of reusable

components lamb allows you to use layers uh layer is a component that gets attached to lots of different lambas um and it's also uh um a cross account resource so while the permissions that you're given don't allow you to update the function code itself it does allow you to update the function configuration which is how you add layers to an existing function and since a cross account resource you can create that layer in your own account and make it public and then update the function configuration on the template or on the target account so now this this Lambda function that already existed in the AWS environment you're looking at now has your layer attached to it and uh when you when you update

that function configuration you also get handed all the credentials uh so you get everything from that original [Music] account um so the next the next piece that's interesting is uh you can impact a lot of different infrastructure uh within the the environment you have access to once you get that key uh the first one that that is interesting is S3 bucket versioning so uh S3 buckets allow you turn on versioning this gives you some protection against accidental or maybe on purpose dels or override overriding uh tampering um quarantine policy prevents deletions but it does not prevent put object or copy uh and it also allows you to put bucket version in which is how you can change that so you

can suspend the bucket versioning uh copy or put an object over the top of one that exists and if you do it two or three times like uh we did here then you've guaranteed that you've overwritten any existing versions as well uh and uh then once you've done that uh you you've got this bucket uh with maybe log files overwritten with a bunch of zero files [Music] now um and and to expand on that one uh ransomware is another possibility so uh if you when when configuring the bucket versioning you can also configure a KMS key that you own this is another cross account resource so it exists in my account and not the target account um

when I overwrite The Bu the objects in that bucket um in order to do so I need get object put object and encrypt or I need copy which gives me all three of those um when I overwrite those objects they get re-encrypted with the new KMSP that's that's defined um so combined with our our previous definitions with put bucket versioning uh now we can encrypt your S3 bucket and uh Ransom you the access to that encryption fee um a couple other other ones that are I think are are seem very small but but maybe have an interesting impact is uh cloud trail so I can stop and delete Trails uh with with the quarantine policy in place um but this doesn't stop

the cloud trail API so um through the research that my team's been doing we see a lot of activity in people's Cloud uh AWS environments also uh azer and other Cloud environments um but in the AWS space we see a lot of threat actors try to delete uh or stop Cloud trails and the interesting thing uh to me at least is that that doesn't stop the logging from happening it just stops logs from being shipped to either an S3 bucket or Kinesis stream um and so when when you it it makes sense to me that those are not prevented because they don't stop the logging from happening overall but the other interesting thing that you do have access to in the logs

is you can look up events so as an attacker when you find a leaked AWS key you can actually go through the log events and look for any leaked uh other sensitive information you can use that to enumerate any cross account permissions or access or trust that might exist because you can see all that activity happening so I think that the logging one is a really interesting piece as well uh guard D is another one um there's a whole lot you can do with guard Duty uh either as a child account or as a a master account um and it kind of uh amazes me that these things are all permitted but um they really don't

change a whole lot other than just disabling those things and there are tons of logs that are generated when you do these things so if you're paying attention to that you'll catch those pretty quick um the last piece is there are two really interesting strategies where you can cause some pretty significant financial impact to uh the the accounts that you're targeting and again we're going back to Landis um you can invoke function uh which costs money every time you do that there's uh plenty of early AWS stories of companies that built their infrastructure around lambas and then realized that their bill was so big that they practically had to fold um so that there's a there's a denial of

service possible there if there are spending limits you can also delete functions uh that's a pretty disruptive behavior um and uh more recently there's companies with a lot of intellectual property built into those lambas so if you want to get the function you can see exactly what they're doing with all of that activity out there um the second one though is uh and this one I think is my favorite of of all of them here is in the ec2 realm we talked earlier about how enumeration is not uh is not really prevented at all if I know all the ec2 instances that exist uh I can really reap some Havoc by both stopping and terminating all of your instances uh so

it's really easy to walk through those and additionally for some persistence if I uh update the user data which is the startup script and I stop those instances if I wait for an admin to restart them now I have persistence on your [Music] inst so uh real quick recap um you can enumerate everything you can assume roles you can uh submit data pipeline jobs access codar those are are a lot of places where you see data or or uh source code you can create tter notebooks you can access blue environments add layers of lambdas overwrite S3 buckets disable the versioning you can stop or delete cloud trail log shipping access all the log events uh really screw up the guard Duty

detections delet a lambas stop destroy or reconfigure ec2 instances so there's a whole lot you can do when that quarantine policy is in place um there are some uh important notes I want to pull up here um first of all for uh for the responders um when you look at this automation that actually triggers when this quarantine policy is attached you will see the user attach the policy to themselves which means that any user that's going to have an access key needs to have the ability to attach user policies or roll policies to themselves um which is a really strong permission uh so you you generally if you want to protect against this misuse but still

allow this automated response you have to add some extra queries extra constraints to the permissions for these things if you're going to allow them on there the second piece is that when this fails to happen if it tries to trigger and fail uh you won't really see it properly in the logs you will see that user try and fail to attach a policy to itself and the extra metadata that is normally present in that log will not have any of the information about the policy that's trying to be attached so it's difficult for you to tell whether an attacker got access to that key and tried to give themselves admin privileges or if Amazon tried to uh

prevent you from shooting yourself foot um and the last piece um Amazon wants me to tell you uh because we submitted this talk to Amazon um and it's really important for them that I tell you that uh this policy is designed to prevent them from issuing huge refunds to customers that move their key and get Bitcoin miners spun up so it's not a Panacea you have to take action so make sure that you pay attention to your leak Keys um I want to open it up for questions I know that was really quick um do we have any questions about about how this uh quarantine key policy stuff works yeah so is the reason that the

permissions still exist because they don't want to break the existing functionality like there's what are the reason is there yeah so uh all the conversations that we had with the amazon security team um they said they were trying to balance between uh early customer requests we leaked to key a bunch of Bitcoin Myers spun up and now we have a two-day bill of like tens of thousands of dollars right Amazon doesn't want to have to refund those things so they're trying to stop the most egregious misuses of that key while still allowing your infrastructure to comp continue to run but the reality is with all these different things that you can still do there's still a ton of

danger and what we're seeing in the research at so is we're seeing from our customers that are actively connected is a hey Will leak and months will go by before anyone gets around to rotating it sometimes a year or more customers see this they see this quarantine policy get attached and they think okay cool Amazon's got my back but they need to understand that they really don't and they're they're not trying to um this goes back to that shared management model that Amazon preaches from the very beginning so it's really up to you to respond to a lead key and not rely on any of this stuff to to help protect you so I have a question which is kind

of technical so I have done some engagement checking the permissions and policy of AWS using the problem M what do you recommend any other application that is useful um to what's your end goal what you want you want to reduce the permissions that are available to the minimum you want to see what's out there when they say that hey go ahead and check your AWS audited what that is I would say that the right strategy is to look uh and this this is the strategy I take with all of our customers today is look at cloud trail look at some historical record as much as you can really and look at all of the things

that happen from that identity what are all the different permissions that it uses today and can you reduce the permission set that it has just to that set now it's a lot of homework and a lot of research because uh some for some of my customers we need to go back more than a year to see those those very narrow use cases where some recurring job that happens once every six months uses one extra permission um but that's that's the strategy for reducing res in Cloud right is absolute bare minimums and now there's some new tools that Amazon's put out there's uh there's the new permission explorer that lets you look at that and it it tries to do a lot

of that homework for you um but that gets you a great start so take a look at the the the tools that they've built recently and they've blogged about because those help a lot too yeah anything else cool um I've got links uh to the labs team that's my research team uh ATO and our blog where we try to be uh public with all the research that we we do so uh more stuff from this will show up there as long along with a real big list of all of the most dangerous privileges that we think are still allowed when you're um so look for that in the near future and thanks to the French fuse guys for a

bunch of ridiculous videos and letting me put them in my presentation thank you guys thank you [Applause]