BG - From keyless to careless: Abusing misconfigured OIDC authentication in cloud environments

great so thanks for coming I know it's 6 PM uh this talk is going to be 19 minutes long so it's going to be short but intense um if you're in the back you might not see everything so you might want to come from forward a little bit um so no intro no agenta for now let's dive into it so we're going to talk about CI pipelines and how they authenticate to Cloud environments so uh when you want to have things like terraform deploying things to AWS you have different ways of of doing that the EAS one is to have hardcoded credentials into your pipelines um that are going to be used to authenticate now the issue

with that is that it's the cause for most of the cloud data breaches and so uh people try to avoid that very hard so the next the next way of doing that is using um a Json web token a GWT that your CI platform gives you so for GitHub actions it just passes it as an um to your pipeline and then you can exchange this for for cloud credentials we're going to talk today about a few of these misconfigurations and um what we found in different places including being able to compromise an AWS account from the UK government and we received this as a early Christmas gift which was a very nice surprise so my name is is

Kristoff I'm French as you can hear um and I live in Switzerland I've been living there for the last 12 years um I work at data dog focusing on open source uh Cloud security research and helping make our security product better so who in the room is familiar with AWS or works with it on a regular basis not everyone but a few people so a short introduction to a few things that we'll need but we'll keep it short so if you forget about AWS um the concept of role is usually used to assign some permissions indirectly so if you have a user and you want to assign it a role you generally abstract this through uh

so and you want to permissions you abstract this through a rule right so you say I want all my devops Engineers to have specific permissions so I give them this role which is just a set of permissions forget that in AWS a role means something totally different so I just want to clear that out of the way um in AWS there are two things that you can use to authenticate so two types of identity that you can use the first one is I am users and the second one is I am roles so I am users it's what you would expect you have a username and you can have a password to authenticate to the

AWS console and you can have program atic credentials to access the API um so you can sign in to the console with your username and password you can use your um access key ID secret key for the CLI and that's what it is and the problem with that is that these credentials never expire and really when you look at cloud data breaches over the past 10 years most of them are caused by leaked I am credential so that's why we're going to try to avoid them very hard um and the solution that AWS came up with is I am roles and it's basically a credentials vending machine so um you give it you ask hey I want to take some credentials

for this role you get back some temporary credentials and a role has two things a trust policy which defines who can get credentials for that role and permissions polic which is what permissions does this role have so it really looks like this you say hey I want to assume this role that's the term you get back some time bound cral that expire after at most 12 hours obviously the question is how like who can take credentials for this role right so that's where the the trust policy comes in uh and there are a few use cases the easy ones are when you are already authenticated in AWS so if you are a workload like an instance Lambda

function is quite easy because you already have an identity in AWS um if you are an IM user or role it's also same so the tricky part is when you don't have an identity in AWS yet how do you you know how do you authenticate yourself to get some credentials for this role um so mostly it's going through S idps for humans and open ID connect idps for machines so the Su flow is you know pretty usual so you are a human you go to your asure ID to your OCTA your Google workspace uh you get back a s assertion or something of the kind and then you can use the AWS API to exchange this token for actual AWS CR

shells now what we're interested in today is basically this thing but for cicd pipelines um so again we have something like this we have maybe a terraform or something like this that needs to deploy to AWS and we don't want any IM user access keys because they are bad and they will get us hacked so if we don't have access Keys we cannot leaked uh keys right so what we want to do is to use this Jen web token that gives an identity to our pipeline um and exchange it for iws credentials so there's a pretty well documented way of doing that uh which has been released in 2021 and um the way you do it is you set

your R trust policy to something like this so on the top you see that we say I want to trust this open ID connect provider and on the bottom we say if the GWT is properly signed I want to restrict that to only these GWT subjects and you see here that you have the org name and the repository name then the manual flow is from your GitHub action you're going to uh get back a is web token so if you decode it you will see it contains the right subject the right issuer and then you can exchange that using uh this API code assume role with web identity and it's a complicated name but basically you're

just giving a GWT to AWS and getting back some credentials and these credentials you can then just put it in in your console um in your terminal sorry and use that that's the manual flow but in reality people use uh you know standard GitHub actions like that where you just give it the r name and it does this flow for you so what the context I wouldn't be here if everything was right so what could go wrong um a few things so that's the the normal trust policy again you just have a condition on the subject that says I want to allow only CIP pipelines from this organization this repo this Branch so first thing that can be suboptimal is

allowing all the branches because maybe your main branch is doing a production deployment and you don't want all your feature branches to be able to get these credentials that's you know that's probably fine in most cases um maybe you have a wild card that that that says any repo in my organization can take credentials for batr which starts to be not great but you know maybe it's it's fine and then if you just remove this forward slash it starts to be bad because I can just create an organization called Data do hello and then I can assume this role and then you know again uh if you just push a wild card any c pipeline can do it if you

just remove it similarly so again if you have a vulnerable trust policy it means that any GitHub action in the world can assume your role um and typically I think we all know that cicd roles tend to be privileged at least mine are um and they have you know some access to the environment so if I'm able to assume this role I get the credential and I can directly access the iws environment now what the theoretical misconfiguration what do we do like how do we prove that this is actually an issue uh because that's on us so the first thing is we need the the RO Arns so ARS is the Amazon resource name which

is just the RO um name and the account ID so it's quite easy we can just use the GitHub search or services like Source graph and we see here that if I give it um a regex and the path it will give me 1.6k results there are a few duplicates but still it's a few hundreds as a second step you can create your own GitHub action so I just created a preate GitHub repo with a GitHub action in it um I run this SC in it to get back the Jon web token you can then extract it and take it on your laptop or on anywhere you want and then you can just Loop and you do you know for any of

these roles um can you use STS assume RO with web identity do you get credentials back yes no so we automated this collection we run that for a few hand of roles and we found quite a lot that was vulnerable which G gave us back some aw scentials as you can see in this case we did report that responsibly sometimes it's quite hard because you are looking at a GitHub hog that doesn't have any name or like not any named individual not any email so sometimes it's very hard sometimes it's very easy now I want to stop on a a pretty interesting case that we found um which is interesting in terms of impact but

also in terms of root cause so we compromised the credentials for one role we put that in our en variables and we see that it's called GitHub action mirror Repose Ro and if you Google this this scroll you see that it belongs to a go UK repo that says it's the terraform automation for kubernetes clusters that host gov.uk obviously that looks quite interesting so if you dive a bit into it you will see in the pipeline that it's using code commit and it's mirroring all the private repositories from the go UK org back to to code commit so even though we don't know the names of these repos there are some references in the rmy so this is the

public rmy that talks about this private repo go AWS data which sounds interesting um so what we did is just we went to AWS docs we basically said okay how do I use code commit to to clone a repo we did that and well it did work um so we were able to clone that repo and to access what was in there which obviously is supposed to be private and not be public interestingly enough uh so since the pipeline had G pull and G push access it means that an attacker would have been able to push some terraform files to this report as well and to back door it so we did report it to the to the UK

government uh the response was quite impressive 26 hours from report to fix um so it was a great relationship they were very easy to work with um so if you do find something feel free to to report it to them now I want to stop a bit on the root cause because this one is interesting so that's the actual vulnerable trust policy that was defined with terraform does anyone see something in there that looks wrong to me this looked fine and I took like half an hour to figure out what was wrong um so I'll give it away there are actually duplicated Jon keys in there and if you look at the Jon spec that's

not valid Jon so how the paror is going to handle that is undefined basically and what terraform does is just says well I'm going to just take the last value that you defined which means that this policy is with terraform equivalent to this one which is insecure because it doesn't have any condition on the GWT subject um so after we reported that Hashi Corp released a security Buon saying basically well we knew about that but you know there are some security implications that maybe uh we didn't think for but it's not something that can be easily modified at least we're aware uh they did relase some improvements to the terraform AWS provider so the AWS IM am policy

resource now checks if you have duplicated Jon keys in the string before passing it and if if you do it will tell you that it doesn't want to to to create it which is a pretty good one they also sent an email so no AWS also sent an email to customers that had uh who had vulnerable roles they implemented a warning in the AWS console if if you had like an insecure condition and they changed the API for create role and update role policy to say if there is a vulnerable policy we we just block the request so that's great it doesn't work all the time for instance if you use repo colon star it doesn't block it

um so you know it's it's good it's going to help but things are still going to be vulnerable to some extent and the previously created roles are no nobody has N Them automatically so um they might still be there now a few tips more on the first on the defensive side and then for for pentesters um this is a talk about misconfigurations in o DC authentication but it's still highly worth it I don't want to give a message that because it can be misconfigured we shouldn't use it so if you are if you using cicd p plans use oidc authentication it's going to be much better than I am users it's easier to configure more secure uh

Etc it's quite easy to check in your account if you have any any role that that is used by GitHub actions um so there's a one liner with awli and from there you can just either manually look at the trust policy either there is a tool that resonate released that looks at this these these trust policies and tells you if something is insecure if you have a chance of using GitHub Enterprise GitHub Enterprise Cloud you can also turn on something to get a custom oidc provider for your or so in which case you see that the URL is different the signing key is going to be different which means that even if you mess up your IM trust

policy um a pipeline outside of your or won't be able to get credentials for this role so it's a good guardrail but it's only for G Enterprise Cloud customers it's actually quite easy also to detect exploitation or exploitation attempts using uh cloud trail so if you look at assume role with web identity events and you filter out things that have user identity. username starting with your GitHub or name you will find anyone that tries to assume some of your roles uh while being outside of your G org so I think that's a quick win um to detect early attempts of exploitations or successful one now if you're aenor I would say that it's interesting to just like figure out is your target

using GitHub actions um there are also some abuse vectors through pool request so sometimes if you open the pool request and you change the CI code you might be able to to steal the credentials from there and there are you know some methodologies that that can be combined with that so typically if you find the N bucket there's a way to get the iws account ID from that whether the bucket is public or private and S bookon names are everywhere like there are in cname DNS things in JavaScript code in passive DNS everywhere and from there you can also perform um unauthenticated enumeration so if you have a word list of I am Ro names you can say does this role exist

in this account yes no um and from there you know it's quite easy to to build the word list for RO names like cicd GitHub actions CI and figure out if one of them is vulnerable so I think that can be a pretty realistic and efficient me methodology we only talked about AWS but uh the same goes for Aur ID Andra ID and Google Cloud so in Aid when you create an app registration you can go to the Federated uh credentials Tab and you can add kind of this same trust policy it's a bit harder to misconfigure I'm not even sure if you can still misconfigure it uh but it's the same principle you have an API an Azure API that you can

use to exchange a GWT for Azure cenal same for Google Cloud it's called work Cloud identity Federation it's the same mechanism as they use for authenticated kubernetes pod in Google Cloud so to wrap up um it's it is better to use o oadc KS authentication rather than I am users it can be misconfigured so it's important to watch out for um l trust policies and have guard rails if we can if we can have them and um we saw that things are getting better for the new roles but the old roles are still out there so I think that's a call for Action to go and find them thank you the slides are going to

be up here so if you want to donate them feel free thank you and I think we have three minutes for Q&A so if you have any question please uh shout otherwise I will be here

thank thank you [Applause]