Demystifying Cloud Infrastructure Attacks

um so yeah I'm going to talk about Cloud infrastructure security and um this is where I come from so this is a red team exercise that we did at a large financial institution uh I think it's like 10 years ago something like that and I think it's interesting in the sense that it's a very typical you know on premises attack so you start by compromising some web server in the DMs you find some kind of dependencies and you jump to like an internal segment somewhere and there you find some kind of uh you know way into the actual domain and the domain Network where you have the like main um active directory and everything from there

maybe you jump to the next domain with some like cross domain dependencies whatever and eventually you end up at your you know age value targets which could be like a main frame if it's a bank um so that that's a very like typical on premises attack so what the question we're going to try to answer here today is what would that kind of attack look like if they have the infrastructure in the cloud so we're going to try to draw that kind of drawing that we had there in the beginning but the cloud version of it and we start with a challenge that there is no Cloud so what are we going to talk about even but yok's aside of course

there is a cloud there's a cloud in the sense of there's apis there are these platforms where you manage your services and people have the same kind of services and it's like a common platform that we can talk about and that's what we mean when we say cloud in this presentation anyways so we're going to talk about today so as um the was mentioned in the introduction I'm a forensic investigators nowadays so I do forensic investigations and these are pictures from real incidents and that's why they're anonymized obviously uh this this is not one attack from beginning to end like a real attack but it's like all the components of the attack are taken from real incidents uh but I put

different ones together just because it would be very easy to De anonymize it otherwise uh but anyways so this whole thing starts with this single Factor log on and that's not good right you shouldn't have a single Factor logon exposed but yeah that's an explanation and the customer says that you know this device is enrolled and it's in the EDR so this all should be fine it should be isolated to the cloud so we just changed the password well they didn't say that but we can see that they changed the password and when we ask them what they've done they like enforced MFA you know you need to revoke tokens you need to like maybe reinstall the PC just to

be safe whatever so case close then I guess well the favorite thing we have as investigators is time stamps so if you put time stamps on these things that we saw the story is something different so it didn't start with that the device was enrolled and then the password was reset which would be mostly natural right it was rather the other way around so the password of the account was reset then the device was enrolled and then we see the single Factor log on so this is tells a different story right and we also have a addition to the security information which we didn't show a picture of but that is something that is I think it's an underrated persistence

technique when it's like you know you have this self-service password reset functionality right if you have an account you can add some additional factors and depending on the configuration you can then reset the password and that's maybe something that they don't check when they used to change the password if they don't do a fur investigation so then you have like a u backup way into the account whenever you want to as a f actor but let's look into that password change because that's the thing where everything started right so we see that the account that did the password reset it's one account that was resetted or whatever that English word is and then you have another one which

is the resetter uh and the resetter is the syncore account and that points us to a hybrid setup so there are three main types of a hybrid Cloud that you uh use which is Federated pass through and password hash sync and to understand what type of attacks you can do we kind of need to understand understand how these three variants work so if we start with Federation that is typically like an adfs but there's other Solutions you can use as well um the user goes to a FedEd app and they say that they want access and they say no you need to go to ashure active directory uh which is by the way nowadays called enter ID but

they keep changing names all the time so I'm just confused and I'm going to mix them up still uh so they go to enter ID and enter ID says no you cannot go to me to go to adfs so the user then goes to the adfs server and the adfs server checks with active directory and active directory validates and checks everything so everything is good and fine and then they give them a token that is signed by the AFS server and that step is kind of important here so the adfs server signs a token with secrets that is stored on the adfs server and says well you know if you have that token you can give that to

enter ID and they can validate that cuz they see that yeah this is signed by the actual ADF server so then they give them a specific token and then they can present that to Federated app and eventually they're in and can do whatever they wanted to do in the app in the first place so what we saw here was like this chain of trust right so the Federate adap trusts ENT ID ENT ID trusts the adfa server and adfa server trust uh trusts active directory so what happens if someone compromises the adfa server well then they can steal that token uh and they signing token certificate and the like distribut the key manager blah blah blah secrets and

that's what they did if you read about the solar winds attack this is exactly the kind of attack that they did at that point which is called a golden St attack um that is super cool and everything but that doesn't really involve a password reset right so that's not really what we're looking at here um there's no password reset needed because you would just be authenticated all of a sudden because the Fret actor is your ADF server at this point um so let's go to p through authentication is dead and in that case the user goes directly to enter ID they ask who they are they give them the username password and that is put on a log on que and then the a the

connect server which is something that is installed on a server in the customer envirment shakes for new logons and then it gets the new one on the que and the que validates that against the active directory which then pushes it to enter ID and this step it just gives a bleen which says that yes those are valid credentials so then the ENT ID can let the user in and do what they wanted to do so same thing here then what if the ad connect server is compromised then there is this technique called skeleton key which is that you patch the function that validates the login and gives a true or false as a response and you can

change it to you know if the password is banana and username I don't care about then let the user in and then you have the back like this skeleton key banana which you can use to log into every single account in the entire environment um that is implemented in aad internals which is a toolkit which I recommend you to look into if you want to learn more about uh AER security um and still this is not a password reset so let's move on to the first one to the last one and in this case it actually starts with the ad connect server so in this case the ad connect server asks the um um the domain

controller um to give me all the passwords because I need them uh that is a you know typical password hash sync kind of thing maybe you've done that as a red teamer somehow in a bad sense this is what it's actually supposed to be used the real use case uh so they get all the hashes um and then it actually hashes them one more time because they don't like nlm in Asher so they actually have a new hash which means if you dump hashes in Asher you're actually not going to get the classic ntlm you know the NT ntlm thingy that you're used to see when dumping hashes in on Prem ad uh so you're going to get this kind of hash

instead uh but anyway so the the the point here is that they sync the hashes and they keep them updated all the time which means that entry can all by itself decide if you your password and username is legitimate or not and at that point there is no need for a connection in between you just need that sync so to make all of this work you need a bunch of Secrets right so you need one secret which is the mol account if you're working a sock you might have seen alerts about that account syncing uh hashes uh that is usually a false positive but if you see that from the wrong server then it's usually not a

false positive and you need to act on it very quickly because it's really bad but it's a very common uh false positive right because that's an account that syncs password and uh then you have a service account which is uh running the actual service and you have the sync account which is talk to Asher and is actually you know pushing the creating new accounts and changing passwords and doing all of that the service account is interesting because that's the context where the um this service is running and these secrets the passwords to these two accounts are stored with TP API which is a encryption like a reversible encryption in Windows and if you're running in the same context you can just

call Protect or unprotect so you can just get the password as is and if you imagine that the Fret actor are on that server and they have compromised that they can get the username and password to the that sync account and that sync account obviously have the permissions to do the password reset that we saw so that's interesting and when we look in dedr we see that that is exactly what they did so they read out those Secrets uh from the ad connect server and you see that this is the actual account and when we look closer into this alert we see that this originates from the wrong IP address so the real password sync uh uh not

password the real sync is going from the legitimate server while this happened from another server so that's another quite strong indication that this is uh something is not right here so if we look closely into this account uh we see that it's a built-in account in Asher or rather it's a uh uses a built-in privilege role which is called director synchronization accounts uh since I wrote this presentation this has actually changed speaking of that they Chang things all the time so the specific technique that we talk about is not going to work and I think uh it did work when I wrote the presentation but now they changed things pretty recently and this is typical when we talk about

Cloud security it's like a moving Target all the time and this like minute you feel like you actually learned something it's already changed so it's kind of horrible to work with but it's also like good for us as Defenders and it's bad for us as Defenders because things can become vulnerable that wasn't vulnerable yesterday and vulnerability can just you know there's no cves for cloud for example like they will just if you find a critical vulnerability in Asher reported to Microsoft they will give you credit in online services there's no cve like you don't you have no idea what vulnerabilities there were or not um but anyways so this is a very highly privileged account and this is kind of

interesting because it's not Global admin but the Fret actor wants to be Global admin so how could they take this account that they have a password to and escalate the Privileges to Global admin how does privilege escalation work in Asher and this is a really good example of that I think uh because this one has two privileges which are interesting which is the take ownership of an Enterprise application and the um yeah um change service principles for it so basically when they have this account they can take a look at what Enterprise applications there are in this organization and in this case was this backup application which has really high privileges because it needs to take

backups from everyone's data and it needs to do a bunch of stuffs to be able to be a backup application and one of those interesting permissions that it had was the read write directory permission and since the Frat director is able to take ownership of the application they now can act on behalf of that application and then they can exercise those additional privileges which means that they can use those privileges to make themselves Global admin so it's quite a simple escalation here right and this becomes much more difficult when you start looking into the arbach model and everything that is in AER and it's really complex in a large organization to not have this escalation

chains um that step there where they acted on behalf or like as the application is kind of interesting also in a persistence sense because as investigators we tend to you know we know this bad thing happened let's see if there was an new new applications registered but you should also check if someone changed existing applications because you don't need to create a new application if you you can just add your principle to a existing application which means that you have a API key basically uh to act on behalf of that application and that is something that is really hard to find actually if those default 90 days or whatever log retention period you have has gone out

cu you have this log entry which says that they added a service principal CR but after that period it's kind of hard to get that out there's no like easy way in the GUI anyways so they added themselves to Global admin and they did all of this right um the thing that is a bit weird still I guess is that this device was in the EDR right so how that doesn't make sense cu the threet actor enrolled their own device and then logged in with single factor and it turns out that this customer have when you enroll your device they automatically push out an EDR agent the Frat actor did not know that so the customer came to us and said

that uh I think we have a full export of the threet actor's machine uh do you want it like yeah sure so yeah uh that kind of backfired on the Fret actor I guess uh they they tried to be stealthy by being enrolled and you know making no alerts but yeah that didn't really work out uh we'll get back to that uh if we draw this uh like we did with the previous thing we see the first thing that we know at this point is that they are on the ad connect server they dump Secrets there and then they take over an Enterprise application they make themselves a global admin and then they enroll the device and they add some

security information for extra persistence and then they do exfiltration from high value targets so that's like the final goal uh of this campaign um we haven't really shown any uh pictures of that because it's just a bunch of logs of downloads which isn't that fun to look at maybe uh so the question is like how did they get to the ad connect server because they can just show up there out of in air right they need to somehow connect to it and we it traces back to a local Jenkins server and Jenkins is this build server that is uh I mean quite commonly used uh I think it's more commonly used in the customers that tend to get incidents

incidents than customers that are really like on the Forefront um in that sense like it's not a bad solution per se but I think a lot of customers are really like built into yaning because they've invested a lot in like plugins to yenin and they have this like really really big yenin server that kind of builds everything uh that at least that's the case with this customer here so uh yankin isn't bad by itself but it's usually used in a bad way so for example if you have the administer uh account there is a built-in remote code execution that's a feature where which they like clearly specify you can execute code on the underlying system

from the web services so that's not so good um it's also quite easy to just dump all the secrets you can imagine if you have one yankin server that is supposed to build to production test to whatever and whatever uh and sync from this repo and you know add container images to this registry like it needs all those secrets and if you just run a script like this you get all those secrets and that is what we found that they did so they got like a lot of Secrets uh because yeah you know you have this one big yenin server which has the secrets to everything and uh the next question is like how did

they control yenin server then so we found that there was a running on the yin server and it connected out to gmail which is kind of interesting and this is a case where fret actors also use the cloud and I call it C2 as a service I don't know if that's a name that's going to catch on but basically we it doesn't have to be Gmail this can be like we' seen it with Dropbox we've seen it with s3e we've seen it with um slack like pretty much any service you can use this where you have the possibility to post them message and read a message that's pretty much everything you need so the way it works is kind of that you know

you create a draft in Gmail you don't have to actually send a mail you create a draft and on the other side the Fret director is logged on to the same Gmail account and they read the draft and then you you sync data that way back back and forth which means that you can create a C2 Channel over that protocol and this is kind of smart because what we're going to see soon is that this customer actually used Google Cloud as like development environment which means that a lot of traffic is going to go through Gmail or at least like Google servers uh with like G gcp and so on um if if it's a smart fat actor they pick

something that you use if it's a stupid fat actor they pick like Dropbox at a bank and that's super forbidden and they have really good detection for that not for security but for gdpr and things like that so it can also stick out a lot but if they look at what is actually legitimately used in this customer and then they pick that then it's actually really hard to see this at least from a network perspective and obviously from the other side we're completely blind because we're not Google so we cannot see what the fet actor do um towards them in Google but if we reverse the malware we actually going to find some kind of API secret right because it

needs to authenticate to that and if it's persistent it needs to keep that somehow within the malware and there are different philosophies here but I'm a quite strong believer that it's a really bad idea to use use that API secret to log on to the account course we can't do that right technically speaking but keep in mind that if you find a secret like this and that goes for you know SSH servers and whatever else and I see a lot of people just going complete cowboy and hacking the hacker but keep in mind that you know the Gmail account is probably not the fet actor's account they probably hacked that account so you're probably hacking a third party

and you're just a second hacker hacking that account and there's so many things like in a while FBI is going to find that account and it's going to be quite hard to explain for you what your IP is doing in that account maybe you're using a VPN or whatever but you know it's really a bad idea Plus in I don't know the law in Germany but in Sweden it's also illegal so it's stupid for many reasons so don't do that instead what we have good experience with is going through communities like first and TFC and other like trusted uh vendor communities and then you talking to the people at for example slack or at Dropbox cuz they really hate it when

people you know misuse their infrastructure as a C2 channel so they will usually be really helpful plus they can actually you know track down the entire frat actor infrastructure cuz they can look from a broader perspective than you can when you try to use some API key so it's just much more effective and it's just like the good way to go so strongly recommend doing that instead uh but anyways so how did I get to the yankin server we see that there's a malware on the yankin server but that doesn't explain how the malware got on the yanin server so let's go into that the yankin server in this case is uh configured to run Docker containers with

Bild jobs and it's just like this one big yenin server everything runs on the same machine um and we see that the logons to the Jenkins console comes from one of those containers and the reason that that works is that because someone figured out that these containers need internet access so they needs you know Fetch dependencies and whatever um so they put them on the like default Docker Network which means that you can also reach Local Host on the host and yeah there you go you can log into the Jenkins console so uh the question is though like how do those containers get compromised because it's you know it's not like they're internet exposed it's

just like offline Bill jobs so we' seen a few different scenarios of how that can happen and one of them looked like this so you see that this is a package which has version 9999999 something and the name of this package is the name of a internal pip Python package of at the customer so like you know company name Library dot whatever and if you look in the official Pipi repo you see that someone has registered the same name and they put a version 99999 and the thing here is that if you use extra index URL to point to your internal Pipi repo then pip is going to just select the newest version from the public or your

internal so what they say is that you need to use index URL instead if you want to you know only use your internal repo otherwise this is an expected Behavior but you know no one you know have the time and interest to sync the entire piie repo and keep all dependencies up to date so people still use extra index URL unfortunately and this is just a closed nonix so it keeps happening time to time it's called dependency confusion and there's a good R up there uh from that guy there if you want to read more about it um another one we' seen uh is kind of interesting it's a feature in PIP which is not a

feature but it's a bug in PIP but it's not a vulnerability in PIP according to there's no like at least there's no um CV for it and the thing is like if you have extra index URL configur uh it would make a clear text request the first request and then it would upgrade to SSL so we basically saw a man in the middle which just responded their own package instead of the real package um another way that could happen is if someone compromises the code repository right so if you have like this gitlab CI something you can change what runs in the container uh you can also change the actual code right that is going to run in the container if you

can push if to the rep post that's going to run there um this is all in itself like a really big topic uh we have the problems of shared runners in general where you have like you know you have usually you protect your branches maybe and like just not everyone can push that's going to run on the um run on the build server cuz you know you have the problems right where you can dump the environments secrets and everything like that if you're able to change the code it's going to run um but if someone is able to create another git repo and that's going to run on the same server then eventually they can just start a

build job and then they can just stick around there and wait for the sensitive secrets to pop up uh what we saw in this case was that the uh fret actor compromised a um gitlab account and we see that they made a push to the repo and they changed the requirements txt file and they changed the name of one of the dependencies to GitHub and the name of the dependency which is a threet actor controlled GitHub repository and if we look into that and we do a diff from the legitimate package to this bad package we see that there is one file that is changed which is a setup.py file and that's something that's only going

to run when you install the package which is exactly what happens in this build environment right so it doesn't really affect the production or anything it's just affects this build stage so we see that they come from gitlab basically and that also connected out to the C2 channel from that um compromised container which and then used the network to break out to the underlying jenin server and from there found a bunch of Secrets we're able to jump to the internal um connect server which made it possible to then jump in to the a environment so the next question is then like because we know that they execute code in the container right but that doesn't mean that they can just get the

uh like one of the administrator accounts in the yenkin interface right that's another set of credentials that shouldn't be in a Docker container and the answer here is kind of long because it starts with gcp and how lateral movement Works in gcp so imagine that you're a threat actor and you compromised one machine in Google Cloud platform gcp so you know this is kind of interesting in in the sense that if we go back to what we were talking about in the beginning um in Windows Active Directory like where I com from at least and the classic on premises attacks like in pretty much every single attack you have that the Fator compromises one machine

and there is some kind of dependency from that machine to to the next machine that can be like they reuse the local admin password it can be that a admin which has higher privileges like a domain admin for example uses RDP to log on and then their credentials are cached in memory and they can dump that memory and then log on to the next machine uh there's a bunch of different you know you have like service principles with um yeah that credentials are stored in L LSA and so on and so on point is this is like Classic on premises lateral movement and what we're going to see now is like what does that look like when

we're talking about Google cloud and there's kind of an interesting default setting in Google when you create a virtual machine which is that by default default you will have a service account running as the machine which is going to be stored on the machine and the default setting is that it has like read access to um storage and stuff which is like a really high privilege as default and you can like curl the metadata for example to get this out um this is something you might have seen if you're into web application hacking where you can do ssrf in the cloud and you can dump the metadata for the uh uh for the um yeah

machine that you're executing on uh I think they changed that quite recently nowadays you need to also specify like a header uh so it's a bit harder to get it out now uh with ssrf but now we're talking about a fret actor that has compromised the machine right so they can obviously get all of this so that's one way to get that default service account um another way would be to get Cloud CLI Secrets because in environments where they have like a pretty good like you know they locked down the environment with vpcs and things like that then you kind of need to jump to a machine and from there access the apis to troubleshoot

something and so on you know we have this infrastructure as code but in in uh in practice it usually requires quite a lot of manual magic to make things actually work right so what the Frat actor finds in those cases is that someone SSH a legitimate admin SSH to a machine and from there they want to use the CLI tool so they do like you know gcloud EIT or whatever and then they offen at the session and then it just that sticks around in your user profile so if they're admin they can just dump that those stored secrets and then they're inside your gcloud um so it's kind of easy to get that um you can probably decrypt things

and do it like in a really fancy way but the easiest way is just copy this folder and paste it in your own machine and it's just going to work so the you know if you start using these admin tools and you uh spread your admin presence on a bunch of systems then uh the fet actor is going to be able to get them when they're um yeah compromis the machine so basically that's what the fror do they look at what kind of secrets are stored here it can be this kind of credentials that we talk about now but it can also be like you know the password to a SQL whatever because the application needs

access in a connection string uh it's a bunch of different things we like try to jump to the next machine and in this case they find a um accounter that has like set in instance metadata that's kind of how it works in Google like you add your SSH key and uh when you run the SSH command and the yeah that would look something like this when you do that and uh then they log on to the next machine and of course they just repeat the process so maybe they didn't get everything they wanted in the first machine that doesn't you know happen often but when they start doing this iteratively and very methodically and just searching for new and new and new

credentials eventually they have access to what they want to to so um this is kind of interesting in the sense that this is very much the same as we see in on premises active directory right it's not a that active directory is sides you have ah well yeah it's soon soon done sorry uh completely lost time here okay so basically it just keeps going like that and in this case we see that they uh yeah dumped the credential that they needed to execute code on Jenkins log in there from uh this escalation in Google Cloud so the question we're going to answer now is like how did they get into gitlab and there's a bunch of ways to do

that you know bypass MFA uh one of them is info Stealers because it's um um yeah it's quite well spread these info Stealers you see them all over the place they focus on like browser Secrets a bunch of other things as well but when we talk about club and so on that's that's the interesting part right so it's kind of easy to dump Secrets this is pretty much a oneliner to do it just uh start Chrome connect with the uh like the bugger end point and get the cookies you have other types of Secrets as well if your account is as active directory joined you can dump the refresh token and so on um we start to get a bit

better when it comes to this uh we start to like have some kind of token detection and so on if tokens are stolen but there you have pretty much the full attack right so it starts with a info stealer on a client they use that to jump into the gitlab execute code and then they're able to jump to machine in Google Cloud escalate and then you have the full attack from there uh we also have a strong indication in this specific case because we you know had EDR on the Fret actor machines we can see them browsing Russian Market which is a market for that kind of thing and the original and the final thing we're

going to talk about today is the source of that Redline stealer which was hosted on the Microsoft's official GitHub account which is kind of weird and their competitors were really quick to point this out but bleeping computer did some digging and figured out that this is not hosted there well technically it is but it's not that the repo is compromised it's actually a feature in GitHub which is that if you answer a issue and start just doing a draft you can upload a file as part of that answer and if you just discard this the uh file is is going to be hosted and that answer is going to be under the repo so the last time I shaked anyways

this was still possible to do um so it's quite an interesting feature that they have and with that I want to thank GitHub for hosting my uh slides in their official documentations um and by any way this link doesn't actually work you don't need to uh it's one 2 3 4 One 2 3 4 the vulnerability is there but it's not a real vulnerability uh so it's not uh I actually didn't exploit it my legal didn't think it was a good idea but yeah so cool sorry for running over but uh thank you very much