CloudShell Abuse: A CTF, API, and Persistent Access to CPU/Network/Storage

All right, why don't we get started a few minutes after the hour. Thanks for coming. I'm Jenko Wong. This is Chris Ryan. We're going to jump right into it. Uh we do various kinds of security threat research and we want to tell you a little bit of a story of some abuse and it's sort of a story about how we we start with some research and then where does it go where it becomes sort of uh interesting from an exploit or a blue team perspective. So let's jump into it a bit. And the genesis of this, just to give you the backdrop, was last summer's um I help out with cloud village. We have CTFs. We're at Defcon. And a CTF um

I created was all around cloud shell abuse. So we're going to take you on a journey like a five or 10 minute hyped up journey of playing that CTF and what it involves. So you're going to have to sort of put on your red team hat for the first 10 minutes. And it's the idea that lasts through the rest of the presentation about what can we do with something with cloud shell, right? There was over 500 players, 300 teams. There was one team from Japan that did solve it. So it was considered a hard challenge at the end of the day. Uh which was part of, you know, the the goal. There was a jailbreak. jailbreak

in the sense of we'll have to define it as you see the challenge but there was a lockdown environment with a flag and and it was a Linux environment and someone actually broke out of all of these things and got root and so that's sort of fun unless they take down environment but it was it was a good time it was a multi cloud challenge we're going to focus on AWS agent mission Impossible theme and just a shout out to the winning team and a couple folks who really banged the hell out of it there were really good reteamers palmonger Mr. Meows just a shout out. So we're going to switch over to scripted session to give you guys a feel

for that. So while we're talking the front end, I'm not going to dwell on the front end was a web page. That was the entry point of the CTF. So imagine you come in, you've got creds to get to this page. Where do you start? Well, of course we're all thinking, hey, web vans, right? You might do a little bit of discovery and find the IP in ASN. Oh, it's sitting in AWS, right? So normally what everyone's doing is throwing everything at it. You might throw burpuite your favorite web phone d from a challenge perspective. This isn't a web challenge. It's a cloud challenge, but it was a little warm-up. Get people in there, right? So there's

various things in there. People spend time on uh stego on the images. That was a dead end. But there's a hidden form uh postinput form commented out on the page that hints that there might be some functionality and endpoint. Once you do that, you're thinking SSRF. So, let's get started on

hey, as attacker as a player, I'm thinking SSRF. It's AWS classic attack. If I can get an SSRF attack, hit the metadata service, I'll get the token representing what's running on that EC2 instance, right? So, at the curl level, we're just showing the guts so everyone can sort of see the ground truth, right? There's a curl command if you can see it. Hey, what's important is that what's in green is the metadata URL. It's a 169. You can only execute it from the EC2 host, but the SSRF gives us a hole. It's usually in version one of IMDS. AWS has closed that, but you can still run version one. It's not taken away blah blah blah. Security by default, all that

crap keeps us employed. It's still there in the CTF, right? So, you follow it with your second call. you first you get the the the roll at the top small fill web roll is what the EC2 instance is running under I actually need the creds I feed it into the second curl call again in green is what you hit in the metadata service doesn't matter this is all standard stuff right if you were on command line in the EC2 instance this is what you could do to get your creds this is uh coming in through the web server remote I've offiscated the tokens but you see spit out the AWS temporary tokens there's an expiration time on it

great awesome got initial access. All right. Now, we go cloud, right? So, what do we typically do? Some way or fashion, usually with tooling, but we're going to do a little bit of manual stuff. You want to discover your permissions, you want to discover what resources, that's sort of a natural starting point. Whether you use pacu, whether you use your own scripts, whether you use your own tools, right? So, we're just going to show you a brief thing of that, take you on that journey. So, who am I? Get caller identity. Yes, I'm that role name. I'm just verifying and now I know the account. What can I do with that? Well, I'm going to find what permissions

I have under my identity. So, what's complicated in AWS, but all the cloud providers, how's that defined? A whole bunch of policies interact, right? Resource policies, IM policies, or policies is a nightmare. It's why you have uh, you know, identity analyzer tools just to tell you what the heck you can and can't do. So, if we do it old school style, you might do a couple API calls, you might list your attach policies, you might list your inline roles, blah blah blah. I can see some errors. Those are by intent from the builder side. I didn't want to let them see everything they could do because it's a dead giveaway. I didn't want to let them know right off the bat that

they could see things. But as an attacker, that's what you're trying to discover, right? So, I see that I drill in. Some policies I can and can't see. I'm getting errors. Oh, one thing comes back. A a big fill policy name doesn't matter. That's tied into the theme of the uh the challenge. But I start to see resources in red S3. Oh, buckets, RDS, database, lambda, EC2. Oh, wildcarded. I might have some uh sort of wider privileges, maybe escalation privileges. So, maybe I go and exploit those or try and see if I can look at buckets, right? So, boom. Hey, a bunch of buckets. Some of them pop out as interesting. Cloud trail. Really? Maybe I look at that. The

syndicate one jumps out because in the challenge there's a spy theme. The syndicates are the bad guys. Maybe I take a look at that. Oh, wow. I can actually enumerate it. Looks like source code. Whoa. What is that? Maybe I download it, right? Again, this is just going deep. I download it. I pull it to local source. Oh my god, it's the cows source code, right? I execute it with WTF and I get some extra gunk. What is this octal dump? Ah, it must be some encoding. Oh, it happens to be, you know, something easy octal. And it's just a little hint that it's actually you're barking up the wrong tree, right? Basically, nothing in the source code

was the hint. Now, a lot a lot of players wasted time with that and you could I actually didn't want them to waste too much time. The point was S3 buckets isn't where the flag is. It's dead end. You could spend all day looking there. Well, what else are we going to look at? You're going to look at every other resource. You look at the EC2 instances. There is one running the one running the website. Hey, I can't get access to it, but hey, there's metadata. There's tags. Oh, there's a suspicious one. Hexonova is a character in there. Oh no, another suspicious looking thing. Must be encoded, right? happens to be maybe the same encoding.

Yes, it is. It's another hint trying to say that in the game play this is the wrong area. Right? I you want to make a fun challenge. You don't want people spinning their the wheels trying to enumerate everything and looking at every bit and bite. Right? The point was EC2 instances isn't where the flag is. So what happens in the challenge? Basically they go through every resource. They find nothing. And that was the hook of the whole CTF. So those of you who knew AWS or pentesting, imagine going into environment, you do everything and you enumerate everything you have access to. No privilege escalation, blah blah blah blah, you go through everything and you find zero.

What do [snorts] you do? So now in theory, we wanted them to think back to maybe I missed something on the I am side of what I have access to. And it's true. Back here there's a different API call called get account authorization details that actually dumps every you have to have permission to it but if you do it dumps every policy in the whole account across all identities and in there I can actually now see small fill policy. Earlier when I did different API calls which are a little bit more known I could not see the details here. So, this was sort of a case of testing the players. Did you remember to run this?

If you're running tooling, you probably would have caught it. If you're doing it by hand, you might have missed it. In fact, early on, a little bit of history, I was thinking of taking this away, but I mistakenly left it in. And the point is in there, you can see that this identity, this small Phil Web roll actually has cloud shell permissions. And that was the whole hook and premise was, so here's the test for all of you red teamers. you put on the hat. If you didn't know this, would you ever thought of cloud shell as a service? If you didn't see it as a permission, would you ever thought that? And that's the hook

and whole premise that started this whole research. Why? It's not an API enabled service. It is a browseron thing for admins to have a shell inside AWS with pre-installed CLI through the browser only. no APIs, but clearly you can permission some things about starting it. So once to get back on track, once you get that idea, at this point you see it, you're not sure that's where the flag is. All you know is you spent hours and found zero. So you got to you got to run with it. How do you even explore that? You got to get to console. So now you have to remember in AWS I've got API keys. How do I get to

the console? I don't have a username password, but there's a get federated URL that you can generate from temporary tokens. Standard stuff documented. It's for a federated system that gives you a token. And of course, you also meant to get access. So, I do that and boom, I end up with this pivot, which is the key. You still don't know if you're on the right track, but it's the only potential avenue you might have as an attacker slash player. And here's where I'm going to hand it over to Chris. But essentially, we now have an idea that cloud shell might be involved. It's not API enabled. For me to even check it, I have to get to console and boom, I can

generate the URL that I can plug into a browser and hop into as that identity small fill web roll, which is the EC2 instance ro into the browser. And there boom is where Chris is at. >> All right. So, we took that federated URL, popped it into the browser. The CTF continues. So, we know we're here in US- West one. Uh, as a CTF player, I would poke around in this environment. It's a just a really basic vanilla Linux environment with some help helper utilities. Uh, but I'm not finding anything in US- West one. Uh, you know, based on the cloud shell uh CTF, I can see that the AWS environment's kind of over Western Europe here. So, maybe I'm

in the wrong region. So, I'll just switch over from US West one over to looks like it's right about London. So, I'll pop into the London environment. Gets me into a new cloud shell here. When this loads, hopefully this will show me some information about where I want to be and what I'd like to uh get into here. So, just preparing the terminal and bam, I already start seeing some weirdness here. So, I think I'm kind of on the right track as far as where this cloud shell could be. It's doing a file system check error. And so I'm just going to see if I can do a little bit of recon here. And for uh the

purposes of the magic of the presentation, I will run this demo. >> Just want to say we have no API. So enumerating and even knowing every cloud shell environment which is by region is sort of a pain in the ass, right? You have 13, 15, 17 regions by default. You're going to go click and you're not even know what you're going to look for. In this case, we're looking for a flag, but let's say you were after a target. What each one you're going to come into the browser. That was part of the CTF challenges just to see how players would figure it out. So, now I'm in and I'm I'm just doing initial reconnaissance.

I'm in the home directory. I already see weirdness. All of my files are root. I'm locked out. That's that's not great. I can see that someone has tampered with this environment. So, not a big deal. I can just get up to pseudo. I know these things have root, so I'm going to do that. So, I'm going to check for pseudo and root. And as soon as we pipe in here, that should get me where I need to go. I'm locked out of root, too. This is a bad day for me as a CTF player. Uh, not a big deal. Uh, I can pivot onto the history. I can see what the other players have done. I can see what the

admin did. Let me check the history. History's blank. I'm really not having a good day here. I'm wasted hours now. I'm finally in the cloud shell. I'm not getting anywhere. Okay, that's fine. I can preposition some infill and xfill tools like like curl wget. That's that'll let me get to where I need to go here. Curl's locked out. Uh wget, which comes with the system locked out. All right. If my infill and xfill is done, maybe I go through an RPM package because I know this is a like a Red Hat based Linux system. So I'll do a DNF install. Get some Recon tooling in here. That's locked out. All right. So now I need to rethink of where I'm going to to

do. So I know as a you know as a red hat you know red teamer uh I know that there are persistence hooks in bash rc pretty commonly. So it's not a lot of people think to look inside their bash rc files for persistence mechanisms. So I'm going to see if someone put something in there perhaps. So I'm going to check and see what my just the tail end of this bash rc. And there we go. I'm starting to see some weirdness in here. There's you know fish the fish shell.com. There's some bash rc stuff. I'm going to take a look at that first one here. So that first bash file, if I take a peek, uh, it's

this conf.d bash. That's like a a binary from bash. That's weird. I know bash is text. Why is it giving me a binary? Thankfully, there's one other file in there that I can check, which is that other bash file. So, I'm just going to take a quick peek at what that is. Hopefully, that's not another binary. And there we have it. So, I can see a few things in here. One is this regular expression for four digits. It kind of reminds me of an IP address. I look a little more in this bash script. I can see it's asking for an IP and a port. It's asking me to put something in the temp folder. All right, I think I'm

getting somewhere finally now. So, I'm going to create that temp file just like that bash expects me to do. I know that whoever created this environment expected someone to be root. I know ports less than 1024. I'm locked out of those lower ports require root access. So I just I'm going to start at 1024 and maybe start banging on these ports as I go up. And I'll try hitting the local the local environment here at the home IP. So I'm just going to go ahead and set up this listener here. Uh what I noticed in this directory there's also a netcat listener. So I'm just going to turn that listener on on that port. And if I go

into another environment to try and trigger that bash RC enablement again, there it is. There's the flag. I I got it, Jenko. [laughter] So, we'll head on back to our slides here. >> So, that was the 10-minute compression of probably 10 hours for most of the teams that took a hard crack at it. And there was a whole bunch of lockdown in the environment that Chris hit because by by default you have pseudo, right? But this is shared environment. You don't want players mocking with each other. They always do, right? Steal it, get the flag, prevent anyone else. You go through all of this, right? This was a challenge to play with a non-managed environment. These were some

of the things that were done on the Linux side just for us, you know, Linux gray beards want to nerd out. It was super hard to lock this out. Mr. Meows broke it. Um, but everyone else couldn't break out. You couldn't download tooling. You couldn't get root. You couldn't change anything. You were just browsing. That's pretty hard for an attacker, right? Or a player in the CTF. And you had to sort of go by your wits. Obviously, we had to show enough like this fake netcat xfill little for them to get the flag. But if you look at it tight, it echoes a string get secrets into netcat to send off to your listener. So, by itself, it looks like

it does nothing. But compiled into the netcat special on there is the flag and a few other goodies made it real hard for you to try to reverse it. You had to take a leap and just say, "What the hell? Let's just try to trigger it and see what comes out the other end." So, we thought that was fun. I think most of the players did. And of course, this is sort of the solution chain or the attack chain, right? This is the stuff we went through. SSRF, get the token, do a bunch of useless discovery, hopefully light bulb comes on, see cloud shell perms, pivot to console, get to somehow the region. We did leave clues.

Once you get to the right region, you get a message of the day that gives you a clue. And then you have to probe and see would you check your home directory? That's the thing that persists in cloud shell, your home directory. Blah blah blah blah over resets of the environment and then finally figure out, oh, something looks like an Xfill and I'll get the flag. But you can see this picture led us as researchers to think about well what about this cloud shell? It's not just for a haha fun game. It's unmanaged non-API. A bunch of stuff goes with it that normally people don't think about because it's off everyone's radar. But if we start to dig and say, "Hey,

maybe we can turn it into an API. Let's watch the traffic from browser to backend." And this is sort of what you get. This is my proxy dump. And you start to see stuff that tickles sort of, you know, your Spidey sense that there might be more interesting here from a security viewpoint. You see create sessions, redeeming of code, put credentials, something with authentication. You know what you get when you log into a cloud shell for free. Your credentials are bound to that environment. You can do an API call right away. You don't have to set environment variables. You don't have to do anything. That binding is tied to some of this private API call. You see a

websocket down towards the bottom the protocol websocket setup. That's how the terminal characters work going back and forth. >> It uses a XtermJS. I think >> XtermJS when you reverse engineering it. It's the same stuff they do with SSM. SSM can do a terminal to an EC2 or container back end. Makes sense. If I'm at AWS, why don't I use that for this? And that's exactly what you find when you reverse it. Heartbeats because there's an idle time where the thing will will recycle. Uh so now we wanted to say after we developed a private AP API call started looking what at the protocol and the behavior a couple things popped out over the first couple

days and weeks stuff with file transfer environments and sessions. So we wanted to take you down the path of uh what started to bother us as security researchers. So file transfer let's talk about really quickly. Okay. What becomes apparent is when you do it through the browser. Browser has a file upload download to in cloud shell little drop down. What actually happens? You look at the protocol. There's a two-step process goes through a bucket, >> not your bucket. >> It's a globally accessible bucket >> owned by Amazon. They use their pre-signed URL feature, which if you're not familiar with it, it's just a shared link time expiration. um you have to have the right permissions to set it up,

but it's good for that time permission. You share it, anyone can do it, you can download it, right? It's got embedded access. So, whatever it is, it's it's good for the use case of upload a bunch of content somewhere, you know, videos, share that, be able to download and upload. So, you get two step one, what happens is the browser after you click upload, you get two pre-signed URLs. The browser uses one to upload to the Amazon bucket. And then janky step three. Over the websocket come these characters including turn off history and execute a curl command over the websocket. Think terminal characters. On the back end of the cloud shell, it executes the history

file is turned off so it doesn't enter the history of the real user. A curl command uses the other URL to download the same file. Two hop step. So, nothing wrong just yet other than this is really sketchy because it's a two-part process. What does GCP do? They have an SSH connection. They do an SCP. What does Azure do with their cloud shell? They've got the websocket. They send the file directly streaming client to B. What does AWS do up to a bucket down to the environment and I'm going to trigger it with a curl command sent there. So Chris, anything that pops up right away besides it's a sort of weird implementation, the more complicated it

is, the more, you know, smoke there is for abuse. >> Yeah, complexity, you know, increases or decreases security in a way. And that curl command, even though we haven't been able to muck with it too much with Unix, you know, in Linux based environments, you could alias that to just about anything else. And so that's an area for f future research here, but the way that this has been implemented just raises a few questions. Leave some smoke in the air there. We all know as researchers when you see the messy implementation or it's multi-art you have more things to poke at and break and inject and insert. So one easy thing as Chris pointed out you could intercept

the curl command once you have control of the command many ways it's not even prefixed with full path but even it was we have root we can lock people out of root we can replace it every time someone uploads a real user after we've been in their environment we can have a hook to xfill that pre-signed URL to us and we can get a copy of every file ever uploaded uh to to an environment right that's the kind of stuff that opens Uh now you can weigh that as what value as a threat actor might have. It's just comes though from exposing the guts, right? It's the Rube Goldberg machine is not only there, but it's behind a glass pane window and

you can see it, right? And it's the same thing a QA person sees when they look at source code. They're like, "Oh, that's how you did it." Right? Or any person scanning source code for bones. And so at the protocol level, this backs up the picture. You just see a bunch of stuff fly over the wire and in the websocket and the bottom shows a hist file command. The foodshell is just the file. There's an exit code, a janky way to get the exit code. There's a curl command and a pre-signed URL is passed there. Right? So that was observation one. You start to see a lot of messy stuff underneath the hood of this pseudo kind

of interactive console only service. Now, environments is a good question and I'm going to let Chris show you something, but I'm going to qualify. What the heck do I mean? I mean, the cloud shell environment is a running container. That's a resource. It's got disk, persistent disk, and it's got some network. Where does that run? Well, we just said the file transfer uses a bucket that's not in our users account. AWS pays for it. When you look at cloud shell, they they say up front you generally get free CPU you get your disk for free but it's limited but the it resets on you the container. The question is in AWS you start this up. Where's it run?

Well, one of it is is also how many of them do you get? Well, by region that's answer number one. You're in US West one. Your cloud shell environment is completely different than in a different region. And you can check the IP address and you can check other things. Okay. Now what we're going to show is something that is not obvious. So this is AWS specific but every environment you have to think about how many do you get kind of thing and where do they run? That's the question of this resource that you generally don't have to pay for. That's an interesting question. Don't you agree? Do you get one? Well in AWS you get 15ish per region. What if

you could harness it with an API? Well, if you're a good guy, you could still harness it. There's one guy out there who wrote a virtual file system across all your regions. All the one gig persistent things are now 15 gigs. That's pretty cool. If you're a thread actor, well, you got 15 from one cracked password or stolen session. That's not bad. You want to think about these things. Of course, no one did because this is an API managed service. It's off the radar, but we do. So Chris, what should we show to about environments? >> Yeah. So the first thing is we know that this is running through websockets. And so what we're able to do is take those

private calls uh take, you know, come over the websocket address uh and start communicating with this outside of the browser environment. So we're already now doing something that wasn't really meant to be done. So we're making, you know, the youname-r command. I can see we're in an Amazon Linux instance. Uh I'm still the cloud shell user. I'm in my cloud shell environment. So what's really cool here is this is outside of the purview of the API. So APIs are monitored, they're build, we're we're getting around that now. And so we're outside of that whole API call thing outside of the browser through a utility and we can just communicate back and forth with that cloud shell. So, because

I'm not sure everyone saw this super small type as we spun it up, this is our little reverse engineering of the API with our own command line utility to set up a connection to cloud shell given some credentials instead of being in the browser with their hokey terminal xtermjs. We've got a way once we have credentials to connect and have command line to the same container environment. This is the start of us dreaming bigger um as researchers of what we can do with this. And >> so now head on back. >> So this is this is that that new named role. So this is the same utility but now we're coming in as a a different

named entity here. >> This is a little deep on the AWS side. So I got to spend a minute here. What is an identity in AWS? Okay. Well, AWS did this thing where they said a role is an identity basically. Most other say those are just groups of permissions, but your user account is your identity. So you're a user account, you're an identity in AWS, but you can assume a role and you're a different identity. That's your AR. Okay, everyone sort of gets that if you're in AWS. It's just a weird thing. Ro is an identity gets logged. trace it because when you assume a role it might be I'm now an admin for a certain period of time you

want you want to track that okay but when you assume the role when you're in control and actually doing that API command you pass in a little tag that's the role session name that's the TA okay when you control it an EC2 instance when we attacked it does that and it uses the EC2 instance ID automatically we don't get to control it it's just that's how it gets its identity okay so where does this It means usually doesn't make much of a difference. You get logged and it's good way to track identity. There's ways to set up policies so you're forced to put your username there. Blah blah blah. It makes a difference when you have a

resource that's tied to your identity. What is that? Cloud shell is one of those. When you're a different user in a a certain region, you get your cloud shell. Different region get cloud shell. When you're a RO in a region, you get cloud shell, but it depends on the role session name. So if you have control over that, open your minds a bit. You could pick any string. How many cloud shells can I if I can assume a role that has cloud shell permissions? Of course, how many cloud shell environments can I create? I think it is infinite until we hit a boundary with AWS. Now when someone else is paying for it and you can create infinite there's some

kind of opportunity there that impacts us all. That's what we're saying. The number of environments who pays for them and how many you can do with one identity makes a difference. Want free you need more than one gig? Well just if you can assume a role I mean go try this yourself. You assume a role in your own account. Put in a different tag name. you get to this picture. The top is what we think about in AWS. Identity per region. I got say give or take 15 resources that are different by region like 15 cloud shells. Great. Normally you'd think, oh yeah, rolls the same thing. It isn't. And there's also a federation token I

can do as a user. Both of them allow me to give a certain kind of name becomes part of my AR gets logged with whatever API calls I do. Oh, that's great. And then there's a thing I can also create cloud shells. And so n * 15 where n is unbounded as of today until we hit some limit. I'm sure there's a limit. It's a lot bigger than 15. And all I'm saying is one identity. Now you might think this all makes sense, right? If you're an admin and you can assume role, you're basically saying that admin can assume many identities. Okay? Right? Just like an admin could create a bunch of user accounts. Yeah.

sort of makes sense from a design. But what where it doesn't make sense or at least where it isn't thought about is did anyone think about that combined with cloud shell which has compute and disk and some networking? I don't think so because this is not never been part of the normal API services. This is where two things sort of innocuous by themselves put together become oh that's interesting right? Okay, so that was the second thing that just seemed wow that's interesting. I wonder what you could do. I mean, if you're into Bitcoin mining, not that I suggest it, I'm just saying, right? Um, as a blue hat who is following the lawn ethically in all licensing, I mean, you could

still think about workloads or disc storage or whatever you want that you would normally do with your cloud shell environments, but instead of creating a thousand users, maybe it's only one user, right? Okay, I think we need to move quickly to our third point. >> Yep. >> I mean, we we're running out of time here. Let's talk about sessions. End sessions. What does that mean? Let's talk about the mess that is not only AWS's mess. I know we're starting with them making them look bad. It's everyone's mess. The number of sessions and what they mean is a nightmare. Normally, everyone gets the idea that you usually have a console session and it's different from your API sessions.

When you get to things like revoking access, you got to be careful, right? Okay. We also know or maybe that there are two different types of session keys, permanent keys and temporary tokens in AWS. That's pretty wellnown. People also understand that when you create temporary tokens, you often assume a role. That's often a set of API calls and if they're chained, if you can reassume roles, you can create more sessions and they have different identities. Pointing out there's token sprawl and all those tokens generally mean some kind of session and access. Okay, it's point out that the number of sessions is non-trivial because what's in red are things that aren't thought about. Actually, identity center is also a way

to move from a console session to an API keys. It's I think the only way in AWS that happens. What I mean is I get console access. If you got permanent keys set up there, I don't see them. You only see them when you create them. Got to download them. Then they're offiscated. So just because I get console access to this gentleman here doesn't mean I get his API keys, right? Maybe I can create new ones, but I can't see existing ones. at any center dashboard. You pop in, you hit access keys and up pops full session keys because they wanted to bring the CLI under SSO. Dumb idea, but they wanted to and they mix the two, meaning you can

move between the two. That's pathways between different kinds of sessions is bad for most security people. The things in red, I get a permanent key. I can mint and get a same identity and permissions and I get a temporary token. I can also if I have permissions to get federated token, I can also do the same thing. The get federated token can do that trick of getting a federated URL. Meaning I can go back and get console access. That's why there's sort of a line back there. Assume role. If I control the assume ro or any I have an any token that is from an assume role, I can also do the federated URL trick. We

we did that in the CTF demo, right? That's all standard. But this all leads to do you know how many session tokens and sessions you have at any one point in time especially if you have a compromised identity I think it's non-trivial are they in the logs of course in your middle of incident are you sure you have everything covered and then with cloud shell there's the bottom left which is there's another session it's the websocket jigory do it's another session okay so why do we care um I think we're going to flip You showing it? >> Yeah. And we can re we need to uh I think we need to kill that access too.

>> Oh yeah, we have to uh show. Okay. So we're going to show a revoke. This is one of the cases where it bites you as a blue hat is the number of sessions, the relationship and whether they can move between the two. You've got an instant. You identified that there's compromise. Time to revoke it. Okay. So if we go back to our case, um this is where in the CTF we have, um I'm going to go to the actual account where we had that small fill web roll. Okay. And we have that still running over here. Yes, this is this is from the CTF, the same role that the EC2 instance was. Let's go ree revoke it. Let's just say

let me get out of this. I actually don't want it. Okay. As an admin, the way I do that in AWS, I go to IM. I find the role small fill web role. We've suspect it's been compromised. This is the story. It's real life. We're in an incident incident, right? Um I'm looking for this guy. I come in here. It's an identity. I come here. AWS makes it easy. I can revoke active sessions. I'm going to hit the button and it should revoke current access. Okay, tokens and browser. This actually does work. You should just note it's messy. It's a band-aid. It doesn't actually kill the token. It puts in a deny policy right there based on timestamp of the

creation of the token. It means at this point in time, I've been compromised. It's in the past. It has a token created in the past. I can put in policy right now that says anything and it will be checked. It's creation time. I'll reject it through a policy. So when it does an API call, I will reject it. It will work, but the token lives. Just so you know, if I put this in place and then remove it and that token's still alive, the access is restored. It's part of that messy part that uh I'm sure all secops guys appreciate is the implementation makes a difference. It'll work, but you got to be really careful.

I acknowledge things. Let's do it. Let's do it. And then with Chris's help, we'll go back and see if it was actually revoked in the browser. Do you want to see that first? Where's our Is it over there? This was the real And if we Okay, this is to show that the real user thread actor, whoever's running did get revoked. It worked. Not saying didn't work, but Chris also has our friendly utility which had its own session [snorts] and it's still we're still there. Well, okay, maybe we got a little bug >> in our demo. >> The point >> theoretically it should still work. We should still have access because this is outside of the API uh blocking here. But

uh yep, >> it's good. The the point is demo gods not shining upon us right now is the websocket does persist beyond the revoke because it's it's not checked against the token. It's just sitting there. That's the whole now API calls won't work. Okay, but my CPU, network, and storage is still there. Now the container will reset. There's a hard reset every 12 hours even if I'm active. So you could say, hey, it's temporary. Worst case, you got free reign to stuff outside my AM boundaries. That's the way to make the problem go away so you don't have to do work as Blue Hat or AWS. The kicker though is I can survive a reset in the CTF. I can hook into the

log the login scripts. When the user relogs in, they start up the environment and I can xfill creds. I can get live tokens and I'm back in. So the temporary stuff, the reset doesn't stop me from regaining access even if you cleanly revoke things. Also, in the 12-h hour period, I can my bots can still run and I can still mine some Bitcoin or whatever the highest value, you know, crypto is, but then I can survive the reset and I'm back in business. And it doesn't matter if a couple of these go offline and don't come back because I've got a thousand of them running and then over time I have, you know, 60 70% kind of uptime. The

point is it depends on your perspective of this. Is this usable? From the red team, it still is that little thing. From a blue team side, you might feel safe like I don't care. Use my CPU that Amazon's paying for. Use my networking blah blah blah. And that's the nanny cam view from uh you know personal people who get their nanny cams compromised so what don't care you're not in my network da da da maybe you care about your pictures of your baby but you know the point being you can turn a blind eye when you don't think of one target but you're aiding embedding creating embedding further abuse this is still resources that came in through your

vector compromised identity so that becomes the crazy thing about yes you have crazy numbers sessions and one of those sessions happens to be a websocket to the container. Boom. That can survive the right setup. So, let's talk for a minute on GCP, Azure, some other things we found and see if we can't get to >> Yep. >> time to ruminate. >> Yeah, I'll just kind of touch on this very briefly. So, it's not just AWS that has these cloud shell environments. GCP has them, Azure has them. uh and they're similar in form and function. With the GCP cloud shell, something interesting we found is you can get to that cloud shell with a free Gmail account. So, we

created just a you know fake email account and within that we just were able to get straight into the cloud shell environment. Um that opens up some interesting possibilities as far as access and and where you can go from there. The second uh it uses an Ubuntu DRO under the hood. So Ubuntu Linux when you pop on in it's in a containerized environment but you can enumerate all the packages on that system and those packages come out at different days and times they might have updates whenever and so they I think they're downstream of the vendor updates. So I don't know when they backport fixes to packages there's going to be a delay between that. So if there's a known

vulnerability in a package there's going to be a delay in getting that backported. That's an opportunity for an attacker there. The last is it has a /google mount point inside of the container uh that gets loaded in every time it runs. Uh it contains a bunch of helper scripts and credential scripts and it's just full of all chalk full of all good stuff that we can poke into. So that's kind of another area of future research for digging in and and poking around in the GCP environment and I'll I'll let Jenko talk about >> a minute on Azure. So a lot of similar points free email accounts opens up attack surfaces. It's the vendors pushing free cloud shell even for you

know consumers and stuff gets really messy but the messiest part of Azure Entra no surprise to anyone who deals with that cloud is the permission scheme is a mess and it translates to the number of logs the identities you have service principles you have ooth you have user there are not explicit permissioning for cloud shell in Azure it translates to other you know graph call here this that the other by default default. It's also you don't have to have many permissions to start up a cloud shell. At least in AWS, you have sort of a start cloud shell permission, right? But in ent it it just makes the job harder of knowing what the heck's enabled. It's enabled

easily by default. I wouldn't say completely easily. Persistence is an explicit storage account. That's a good measure actually. You need permission to attach or select a a storage account for persistence. So you have to have a subscription for billing set up. So there's a few things a thread actor would have to do. Uh but one of the things they don't have to do is worry about permissioning. Oh, have I been granted permissions? This identity I stole. Okay. And then what it translates nuts and bolts is if you have Azure CLI permissions into cloud, which is common because that's what that tool does. If I've stolen that token, I can do cloud shell. that marries two interesting

abuse areas. The cloud shell stuff being one that we just talked about, but also Azure CLI opens up all of this OOTH device code abuse that has been part of Storm 2372 a year ago, five years of OOTH fishing that's been documented by researchers, Dr. Azure AD included. Now we have good social engineering attack around oath tokens to get an Azure CLI and then I can do things like the normal abuse of all your apps the 30 SSO and then I can also play on the edges with cloud shell which no one's thinking about that just makes an endto-end attack really really really easy or at least um exploitable in uh Entra just adds to the mix of what's

already there right not telling yanking new we can still get Azure today but who's thinking about Azure cloud shell as part of that blast radius so that is that we we have a bunch of things that we probably don't have to every shell has its own quirks and that's just it's like green field territory because it's no one's on it's not anyone's radar because until you have a private API you don't start exploring this no one wants to poke around in a UI and and try to figure things out you won't even see the picture so this is just hey scope permissions, runtime, reset, logging. It there's all these nuances that we would normally look at hard if it was a normal

API service. >> Yeah. And I I won't dwell on the red team stuff here. We've kind of covered a bit of this, but because there's unlimited free compute and unlimited free networking as a red teamer, this lets you put, you know, CPU intensive malware on there. You can run a crypto miner for networking. Uh you can use that for different botnet stuff. You can use it for uh uh for proxies. uh basically unlimited ability to pop out of different places. Uh here on the right we've got a reverse shell. So on the cloud shell at the top we've got netcat piping out to an attacker controlled box and we're running commands remotely through a reverse

shell. So not to the other thing too is the CTF shows that an attacker can also lock out other people and admins from the shell. So just some brief highle points on some attacker uh tradecraft here. >> Yeah. And if you search for cloud shell abuse, you find like one or two articles a year over the last five years. But you find container escapes CDs in the last year depending on on which one the persistence that has to be defined. There's a whole bunch of things. Do you mean persistence of access? Do you mean sort of locking the user out and keeping your stuff in their cloud shell? That just means there's more use cases. I

pointed out one or two things in the footnotes. Networking one. Uh, someone did a nice experiment. Really good code. Shards your files, stores them across your regions in your cloud shell. Virtual file system. Pretty nifty stuff. Really good coding. The self-healing one is pretty ne new. These are all in the last month or two. Not yet. Is an open source tool uh by some of the guys uh Edward uh at Offense AI. Basically, he noticed there's a 4-se secondond latency from revoke time to when it gets distributed globally across the endpoints at AWS. So, he can detect revoke, but he still has a time to you put in a deny policy. He's able to remove it if you had permissions to

remove your own policies like admin permissions. It's basically a very robust self-healing implant, if you will. Now, it's noisy because it'll be doing a whole bunch of things with the API once it detects someone's trying to to revoke it or take it away. And it's crazy how much just that one technique, which I call self-healing, is in there. Imagine this is part of the mix of what you do when you actually gain control of environment. Pretty pretty interesting areas in terms of what that means or implies about abuse and ability to protect your access again whether you're a red teamer or you you can think about it from a blue team perspective which let's take two minutes there and try to give

you time for some some questions. Um so look if you're a blue teamer or just put on your blue team hat after you think about that and we're just scratching the surface. This is the 1% of the iceberg, right? I mean, this is like four weeks of work between Chris and I with some, you know, vibecoded nightmares along the way, right? So, one, you could just deny all like with anything. Like, I don't need this crap. Let's just take away all permissioning. And that's valid, right? You should do that with everything. It's just you probably wouldn't have thought about this as this little interactive freebie thing that's supposed to reset every so often. You could try to jump on

the wagon and control it, manage it, protect it, and obviously you could scrap it, but actually try to have a replacement. So that's easy to say these little catchphrases. If we dig into it, there's the rigor that a blue team operator would have to think about it. You you probably start with uh do you even know how much cloud shell is being used? You don't have an API. How do you discover this? Uh ask I mean you could look at the logs. a creation of environment. So you could comb your logs and tab tally it up. That would be one way. The tool, this is an output of one of our little scripts using a private

API. Well, you could enumerate. That would be a little bit easier. You'd have to figure out how to become every identity in in the system. There's a little impersonation problem there. But but again, it's within reach for you to think about. Okay. You could prevent prevention controls. The good thing is AWS GCP, you do have permissions you could lock down. You you can do it in in Azure. It'll be harder. A AWS is is a little bit cleaner. It's it's the start environment, create environment, lock it down. That's good. This positive news, right? Um from a logging perspective, okay, they're private API calls, but you do get bunch of things logged. So, you can have detections off of sort of uh

suspicious activity. I don't expect this user to create this cloud shell. I definitely don't expect it to be in, you know, whatever Asia, some region I I never do things in. Right? you have you have a fighting chance to detect usage or past usage lead you to auditing um for anomalies bursts volume bursts will be suspicious thread actors will probably be noisy if they ever jump on this right they'll be noisy until they need to be stealthy so you have a chance it just means more work for the blue teamer um luckily we have you know the rage of the moment of maybe I can throw some LLM coding at it and just try to

make it pass by the bosses because it's not really production stuff. Hey, I just run it every so often. So, yeah, the usual stuff. I don't believe that either, but the usual stuff um would apply. You have to go over remediation playbooks because this isn't even in your playbook because it never was a thing. So, um if you knew where they were running, if you're revoking access, do you know what else to clean up like deletion of the environment, which really means the container and the persistent disc and all that? Well, you'd want to cover it. How to do it? That's a question mark, right? you can go back and see if well you can API

enable it and then you could probably cobble something together whether it's you know rocket proof would still be open and if you're on the manage it improve it manage it bandwagon you could say no I'm going to use the API for positive and good I'm going to install agents to monitor I'm going to lock down my users from mcking with it I'm going to do a whole bunch of things including locking out thread actors it is more work do your own but it does become possible if you believe you could and wanted to jump on that bandwagon. So there's a whole bunch of variations. Cloud shells can run in a random environment. You can do it in your

control VPC, right? You look at all these things. Um, and sort of on the end note, we'll let Freddy talk for two seconds. Where do you go from here is sort of the question. So besides making slides available, we do have some tooling though I'm gonna be upfront. Chris and I have thought about we're we're gonna slow roll it partly because it's vibe coded nightmare stuff two you can do it you can reverse engineer the protocol and build tools to get working examples really easily today that's partly what we did to aid our research it I think we have an ethical responsibility basically before we put crap out there is this going to help or not so one

we're going to share our resource research we will get stuff up on sort of the GitHub link, but we're being very cognizant of the last thing we want is someone to then throw it into some models and then it's only going to benefit threat actors, right? You can think the defensive side is much much much harder. Get some pseudo working stuff so I can own a cloud shell environment is really really really easy. You can do it today with off offtheshelf. So the point here is I think outside of all the cloud shell abuse I think we all in research community wherever you lie you should be thinking about where does the fact of our stuff go here raising the issue

helps not sure throwing tooling halfbaked tooling out there is is helpful because you just get a bunch of script kitties research kitties doesn't really matter um I think it's more thoughtful about how do we deal with these kinds of problems going forward so love to share share. Anyone who approaches us, we'll share full full site, but I'm not here to put a public repo just yet. Um, a bunch of stuff that people can feed into LMS and turn out a >> Well, no, we're very grateful for you guys coming today. You could have been anywhere and we're very thankful that you're here with us today. Thank you so much. [applause]

>> Top of the hour, so if we get booted, uh, please reach out to us, find us. We'll be around all weekend. Love to chat anything that interests you or or sort of questions or or so on. Thanks,