BSidesSF 2026 - CloudShell Hide-n-Seek: enjoying the sweet persistent... (Jenko Hwong, Chris Ryan)

with a talk called CloudShell hideand seek enjoying the sweet persistent sounds of silence. Please welcome principal security researchers at Huntress Labs, Chris Ryan and Jenko Hang. >> Thank you, Isabella. And since some of you future viewers will be on YouTube, we got to describe how packed the theater is. Standing room only. We got about 300 people in the AMC metro uh you know metron especially uh bosses who paid for this. Uh it's awesome. Well worth it. I'm Jenko Chris Ryan. We do security stuff. You can read our bios but you can quickly pick up that the serious one is uh Chris cuz he has no social media, right? So he's the one who has no stuff. You'll find out during the

presentation. I'm the ranter. So if I get excited about the topic, it's cuz I'm passionate. I drop a few fbombs aimed at three-letter companies in the cloud. Maybe fiveletter companies, so it'll be all good. All right. Uh, we can't see you too well, but uh, don't fall asleep. All right. I mean, this is weird. It's like very dark and dim. Looks like an, you know, feral theater. A little more like a porn theater. I'm not sure what's going on here, but we're going to make this entertaining while we talk about persistence and abuse in the cloud. Okay, Chris, you got anything to add before we jump right in? >> Let's dive in. Let's do it. Okay. So, as

you might intuit it, we're going to head to cloud shell. But before we do that, I want to tell you a little bit of a story. Want to go through it. So, we're going to end up and deal with cloud shells, which are compute environments aimed at interactive use, usually by admins, right? All the cloud providers have it, but it's unmanaged. It's not API wrapped. And the story of this, we'll go through all of this stuff, but we want to take you through the beginning, which is a CTF uh that I built last summer for Cloud Village at Defcon. And so we want to take you through that experience and uh a lot of things that came up from that. And the

the the idea in Genesis was why don't we screw with the participants and put the flag in the cloud shell environment because everyone's tooling isn't going to get at it, right? you can't make API calls to even get there. So that was the genesis was a screw you attitude by me. So come and we'll interject a little bit of the builder perspective as we go. Chris is going to play the participant. We're going to compress into like five minutes playing a CTF and then we'll launch into the real world aspects of what's this mean for red teaming and blue teaming. So just to set the stage and we're going to jump into demo. So, we're going to

get off this static graphic and I'm going to help you guys out. Remember my role. I'm the builder, but that's closer to blue hat. I'm going to turn it over to Chris. We've got a Mission Impossible agent theme. This was actually a multi cloud challenge. All right. This was a graphic and entry point was a website to all the participants. I'm going to let Chris jump into it. Okay. But basically, this was the starting point of the challenge was, hey, secret agents penetrating the syndicate headquarters. They might have three headquarters in the cloud. The agents disappeared. You're a follow-on agent. You got to figure out what happened. Okay, Chris. I'm going to turn it over to you.

>> Jenko's about to torture me here. So, let's see how this goes. Uh, this will be a live demo, so hopefully the demo gods are with us today. >> Yep. Go big or go home. All right. This is could be, you know, hurting our reputations here. Let's see. Let's go. >> So, I've got a bit of a cheat sheet so everyone can follow along at home so you know what I'm doing. There's no handwaving here, but um let's get started. So, we know a few things right off the bat from the C2. We know we've been given this this URL to help us. So, let's navigate to this URL here. >> Yeah. So, this is the entry point for

all the participants and they come in. >> And so, Chris, I'm real curious since you're the red hat here and the participant. We always wonder what's in your mind as you come and solve this. What What's your sort of first move here? I love websites. Uh, so the first thing I'm going to look at is just what's on the page here. What can I see? Uh, not a lot of stuff going on. I've got, you know, a menu. Maybe it doesn't give me a lot to go on. There's an image. Um, I might look into steganography. Did you Is this something I should >> No, no. My challenge is although some people ask participants, I don't like

stego stuff. So, there's imagery. Although some participants did feel compelled to look at that and every asset on the web page. But no, this was a front end to make the okay a cloud challenge, you know, entry point. Go ahead. >> So, I've now just wasted two hours trying to do cool Stego stuff. Um, but we now know there's a couple other things we can look at. All websites have code. I mean, we can take a look at the code back here. But before we do that, the other thing we've got up here is this URL. Uh, I pasted in just a a regular, you know, URL. But now we've got all these parameters here. Um, if we

know anything about these parameters, we can start messing with these things. We we've got what looks to be a string at the end. Uh you know an N and you know right after the question mark the all famous U. Uh I'm going to guess that's a user. Uh and so you know going through regular ID door practices here. I'm going to just change that and just see what it comes back. Let's be curious. All right. Uh so now now we're on to something. Jenko. >> Yeah. Well, you know, clearly the uh web page changed. Uh I wonder what you found with uh that there. But ID door was introduced just as a cheap way to to

warm up the the participants. Uh but they don't have initial access yet into the cloud environment. Uh so what's what's behind the cassette? You want to you want to click on that? >> Looks like looks like our secret agent led us something here. >> Took took a lot for my poor coding skills. >> Nova, this is agent Pettit. If you haven't heard from me, I'm likely dead. >> Okay, we can cut that. It's basically some clues, right? Pet it's a play you know not everyone appreciates it when we build we try to build in ponds there's a tightroppe walker Philippe petite in the 70s walked between the twin towers right so I was playing off that graphic this

is just playing into hey I I I found some bottom codes to diffuse some bombs and threat but I'm I'm in the syndicate so follow along so Chris that doesn't necessarily give you anything what what are you doing from a pen testing red teaming perspective >> yeah so I I've got our clues and that that's really cool and fun But from a technical perspective, I'm I'm still dead in the water trying to figure out where to go. So, I'm going to go back to the code again, just like we talked about. So, uh, as we can see here up front, standard run-of-the-mill HTML, nothing too crazy. I'm looking for code loading. I'm looking for JavaScript. I'm

looking for active stuff. I'm just not seeing it here. Uh, you know, CSS looks pretty normal. Um, after that, I'm going to start looking for inputs. How can I start injecting things into this environment? And right here we see a form. That's perfect. That's what I need. And helpfully uh the engineer or person who coded this up said, "Hey, I'm going to comment this out, >> right? The engineer over here." So yeah, super obvious, right? Uh but but clearly Dom and various things will will give hints at uh the server side and the website. So again, the focus was not a web challenge. It was just to have a little bit of fun and get people warmed

up to get initial access. So therein is maybe hinting that there's not only an SSRF van that maybe there's a endpoint a submit endpoint for that input form that could be exploited to that end. Now we're on to something. Uh and as we can see here the action for this form is search. So going back to our handydandy cheat sheet to save time and the magic of the presentation. I'm just going to use this curl command to execute that SSRF. I'm hitting the IMDS service. Uh, and that's just this loop back. And then I'm hitting that endpoint here at search. So, let's see what that does for us. I I'm feeling good. Hopefully feeling good.

All right. Now, we're on to something. We got back a web roll. This small fill ro. That's something that I can work with. >> Yeah. So for anyone who's into AWS, but all the vendors do this, it's um how do you have identities especially on workloads like EC2s or compute instances. So they build in a metadata token service. It's available. Usually you don't need any secrets. If you're on the box, you can do a curl command. You get live API tokens depending on the cloud provider, right? AWS, we've known for what 20 years that they had an SSRF hole. They introduced V2. They don't require it. you have a lot of v1 still out there. So the idea is can you

remotely through an SSRF in the web service web app have the code of the web app execute that local uh call to the the loop back address, right? Classic stuff. Everyone in AWS knows about it. Chris is about to exploit it. What happens? >> Yep. So I'm going to dump the creds here. And what that will do is I I just put them out to this uh web file. I I added that roll to that same curl command. So it's the same thing. I just added the roll to the end now. And now I've got those creds dumped to disk. So just to see what those look like. >> Yeah. And it's important to know the

context. We're in command line mode. Again, if you know AWS, this sort of makes sense. But the initial curl command is just our injection point to the SSR RF through the website. He he could have typed it into the form, right? In theory, there's some search functionality at that hinted as has a vulnerability that might effectively reflect the query payload the URL because it was really sloppy code, right? Obviously, if I compromised the box or Chris did, he would just do a curl command with the 169 direct. And the point is, Chris, what's this what we're looking at? >> Well, I've got some temporary credentials now. I've got my temporary token. So now I've just gone from a web

environment and now I'm looking at an AWS environment. This is making me very happy. >> Okay, so we have real API calls happening. Thanks for scrubbing it cuz this is you, you know, we we never know who's watching us, you know, from I don't know, North Korea or Russia or whatever in YouTube land. So it's all good to be safe, right? Uh but live accounts that we're doing the demo off of and now you have API tokens, Chris. So what's your next move? I mean, uh, sky's is the limit here. I can start poking around AWS. I can start running some AWS commands. And so, I'm going to try doing that. I'm just going to look

and see if I can enumerate maybe some some S3 buckets here and see what I can find. See if something comes back with those new creds. All right, I've got S3 buckets now from this temporary thing. So, I know at least for this role that I found, there's at least some permissions here. Um, and now we're going to spend the next 40 minutes of this presentation digging through S3 buckets. >> Yep. Thanks, Chris. And funny enough, participants did do this. So, from a builder's perspective, what's this? I wanted people to come into API land, which is the usual thing we think of the cloud. Find a bunch of resources, look at them, and of course, the flag's not

there. The flags in cloud shell land away from APIs. Now, then again, it's not fun to jerk people around. That's not a good challenge, right? So, you could spend all day enumerating and we get discord hate mail about are you sure the challenge is working? So, what Chris had found in reflecting real participants is I left some clues like metadata tags on buckets just saying yo the clue from the agent that in the story line decided not to hide the bomb codes here. It was too obvious to put it in S3 buckets. >> Yeah. >> That said, Chris, did you hit the calc that I unpacked on a bucket? That was a gem. I >> Yeah. Yeah. So the Cal one where you

have ASKI art and stuff. I did put a clue in there. Someone actually felt compelled to compile it, see if there was stuff in there. So really, even though it's a cloud challenge, we're going through discovery. And what amount of discovery is typical? So what what happened Chris? What what was sort of your your process after you you you maybe continued on? >> Yeah. Well, now that I'm 12 hours in and wasted part of my day, uh I have all these things that I tried. So to save you guys the pain, I'm, you know, seeing what EC2 instances are around here. So we're going to spend the next 40 minutes looking at EC2 instances or or maybe

not. But uh so I just took a look and said, what else can I find here? And I'm I'm not finding much. I'm I'm running these AWS discovery commands trying to see what's going on and I'm just not finding much. >> And I want to put in a little note here again for people who every IM is different in the cloud, right? But AWS is sort of notorious for their confusing overlay of policies. Actually, every everyone has their own. So you have different ways to attach permissions and policies to an identity, right? You have managed policies, inline policies. You have stuff at the org level. It's like as bad as, you know, Windows AD and and

group policy. You just actually don't know what you have. But the thread actor is enumerating that. So if they can read their own policies, you get to see your privileges. Do you have privac what resources? So as a builder I know that and I I purposely did not have this identity be able to introspect which is what you sort of see but all right so now that I've got to this point there's one last trick that I can use and that's I want to see what authorization details this account has. What what am I allowed to do? So, I'm to save you some time. I'm going to run that command here and I'm I'm again I'm looking for things

like what EC2 stuff can I get into? How can I poke around and I'm looking here and cloud shell? Yeah. >> Did you really do this? >> I did. Right. So, you didn't know the title of this presentation in the future and cloud walker might not have triggered you to cloud shell. So, yes. Now, the participants in a fair number eventually got here. Now, the funny thing is this was actually a mistake. I was initially going to block every hint of cloud shell permissions to that identity and see if anyone had a light bulb moment of saying I should check cloud shell even though it's in the browser only. It might not have been fun. That might

have been a fail as a challenge. So I left this loophole and and different participants uh sort of shared that that's how they actually saw that oh I have cloud shell permissions. That's a trigger of maybe I need to go explore that. Right. That's right. But from both a CTF perspective and real life, we'll talk about it. How you block enumeration for the thread actor is an important topic. Okay, Chris. So, what do you do with that now that you know that maybe I did something to, you know, f with you? >> Well, now I'm really angry because I've just wasted most of my day and now I'm dealing with a cloud shell environment. That is not a happy day for me. But it

gives me some direction of where I can go next. And that that clue to me means I know that cloud shell environments are browser-based environments. And the way that we get to doing that is through a federated URL. And this is a a really well doumented process. Uh there's a lot of stuff on the internet. So we will leave it as an exercise to the reader to to get that federated URL. But we have a handy dandy script here that does just that. >> And it uses the credentials we've just used. So this script that I'm running uses those live credentials right now. And this is worth a comment as as Chris switches over into the browser. The

comment is outside of CTF, we'll hit it later in the presentation. You're going from context of API secrets or tokens to browser access. If you think about any app or any cloud environment, those are usually firewalled and separate both directions. You steal someone's console access, they left, you know, no screen saver on, they went to lunch, you hop on, you don't get to see the secrets, right? You get them when you provision them, you download them, and then you never see them again. This direction is the opposite. You steal my API token or my session, you don't necessarily get to jump into a console session. It's one of the leakages that uh AWS did and it's

for their federated SSO model because sometimes you federate with another system and it's not browser sample based and then they wanted to give you a console into the AWS environment. You sort of understand why but it's insecure, right? And these are the things that get exploited and I wanted to put that into the challenge to start them in API land but give them a pathway to shift over into console land and the cloud shell. So AWS was rip for that. >> That's right. So now I'm in. Job's done. I did it, guys. We're in. The flag's got to be in here somewhere. So I'm I'm just going to start poking around. And you know, as I start digging in here, I'm

I'm not seeing the things that I would expect to see. I'm I'm not finding what's going on. I can poke around the environment here, but I'm I'm not getting what I need. I I'm getting frustrated. There's nothing in this. >> So So just a minute here, Chris, because I don't know that this is obvious. You jumped into a cloud shell. Maybe why I wasn't looking. You got a pseudo terminal hokey thing in the browser where you're on shell in a cloud shell environment. What? It's running Linux something. >> Yeah, we're in a full Linux environment here. >> AWS has this regional breakdown for a lot of resources. So what what's in the top right uh drop down is all the regions

you can switch to. So by default 1517 and AWS gives you a separate cloud shell environment if you have permissions. >> That's right. in each region. And you're in West One. Go California, >> right? >> That's right. >> Okay. I'm in US West one and I'm getting frustrated again because I'm not seeing the things that I need to look for in this cloud shell environment. I'm, you know, I'm starting to grip. I'm digging around. I'm running, you know, a find command for the things that I need and it's just not coming up. >> So, as a participant, you don't even know if you're close. >> Yeah. >> To and whether I even put a flag there.

I mean, you had permissions, but that's all you're going on. >> That's it. So, so what are you gonna do? >> I got to go back to basics. I have to go back to the original thing here. I'm I'm stuck again. So, I'm poking around and I'm I'm just going to start looking for clues here. And I mean, the best I can come up with at this point is I'm I'm seeing whoever came up with this has given us a map. Um I see we're here somewhere in Western Europe. AWS is parked right over Spain. You know, I know there's some big data centers out in Western Europe. So, that's my best hope. So, maybe I can start switching

around there. Okay. So, no APIs to help us enumerate or check things out. But worst case, you can brute discover the regions. And I'm thinking that's what I planned. I'm going to waste your time a little bit. But it's solvable even if you're pissed. >> So now, now I am really angry. I've been wasting a ton of time on this. And so now I know that I'm getting into uh EU West 2. I think this is the London data center. >> It is the London. Can you do a full screen on that? >> Yep. >> And what do you get? So now I'm starting to see things that are a little more interesting to me. I can see a file

system check. I'm seeing maybe what looks like a message of the day. This this looks weird. I think I'm getting a little closer to our agent buddy that left us some stuff here. >> So this is my hint, right? You don't want to jerk around people too much, but uh you don't want them coming back to play. So one of the things is the subdomain in that dump which is from a system file of of downloads and packages does give a clue that the agent in this story this is the environment where they hid something like the flag. What what did you just do there Chris? What was that? >> So I'm trying to run pseudo in my other

environment I had pseudo. I just got locked out of root. Whoever is controlling this Jenko uh has has blown away my root access. >> Yeah. So most cloud writers, all of them provide pseudo because this is geared towards admins being productive. You own it. It's ephemeral. This thing resets after time, right? So I couldn't have that because participants will stomp on them. First one in finds a flag and blows crap away because they're root and that wrecks the challenge. That's right. >> So, I had to work hard to remove pseudo and do some heavy in your area some old Linux gray bearding nasty stuff. >> Took me out of the clouds real hard to do this lockdown.

>> You did admirably, Jenko. It's good. And uh I Yeah, I'm now I'm getting frustrated. I I don't have my root access. I'm in what looks to be my home directory. I you know, I think I'm still I am the cloud shell user, but everything's owned by root. I don't have root. Uh, you know, I can't do my my normal poking around in here, but you know, after some digging, you know, we we're given some files. I'm going to start thinking traditional tradecraft here. Uh, there's a bunch of different Linux persistence mechanisms. Uh, one of my favorites is is looking in some of these init files here. >> Yep. I'm going to let you carry there.

We're going to have to accelerate this. We got a ton of crap behind, you know, after the CTF. All right. So, we're kind of we're compressing this experience for you guys. Hopefully you're you're enjoying the perspective of what sort of plain this kind of CTF was. Um I'll just shed some light on there's a whole bunch of things that had to be done to prevent people from screwing with the environment but also embedding the flag where it wasn't easily discovered in plain text. So the hint in theory that map to real world is getting to the home directory. The only thing that can persist cloud shell resets and container resets is the homed. And so that's where

I had to do all the stuff. And deep down, what what do we got here? Any anything that gives you a clue on where to find that flag? >> I'm going to start digging around these bash files. Um, they're looking a little weird. Uh, if I look at that first one, I'm seeing a binary. That's bad. That's not great for me. I'm looking for just plain text bash files. Uh, I could dump into a hex editor here, but I'm nobody's got time for that. Maybe waste more of my day for that. Uh, but I'm also going to see there's some other bash rc files here. And now this is looking a little closer to what I want. Uh, I'm seeing

something's looking here in temp. I'm seeing a a reg x that looks kind of like an IP address. 0909. There's four of those. That looks like an IP to me. Um, and it's looking like it's using netcat to an IP and a port. And we saw that Netcat up in here. It's already in the directory. So, uh I Jenko, you're you're frustrating me. >> So, between August and now, netcat has been installed by default. But I did compile it, make it part of the challenge. >> The compile binaries were to offiscate stuff. The flag particularly that shell script was hinting that there might be a backdoor mechanism close enough to reality that that was what the

participant needs. Set up a listener. something would trigger on log to this environment, pick something up, and send it to uh send hopefully a flag, not that get underscore secret static string. That's right. So, I'm going to I'm going to just fire up that listener. Um I know that I don't have root. That means I don't have access to those lower ports. So, I'm just going to start looking at ports. I'm just going to hit these one by one, starting at 1024, the first port that I can hit without root, and just see where that gets me. I also know in that bash file they were looking for temp files being added. And so I'm just

going to try and do that really briefly. I'll just see if I can uh dump out a a text file there and see where that gets me. So I'm going to assume that this is just listening locally. So there's some lock down. I didn't want to make it too easy. So you couldn't set up a local listener, but I also didn't want people to uh have to set up a public IP listener. So I opened up 1024. The netcat that is referenced is a a wrapper where I can process this. I have the flag embedded. Basically try to make it impossible to get the flag unless you set up a listener and triggered it with

a temp file which is what is hinted at in those scripts. And Chris did it. >> And there it is finally after all that time. >> There's the flag. And there are secrets. Chris, can you do a history? Because this was a multi cloud challenge. It was you solved the first step. And this is frozen history. Had to freeze it, do all this stuff. And in theory, it would say Azure is the next step. Away you go. All right. Okay, Chris, I think we need to speed it up. This was all fun and games last August. It turned into >> you thinking Red Hat, you've got black if you want it. I'm got the blue on. So,

what happens when you take this to the nth degree, real life? What do we got here? >> I'm going to put on my Red Hat and we're just going to dive right in. So, I'm in a full Linux environment. I can start enumerating this thing. I have OS level access and uh I've got OS level access and that means I can look at if there's CVE that I can dig into in the packages uh and all the other traditional trade craft going on here. Um I can run a crypto miner. It's free compute and almost free networking. So this is just the sky's the limit here. Uh to really hit this home uh I've set up a little

environment here up in Digital Ocean. I'm going to set up a little listener just like just like we've been doing. So here's here's my listener. And I'm going to come back into that cloud shell environment. And again, we have netcat. I don't know if anybody's looked at this stuff. This is just incredible to me as a as a you know, a redte teamer. But I'm just going to see if I can fire up a reverse shell here. Uh and then using my handydandy script, I got to grab that. So, while you continue exploiting, I want to point out that's great, Chris, but after you reverse shell out, I get it. I get that now we're thinking real

world and this is what what what you might do is is remotely control that. >> But I'm going to also point out this is heavy lifting because you're still coming in through the browser. You still have to get into it. Um, and this stuff resets every now and then. But the bit of a dirty secret is even in the CTF challenge, we solved the container reset by hooking into the login script. We also removed sudo, but you are still manually slogging at this. Go ahead and show what you were and then I want to ask you how you're going to solve some stuff to do some damage. >> So I I'm already setting up a reverse

shell here. Uh you know, I'm already getting data back from from stuff. So, uh, this is a bad day for for the blue teamers here. So, I'm I'm getting in via this this reverse shell, and I could set up a crypto miner, too. But, you know, in the interest of time, um, I'm going to I'm going to show you something else that we can do. As a redteamer, I don't want my API calls being logged. And Jenko and I dug into this. And this cloud shell environment is just, you know, a JavaScript terminal that runs over websockets. I can get around that completely by using a websocket protocol. And so what I can do is using

these creds that we've just fired up, we created a handydandy Python script that will connect to that very same environment. And now I've got another interactive. >> And what environment are you on right now, Chris? Cuz you're worrying me now as a red teamer. What did you just do? You just connected. And >> I connected to US West one. And now I'm going to run those same commands that I was looking at. So I'm now in a full cloud shell environment if I could spell uh in a full cloud shell environment outside of the browser environment which is something that wasn't ever meant to be. >> So you basically took a hurdle that I had in the challenge which was this was

unmanaged not an API exposed environment which would pose challenges to participants who are redteamers in the end. >> That's right. And you said you reversed enough that you can pop into start and pop into a re a cloud shell. >> That's right. >> Because you're in terminal, you're not in the browser. You haven't gone through that pathway. >> That's right. And I'll show you one thing better, Jenko, which is this cloud shell allows me the ability to upload and download files. And we looked into this a little bit and that upload feature also has a curl command behind it. And so that's something else as an attacker I could mess with potentially. I'm going to upload some malware here,

Jenko. >> Okay. All right. I I got to I got I got to push you aside here. We got to interrupt the red team thing. So, enough of this is smelling bad and making me worried as a blue hat. One is this might be able to be API enabled, right? At a minimum, drive the browser session, connect to the the cloud shell environment through a websocket, issue commands. So, Chris, I got to say, uh I don't care how you got here. I bet I'm going to pick up some stuff in the logs and I just want to start with the end, which is we're gonna I'm gonna say that I can revoke access to you quick

enough that you're just not going to be able to exploit that. And basically really quick for everyone that entails if I'm admin part of remediation in the middle of an incident. I am I'm going to give you this. I'm going to switch over. Yep. >> See how fast I can talk and uh chew gum. Do a little bit of AWS stuff and all the people there. So I see that you came in under the web server role. I don't care. I picked it up in logs and I'm going to do this little bit janky stuff. I have to revoke your access as a role which really means I mean do we have live sessions here? Um you've got live

sessions. Where where were they Chris? Do do you have that? Before I revoke, I just want to prove that this is live and then I'll show you my bag of tricks which which are my remediation playbooks. That's right. Are you up? Is that cloud shell? So you're saying that's a live cloud shell, right? >> I'm in it. >> All right, let me switch back. So what we do in AWS land is we go and do revoke sessions and I pop this button. If I can use this freaking mouse pad. Seriously, you got reverse scroll on this man. We got to come into the modern days, man. Must be a Linux guy. All right, cool. So I revoke it which is

messy as hell. So your token doesn't get revoked the dirty secret AWS has but it attaches this timebased thing saying any tokens for API access created before 2 seconds ago which is what you had before earlier in this presentation should fail in any API calls. So, you know what? Take that. Take that. Take that echo. Um, and tell me if you can defeat it. Then I'll pay a little bit more attention to this. >> Jenko, I I hate to break it to you, man, but that works really well for AWS stuff, but I'm still here in my websocket environment. So, I I can still persist and I can keep up doing those OS commands as long as that token is valid.

>> Ah, [ __ ] All right. So, I thanks for giving me work to do. I think it's time we should switch over a bit to the blue team section and you know as usual give it less time than it deserves because they always get screwed in talks like this. It's all about red teamers and how they win. But let's give it a good try. Basically the nutshell of that is what we saw when we actually put our minds and reversed the protocol and started playing with cloud shells is the concept of session and tokens is really messy. We have browser stuff. We have API tokens, stuff we know about. And then there's this websocket spun up

to issue commands that you see in the browser terminal, but now we have a private API. That thing has its own life and is not revoked when we do the normal revocation. It's the messiness. There's like three concepts running around when you start a cloud shell uh session and two of the three are revoked and you've get some lifespan at the OS level. It is true you can execute API calls. It is true. But uh you've got a a Linux environment to play with. So let me tell you what we would think about when we really put our blue hats on. Okay. So first of all, I got to do some risk analysis. And the problem is I have to

go back to the protocol and do some stuff. So here's a little bit of dump of the HTTP level of what happens between the browser doing CloudShell and the back end of AWS. Some of this is documented, some is not. Okay, this is a well-known AWS research area. As you look at this traffic and you can find a lot of stuff. I'm going to do it as a blue teamer cuz I got to defend this stuff. Got to defend it completely. Some of it's logged. The stuff that maps to UI actions, upload file, download file, start a session, the ones with environment in them are logged. Okay, that's sort of good news. I have a

fighting chance to maybe detect some of this cloud shell activity. I'm going to steal the API and protocol analysis and use it for good, not evil. You know, like those AI people who are scraping all this stuff and putting it into their models. Um, I'm going to use it for good. And what I can do is enumerate in my environment to do risk analysis. How many of my users in which regions are using cloud shell? If you look at this output, it's through this API sort of set of scripting. I see three three regions where we've been playing for this identity. I can at least know some of my exposure and maybe go remediate it. And at some point early on after I

do this as a blue teamer in response to Chris is I've got fundamentally three choices. I can leave the band cuz I hate this freaking fourband Beetle, you know, band, which is basically saying don't use cloud shell at all. And I go down that path of just getting it out of my environment, deny all. But realistically, if you had it, you probably had a reason to it. make your admins protected inside the cloud and do some admin stuff because CLIs are installed by default. I can change the band. Maybe I can use this API stuff, a little bit of vibe coding because why not? Uh control the provisioning and access a little bit more. Install

agents, remove pseudo, do a bunch of stuff and maybe get some free compute and storage care of uh AWS. Or I can look for an alternative, different band, go join a different band, which is instead of cloud shell, I go straight to a compute environment that's API enabled like ECS or compute. still have to do a little bit of D di you know do it your own but I have SSM for that terminal session well-known beast I'll take it instead of dealing with this unmanaged thing so going in detail all right what am I going to do all right I'll also let's just say I plow ahead and say look to make this assessment I got I got to see if I can

do better let's not treat this lightly like most vendors do in advisories or documentation and it's not just AWS it's Azure And it's freaking GCP. All these freaking jerks. Hopefully I'm not violating YouTube terms. Sorry. Sorry. Keep liking me. Um, you can tighten up permissions. I can be specific on the cloud shell permissions. Of course, I can restrict regions. So, I don't have 15 or 17. It's always US West one. I can even add in only, you know, if you MFA off and did this and that can you actually start a cloud solution. Good hygiene should do it. Most people don't. But I can be smart about it. I can be a little bit smarter. I can choose what

not just what regions and whom. I can be more specific. Most of the vendors allow you have two different cloud shells. One that's standard, which we've been playing with public internet access. One puts it on your network. You as a organization, that's your VPC and AWS land. You can now control that network traffic a lot, right? Put in your security groups, your firewall rules, blah blah blah. Awesome. All the vendors have copied that. That's not a bad model. There are some differences that I dumped, which you should zero trust because I will admit some of it's a little bit vibe researched, okay? But I've checked it. I'm not giving you total crap. All right. One of the

differences, no persistence when you do that. Um, more network control. You got to look at that hard as a blue hat. No persistence. So, you know, I might be taking away features from my admin. More network control sounds like a positive. Can I live with this? I don't know. I think it's a big question mark. You can add do-ityour on top of this. Instead of allowing a user to click that button, I could craft a whole bunch of API stuff underneath. I could pre-install. I could go in there and pre-install an agent, take away SU, then give access to the user. Even if it's a completely different CLI, forget the browser. That's janky. If you want a cloud shell

environment, here's our custom CLI. I could do a lot. And maybe I could do it easier with a little bit of clawed bot coding or whatever, you know, heck I want to do. I could pre-install packages. Only things that I I want to allow like not netcat maybe as an example, right? It's more work, but I'm just being real. This is something I could go down as a path. What about detection and auditing? I'm sort of moving from protection provisioning. What can we do? Even if it's auditing that's not real time. Once a day I check logs and do stuff. I I've got stuff to work with. Some stuff is logged. I can tell when people are creating sessions.

Sure as heck don't know what Uni Unix commands they they they uh enter. But create session bursts. Why are there 17 cloud shells being created for this one identity? How about beaconing behavior? Why would beaconing show up? Regular intervals? Because we have to overcome these idle timeouts of that shut down the container environment. So if every every uh minute I get a ping or a heartbeat, I know someone's that's a little odd. And if it extends to 12 hours, which is the hard reset time, and right and this this regular behavior just like malware C2, I can apply here. More work, but I've got something to work with. Long running sessions. Why is my admin always going to the 12-h hour

reset limit of AWS? That's odd. And then another 12 hours shortly after that, 24 hours a day. I got a hardworking admin, right? Anomalous activity. You know, we we could get fancy here. I've got at least some stuff to work with. May not be easy, but it's better than no logging. Thanks, AWS. Let's let's end with this, Chris. Even if I don't get through that, I'm thinking through you've got that persistence through websocket that evaded my normal playbook of revoking access. I still have a nuclear option. There is a delete user action that blows away your home directory, which is where you hooked in all this crap that I sort of introduced in the CTF and you ran with it. And that

will blow you away. And I think the combination will get you out of my environment. And in fact, it's done when you sniff the protocol in the browser, but it's in there. And we can we can absolutely turn that into a little API script. So that's where I'm going to leave it. And I just want to know if you think you really have that much of an advantage. >> Jenko, this whole time I have not been sitting idle. And uh I hate to break it to you, but I got one more thing for you Jenko. There's a a little a little known fact about rolls called roll session name. And I'm going to start setting up my

roll session name. And that's going to allow me to start bumping in to all the different all the different regions. >> Oh, no. All right. >> I have virtually unlimited >> access. >> Never trust a red teamer here. But are you saying Okay, here's the dirty secret. Most things are scoped by region and user in most cloud environments. In AWS, a cloud shell, our normal common wisdom is one identity, whether it's a user or role. Whatever regions they have enabled, you have potentially let's say 17 regions or whatever it is. The default me as the blue hat are worrying about my exposure. If you compromise that identity and put all this fancy crap that you just delivered, the RO session

name is if you do the sum roll call in in AWS that API call you pass in a string that helps identify you and it ends up in logs when you do API calls. It's attribution. And why would that be useful? Because roles are shared. They are identities like first level principles in AWS. The valid use case is we're both admin same company but we don't want permanent elevated privileges to do bucket admin. But when we need to do it we assume that role which is elevated privileges on buckets to do our thing and then it's temporary and we go back to lower privileges. It's a just in time model. Totally makes sense with the

temporariness but we're sharing an identity. So obviously we have to know if Chris is doing crap versus me doing crap when it's the same role. We both when we execute that API call he could pass in his user ID. I could pass in me. It's opt in. You could do thread actor. I could do Chris Ryan. It's only if you're in control of the assume roll call that's often hidden like when we get it through the web server SSRF we get the role of okay long story short Chris is saying and it's validated I'm sorry to say that when you assume role so if you compromised an identity that could assume role you get a separate cloud cell session a

container and persistent directory >> that's right >> not just by region, not by this one compromise identity, but whatever the hell you put into that. It's a specific compromise attack vector, but it could be infinite. >> That's right. And it's >> and me as a blue hat, I know I get logging. So if I see funny names in there with that same one role, I might be able to know all of these lingering environments and I might miss stuff. So this sucks. Was this intentional between the cloud shell people and the identity people at AWS? Probably not. But it's one impact where resources are attached at a more granular level. And we're talking about cloud shell

container compute environments, persistent home directory, 1 gigabyte each, and unmanaged because I have to think hard of how I'm even going to discover that. >> That's right. because I can't log in with the same thing because I don't know what you picked unless it was dropped in the logs and I go back and find it. >> And that's where I'll put my C2s. All right, on that depressing note, which is there's more crap to go through. Um, I think we should jump to a little bit of we spent a lot of time on AWS because that was the start of the genesis idea. We were very aware of our other two friendly uh partners in crime. And

Chris, maybe we cannot cover these in detail, but just let people know our ongoing research with some of the findings there of like is there room for abuse and things to worry about. >> Yeah, that's right. Each of these environments is uh happy in their own way and so they're all Linux environments. They run the same kind of websocket issue that we're going into. Uh and it allows me to dig in further at the OS level for each. So this is a open field for research for us. >> Yep. You find things like personal Gmail. What did you find? Has an automatic uh uh cloud shell in GCP. >> It does. I can open up a a free Gmail

account and I get right into cloud shell. So it uh it allows me to get easy access there as well. >> Think about that model. Most people try to be deny you have no privileges till I explicitly grant you. That's the complete opposite. Plus, it's a consumer entry point personal Gmail, not a full-blown workspace directory. >> All right, let's Freddy. >> Yep. So, uh, we appreciate you guys coming out to this today. I think the last words we'll do and try to end sort of on time is so we we have ongoing research that we haven't fully shared but you could bookmark the repo at cloud shell runner. We to tell you the truth partly because it's a work in

progress and tooling of any kind that's released when it's work in progress tends to benefit um the threat actor side way more than the defender side. What I mean is if you got an API that helps you manage this environment only he needs to sort of work for a thread actor to use it to abuse and get initial access or persistence for defender to actually counter it which we tried to tease out it's got to work solidly for discovering for provisioning for whatever you use and frankly I personally I don't know about you Chris I have a few reservations in this day and age of sorry AI companies scraping stuff without consent and throwing into

their models. All of the research we did, you can do. It wasn't that hard to even come up with the tooling. They already have protocol analysis cloud vendors to reverse engineer stuff and obviously they can code, right? This is the day and age we live in. I'm not here to jump on that soap box. But it means we as researchers and anyone, even if you're at a vendor, even if you're a customer, I think we need to think about whether this tilts the needle. some of this research for net positive or net negative. We're going to be careful. You approach us directly. Well, at least me, cuz Chris doesn't leave his social stuff. If you approach me directly

after, we love to talk to you about whatever you're worrying about or not in this field. Totally share our stuff. I just don't want um the bots of the day to come in, suck this up, throw it in, and suddenly someone some script kitty is using this to launch some attacks in in cloud shell. So, that's it. >> Thank you for coming.