Evil Neighbour Attacks On SaaS Platforms Cloudflare, Bitbucket, Etc by Yash Kadakia

foreign [Applause] yeah all right um so I'll give a little bit of background how I about the talk and a little bit about me as well um even neighbor attacks it's something we essentially came up here in some of our head teams that we were doing um essentially we're just talking about bad neighbors on SAS platforms right uh that's the basic premise um a little bit about me really quick I've been inside basically for about 20 years grew up in India moved to London about two years ago now um social consulting firm called security Brigade which has been around for about 18 years and the other one's not coming through for some reason um a SAS platform called Shadow map as

well which does document monitoring attack surface management Etc um so as part of our red teams right we run across a lot of really interesting scenarios and the traditional SAS I mean the Assassin cloud has sort of changed the stack that we typically deal with right um I've spent about 18 years doing red teams now and traditionally red teams were always oriented towards your traditional perimeter right you're looking to get and get an interactive directory you're looking to get onto lateral movements once you get onto the network and sort of go from there right um in the last couple of years we've been doing a lot more red teams that are startup oriented it's a lot more SAS

oriented a lot more Cloud oriented uh a lot more AWS and it sort of changes things in terms of our approach and what we typically do in a red team right um so the whole talk that I'm going to give you today is a redeemer's perspective which is a little bit different from of course the devsecup folks a little bit different from the pen testers it it kind of looks at a broader sort of perspective right um uh just a quick sort of uh example of a sample red team Journey that we did a couple of years ago actually um just to give you a context of what what cloud or SAS red teams typically

look like how they differ from your traditional uh perimeter with behavior red teams uh so this one started with an ha proxy server that was running publicly misconfigured let us start talking to things behind the network that weren't supposed to be public right went from there to accessing an Internet domain so a private domain to accessing a Django server that was running on debug mode debug mode gave us a bunch of password hashes that had weak passwords crack those passwords back um and they were using the same credential for VPN or very limited VPN right so this was just the Gateway VPN essentially you get you route through that IP address but you can't really

access any on the land right um turns out they had a bunch of instances running on AWS they were white listed to the particular office IP address so now we had Jenkins uh running on a white listed basis we could start accessing Jenkins and go from there right uh Jenkins had Google single sign-on we were able to get around that because they weren't um they weren't checking what domain you were coming from you could log into your personal Gmail and still get in uh pretty classic flow there uh use the Jenkins build functionality to start executing commands um so Jenkins has a really interesting functionality um eventually admin is essentially you can run commands locally in the Jenkins

instance with a scripting interface ran the commands extracted the passwords that were saved secrets that were saved inside Jenkins decoded them got GitHub I believe credentials that had 2fa though uh but when what's interesting is when you try to enumerate GitHub through the API it actually doesn't ask for 2fa in most cases as long as you name the report know the repository names it will let you check them out and download the code um so did that got about 200 repositories one of them was called devops contained a bunch of SSH Keys uh inside them SSH keys to find some jump servers connect to those servers get onto their SSH they had sudo enabled got root bunch

of Saved SSH Keys hopped onto a bunch of different machines on the network and pretty much had everything at the end of that right um so that's typically what our Cloud SAS red team Journey looks like and it's really interesting because it's very different from what red teams traditionally used to be right right no active directory no real um Enterprise compromise right everything sort of happened outside the network so what I'm here talking about today is essentially evil neighbor attacks right I'm talking about multi-tenant SAS platforms so your bit Market your cloudflare anything right GitHub any of those solutions that most Enterprises use uh where you have a single set of infra and then of course multiple

different organizations using it and we've seen some interesting attack scenarios that we were able to come up with during our red teams um that we could leverage and sort of use that to compromise things right um so I'll give you a couple of examples and I'll talk more about these as well so one is abusing certain functionality within cloudflare to get past the back so even when the origin is restricted limited no Public Access we could still bypass the graph and sort of Hit the endpoints and get around that second is a cloudflare access scenario where we are able to uh I'll talk a little bit about what plot for Access is for anybody who doesn't know it uh but

we're able to get around Cloud for Access and directly talk to the end applications essentially bypassing the zero trust authentication that they have in place and the last is uh was bitbucket where we were able to go and talk SSH to servers that one service would be on the public internet at all um so your Enterprise servers that are uh behind your firewalls IP whitelisted restricted from bitbucket for everyone but bitbucket and we could still get around there and talk to them so I'll go into more details but that's um the other thing I want to add right now is the this the attack idea isn't really limited to cloudflare or bitbucket these are just examples that I

saw During certain set of red teams um this would work for more SAS platforms where there's white listing or limitation that some sort of controls that you're putting in on your Enterprise and uh and you'll see what I mean when I get when I talk a little bit more about it so getting through the web right so again just a little bit of a refresher cloudflare Waf essentially Cloud any Cloud Web right is primarily going to work by routing your DNS to cloudflare's uh uh DNS it's gonna route all your traffic to cloudflare swap and it's then going to talk to your origin server right um that's your standard Waf implementation they typically recommend that you use cloudflare tunnel or IP

restriction on the origin server to make sure that an attacker can't get around it and talk directly to your origin server right which which typically happens um when when you look at bypass there's typically two ways that it will happen one is you're either using some sort of uh input manipulation to sort of bypass the signature and the second is you're trying to talk directly to the original server right what we're doing is a completely third approach um and it's it's pretty interesting it's really straightforward so I've taken the example of security brigade.com which is one of our websites uh where if you try to access wpe hyphen config.php uh it's going to give you it's cloudflare is

going to pick it up and block you right and this is the standard rule that cloudflare has and you can use any rule essentially to test this out um so anytime you're trying to access something potentially malicious gonna pick it up block you and and take that action right and what we realized was that um okay so to do this attack the first thing you're going to do right is we have to buy Cloud for a pro so we had to pay for that uh gets a little bit of an audit Trail there um so you need to buy a cloudflare pro account it can be for any domain right so in this case you can't really read it

uh it's for evildomain.in it has nothing to do with any of the topics right it's a completely independent uh account that we've set up with no connection to any of the domains we're targeting right once you buy cloudflare Pro uh you have something called health checks so cloudflare gives you the ability to start uh pinging the servers at a regular frequency to see if there's they're online if they're working fine if there's any sort of downtime you can trigger the load balancer you can check out all of that right um so what we've sort of realized right and what is really interesting and we actually use this as part of our red teams was you could create a health

check for pretty much any website in the world it doesn't really restrict it to your particular scope or your particular set of domains that are in your account so ideally it should probably restrict me to email domain.in which is the domain I have linked to the account but in this case what you can see is I mean you can't really read it very clearly I'll sort of walk through it let me say again a pointer is I've got the original IP address which we identified through uh through just looking at past DNS records in this case right historical DNS records um we've got the original IP address the origin again let me just flag is IP

restricted no public internet access you can hit it with nmap and it's going to come back with no ports open right um so that's not the attack Vector we've then asked um Cloud fairs alert to look for the wp config file that we got blocked with previously we've asked it to look to make sure that it's getting a 200 okay header response back and we are asking it to add a host header for www.security brigade.com or any internal subdomain in this case right so you could add a lan subdomain you could add any subdomain that's mapped to that particular IP address and we should be able to access it uh getting around a lot of the controls

there and once you do this you're essentially going to start seeing the response comes back and says it's healthy which basically means it was able to hit that file and got a 200 okay response versus a four or three or five or two or anything that's limited right uh on the server end you can look at the logs and the logs will tell you that WP config.php was hit a 200 response Got Served the hell check ID that was used which is the same idea as the health check I created um and the attacks are coming in from cloudflare's IP addresses uh owned by cloudflare which are the list of IP addresses that cloudflare asks you to

whitelist uh when you're implementing the web so put it really simply uh we're abusing the functionality to use cloudflare's own IP addresses to talk to the endpoint uh getting around that origin bypass that's there and getting us and bypassing the vaf in the process as well right so we can force cloudflare to make those malicious requests on our behalf and it's going to hit the end point it's going to get processed and served and we're going to get around that so that's that's one attack that we've used a lot in our red teams and it's been really interesting right um the next one is again the same attack but I'll I'll cover both of it I talk

about mitigation and controls that clutter already has available um is cloud fed access right uh so we use plot for Access as an organization so all of our assets are behind zero trust cloudflare access there's no VPN essentially being used um so for those that don't know what's right I've just got a quick little GIF that plays with Cloud for Access but simply put you're gonna browse to your application so in our case it's lemon.securitybrigade.com it's gonna check your rules on cloudflare if everything's okay it's going to give you some sort of single sign-on right so it's going to be Google or whatever whatever's been configured by the administrator and it's going to potentially let you in based on that

so it has it's as simple as that right uh if I look at trying to access lemon.securitybrigade.com which is one of our websites it gives me the cloudflare access Google's SSO page if you look at the actual burp requests you can see that initially the request goes to Lemon which redirects me back to shadowmap.cloudfed access.com which checks whether I meet the rules to authenticate so whether I'm coming from a particular IP set whether I'm coming from a geography a bunch of rules that we can Define and if all of that's okay it then redirects me to Google and let's go to Google authentication and then come back and access the application right so in the same exact way what we're able

to do and what's really interesting is we're able to again use the health check tool within cloudflare to start talking to those applications which completely bypass Cloud for Access for us this is again I'll talk about this in the mitigation phase but this only works when you've done an IP address bypass uh when they've done an IP address white listing right um there is another method that cloudflare recommends as well I'll talk about that in a few minutes but the I and what's interesting here is they have uh they have two options right one is expected code what response are you expecting on the server and the second is expected content so what you can now

do is you can actually make your requests and then enumerate the content that's actually on the page that's behind the authentication right so for example uh in this case I put in the word lemon which is just the title of the website um and you can see if it's actually legitimate it comes back with a healthy status if it's not it comes back with unhealthy and you can see in the if you Mouse over it it says response body mismatch error so if you're looking for sensitive files ready if you're looking for password files config files something of that nature you can actually enumerate keywords letters you could use a blind SQL injection method where you can actually enumerate one

letter at a time uh figure out the letters that are in use and then use that to go back right so there's a lot of um interesting things that you could use this for potentially um but yeah so I'll talk about how to fix it as well uh and this particular set of bugs you can fix pretty immediately um but yeah we we've had a lot of fun with this during our red team so far at least uh yeah so in terms of fixing it's a cloud fair when you're implementing graph or implementing access typically recommends uh restricting connections to your origin server from anywhere but cloudflare which is what we're abusing in this case

so if you've chosen to go with that particular method this attack will still work right it'll be it'll still get us to access your web server and the other one is cloudflare tunnel which is where you use Argo Auto flat tunnel to essentially create a reverse tunnel where cloudflare Daemon is running on your server talks back to the cloud friend Network and routes traffic through that way right so if you've happened to use cloudflare tunnel you should be okay uh if you're using the IP origin method then you're most likely going to be vulnerable to this you are going to be available to this attack right um so that's that's one set of sort of

attacks and one sort of vectors that we've seen here and there's a lot of opportunity within cloudflare as well so I think uh we were experimenting using cloudflare workers and using scripting within workers to try to leverage the same attack as well which I'm reasonably sure is possible which is another time to explore it enough uh but again not just just point uh I mean it's not a cloud specific issue right this is the issue that you see at a whole bunch of different SAS platforms um after being the one that I use often enough that I could pick it up and and sort of talk about it comfortably right um yeah so that's one Set uh bitbucket

again something very similar so what happens then you're a lot of our red teams um is we end up with credentials right we'll we'll get access to SSH Keys we get access to passwords but we won't be able to actually talk to the server eventually because the server is behind some sort of controls and that's exactly what happened to us studying one of our red teams that we were doing um and we comp I I believe through Confluence or jira or some sort of somewhere that we compromised we ended up getting SSH keys that were being used on some particular set of servers uh we just could not talk to the server right so we had the keys we couldn't get to

the door and it was really frustrating um eventually looking up DNS records we realized that the customer had was using atlasm products um so at least when you do a domain verification it adds a text record you can enumerate and we use that to figure out that um they were using SSH um we did have the original IP addresses in this case we knew the public IP addresses from DNS records but they were completely restricted right so SSH is filtered you can't connect to it times out um iptables is being used on the particular instance to just block all other IP addresses out right uh so that's where we were stuck and we spent a couple of days brainstorming trying to

think about how do we go about this um and as you can imagine it can get really frustrating right you've um you're running a red team you've got credentials you've got access you've compromised you've essentially got there right um and just when you thought you you about had it you realize that you can't really do anything further with it um so that's where we're at and we realized eventually at some point so we were actually using our own bitbucket pipelines for some deployments um and we realized that bitbucket has some SSH tools built in right uh so it has a SSH tool built in where you can plug an IP address in add host it's

gonna tell you whether it can connect to it or not if it can it gives you the fingerprint pack right so great we could now use this to essentially run across their subnet come back with a list of IP addresses that they had white listed for bitbucket right um so we were able to enumerate a list of IP addresses that we knew are running SSH we knew our um whitelisted for bitbuckets IP addresses and that can't be accessed from the outside right uh so that became the next step we went and wrote a whole bunch of pipelines um I mean no rocket science there right we added the SSH keys that we had compromised or the password that you've

compromised uh it doesn't have to be limited to SSH could be RC could be any protocol that they're essentially using for deployments right uh since it'll be an IP whitelist you could realistically use that so we could then go plug in the IP address the user the SSH keys of course uh the attack was based on the fact that we already had the keys and we had the passwords uh but it got us to a point where we could now talk to that server that was otherwise ipy listed and we could then start running the pipeline and getting access um to execute commands on that particular server through bitbucket right so we would simply go and edit the

bitbucket pipeline each time change the commands if you were running uh get a reverse shell going or get whatever we need to go going as part of the process and then bitbucket pipeline would execute through its whitelisted IP addresses if we talked to the CZ server execute the commands come back give us the response so the results in the log file or a text file or the output uh however we set it up and that would work as well right so yes um give us the wind that we were looking for and these are a couple of examples right there's a lot of other platforms that I think are potentially vulnerable uh it's just we haven't had the

bandwidth to go and test each one of them out and and sort of figure it out right um mitigating this particular attack is a little bit of a challenge right so um the currently I I mean we've spent a lot of time brainstorming and the the customer we did the red team four was not happy we didn't have a solution uh there wasn't much we could do there but um we thought about it a whole bunch and the only way we could really recommend to them uh was to sort of restructure to use Docker instances versus the direct SSH push right so when you when you're using bitbucket pipelines you can choose to use SSH to SCP code over to the

server you could choose to use rsync to move squad over to the server um in in all of those any push-based method this attack would essentially work right because it's all leveraging IP white listing so what we eventually had to get them to do was um um to use git on the other end to sort of check out as part of the bitbucket pipeline so they would essentially use either git to do a check out of that and download the code so that there was no direct connectivity or they could use Docker to then push an image from here to the docker instance and then use Docker to come back through kubernetes and download it

um there isn't an elegant solution Cloud Fair of course had a great solution already uh through Argo Tunnels for this we haven't thought of a good solution yet I'm not sure um how bitbucketer and all GitHub even would go about fixing it um because short of giving static IP addresses to customers um they're gonna have to tell folks that uh they'll have to be some sort of secondary certificate process check out some sort of check that they'll have to add to fix it but I can't think of a good fix to actually go and uh adjust this particular issue right now any questions so far happy to sort of answer as we go along as well

they could but I'm not sure if anyone's gonna um agree to deploy that because for cloudflare you're inherently using the Argo tunnel as part of your network in any way uh it's not a separate measure that you would have to use in this case it would have to be a whole separate client and implementation that would happen yes I have I have reached out to them they haven't come back with a concrete solution yet but I think yeah because again it's it's complicated for them because uh you might have multiple web services and multiple applications that you want to sort of do a health check for outside of your scope and fixing it would mean

essentially breaking every single health check that's not within the scope I mean not within those Scopes it would be a big breaking change for them to do so I don't think it's as um yeah they probably would have to uh I think for now you're probably going to see them change the IP addresses uh they'll probably want to remove the uh not let health check use the IP addresses that are within the whitelisted pool uh so that then it has to go through the public internet anyway uh but again I'm not sure because in that case again uh you wouldn't be able to do an origin server check so if I want to make sure that my origin servers

are working correctly and I have 10 different religion servers running I can't do that I would still have to route my health check through cloudflare uh so cloudflare demons the only really good way to fix it and still retain functionality so um I am not sure to be honest if it's fade again anything else

like I said essentially the the challenge isn't really limited to these folks it will be to for most Enterprise SAS Solutions so uh github's uh build feature as well Jenkins as a service is gonna they're all gonna fall into the same bucket right um it's not a cloud specific problem or it's not a bit bucket specific problem um any Enterprise solution SAS solution where there's whitelisting required so um from a security perspective we're gonna have to look at a process that stops um that stops rather asks the question when white listing is required for third-party SAS Services uh because these sort of attacks I believe aren't really going to be mitigated easily uh it'll break a lot of functionality for

it to be for it to mitigate so it's going to be an accepted risk I think um so yeah it's going to be um um it's going to be interesting but this will um yeah so I think it's gonna have to be more of a security process change where operational teams will have to sort of ask that question when Dev teams or devsecop teams come back and try to figure out how to uh how to get something whitelisted for a shared Enterprise service

us yeah and that's that's what any Enterprise service that has shared IP addresses uh we sat down and tried to make a list but it was just um it was it would just take a huge amount of effort to go and evaluate every single one that had shared IP addresses and some sort of human human control but logically they would all uh fall to the same the same issue logically of course cloudflare is the only one that had an exception which is why I sort of took that as an example doesn't have a have a good mitigation anyway so yeah take away from this I mean honestly so this is the the Crux of the talk right uh I'm assuming that as

folks dig into it more it's gonna more and more services will kind of get vulnerable to it and it's pretty interesting but the uh the Crux of this is essentially the need for the need for outside insecurity right um typically when we deal with a lot of our customers folks are extremely focused on sort of I mean to be honest if every time I had a seesaw ask me about dark web monitoring versus just looking at their perimeter their Cloud Solutions and that sort of thing right it's a we tend to sort of lean towards the what's being talked about a lot more than the Practical risk that's actually there um and I think red teams are something

that add a lot of value from this perspective uh because traditional pen testing traditional apps like of course adds a lot of value but it's typically looking from uh accepted pathway right I have an application I'm looking at an application I'm trying to get there red teams are forced to look from creative interesting ways to sort of get in um and that then in turn typically identifies a lot of these really weird uh scenarios where we're able to access things um through ways that were completely uh not the intended purpose of the design purpose right uh and then that drives a lot of value so we do a lot of red teams where we're able to pick this sort of

stuff up and it might just be mess-ups on the customers Android it's not necessarily always um going to be cloudflare somebody else that's really messed up uh but if you if you look at a lot of project teams that we've done right for example um uh there are just tons of issues where uh it's something extremely small that results in everything else getting compromised right um I'll give you an example um so we were recently doing a red team for uh for a large Enterprise customer um they had I mean every every single important asset was VPN to fade VPN uh you had to specifically raise a ticket for VPN so that you could only access

that one particular endpoint inside the native that you needed access to and get all of that down right uh eventually turns out that they had a um what's it called a remote support platform um that their ID teams were using to support users and that's the one that didn't have two fa we could get access to use that to access all pull a list of all the machines access all the machines execute software do everything RSM software I think uh but yeah and start compromising all of that right so uh it's typically that outside view outside perspective the red teams add a lot of value to um that typically gets missed out and um yeah so that's it any questions I'm

happy to answer we're a little bit early um which uh James if you have any questions with the mic uh but otherwise it's possible yes awesome thank you very much

I'll run around with the bike um no I think we've maybe had the questions and cool well we have finished a bit early so we get some coffee or a bit fresher brilliant thanks again yes that's great thank you