BSidesSF 2022 - Hook, Line and Sinker - Pillaging API Webhooks (Abhay Bhargav)

our first uh presenter in this track for the afternoon will be abbe speaking about uh pillaging api web hooks hook line and sinker so abe with no further delay i will turn it over to you and we are looking forward to hearing about them thank you and good afternoon um let me just start my slides all right so the first session after lunch so i guess i'll have your attention for about seven minutes uh anyway i'm going to keep this snappy and hopefully have a little bit of time for questions as well so this talk is a result of a lot of research that we did last year and we have tried this out on

several bug bounty targets as well as you know paid consulting engagements that we do so that's what i'm here to talk about i'm not going to be able to talk about all of the stories but i'll do i will talk about a couple of them and basically the talk is to highlight a broader set of issues that this stock is meant to focus on which is basically web hooks and how you can attack web hooks right so i'm going to be focusing on a very specific set of attacks against web hooks so first of all my name is abhay and i run two companies one is an application security company which does pen testing and

devsec ops and cloud security services the other appsec engineer we do a we have a training platform that uh we use all the research from what we do at the consulting company and put that to the training company for hands-on labs and a lot of lessons and so on so it's a lot of fun uh i started off my career as a pentester mostly in offensive security but more recently my work has largely been in the defensive security side a lot more focus on cloud security and devsecops and so on i do a lot of training i'm training in fact at black hat um and i've trained at several oasp events uh and at previous black hat

events as well so this is me i do share a lot of content on youtube so do check us out if you're interested as well as my blog and on my twitter as well we do i share a lot of useful information and sometimes opinionated takes on the appsec industry anyway so my talk today is basically if you are thinking that you are coming here to witness a very serious talk my talk is nauseatingly full of memes so be warned right we're gonna have a lot of memes uh and smart speakers record demos right smart speakers that's what they do unfortunately i'm not one of them i'm going to be doing a live demo and when

you're doing a live demo 300 000 things can go wrong i both my demos just failed before i came on stage so let's see what happens right so i'm going to be praying to the demo gods that my demos work today anyway so the agenda today what are web hooks how do they work i'm sure most of you sitting in this room have probably dealt with web hooks you probably build web books you test web hooks sometimes perhaps if you are in bug bounty you perhaps found a whole bunch of flaws around web books as well this talk is about web hooks but in a very specific sense about a very specific attack against these web hooks

and what you can do with that particular attack so we're going to look at some common web hook attack patterns this is a very red team talk although i will spend probably the last two three minutes on the defense side as well um we will be talking about ssrf which is a favorite flaw for a lot of us right we all love ssrf ssrf is great because it gives us access to a lot of interesting things this talk is in essence about ssrf but applied slightly differently right slightly differently it's not the typical ssrf attacks that you would see against a normal web application this has kind of a boomerang effect that i'm going to

be talking about in a little bit so this class of flaws we dub webhook boomerang flaws basically because they have a way of working like a boomerang and you'll see why that is uh pretty soon and i'm going to talk about this and we're going to be talking about some sub variants and some gotchas here and of course finally i'm going to spend a couple of minutes on defense but defense is not the core focus of this talk it's largely about the flaw how it works why it works the way it works all right um so so when you're building an api there's a good chance that you are building web hooks as part of your

application right whether it's an api or a more old-school application there's a good chance that you are building in web hooks as part of that now web hooks are a very important part of our modern application stack because they allow you to integrate with thousands of other applications right so web hooks are a very critical piece of your integration puzzle because it allows you to integrate with thousands of other applications so basically web hooks are what you would call user generated callbacks right so let's say uh you're setting up a stripe uh payment interface or you you have you're using stripe for your payments to process your payments and you want to know whenever you have a new

customer right so what do you do you go to stripe set up a web hook and say that as soon as there's a new customer that signs up uh make a request to my web hook that i have set up on blah blah blah blah dot com slash web book receiver and you can process that event however you want you can push it to slack or you can do something with it send them a welcome email whatever it is right you can do a whole bunch of things with it the idea here is that you as the user are generating callbacks from your application based on certain events that are happening right so these are

typically triggered on certain events that happen a new subscription comes in a new payment comes in a user is deleted user is created any event that you would like to you typically trigger web hooks for and if your application provides your application would allow you to trigger those web hooks for those events for a variety of those events now webhook transactions typically have two parties one is the provider or the producer and one is the consumer now the provider in this case is stripe right in our example the provider or the producer is stripe and the consumer in this case is your application right whatever application you have set up to receive those web hooks that's the

consumer stripe is the provider uh so that's typically how it works and it's typically a post request right so if you look at where books are all over the place it's not it's not a uh it's not one of those very very rare things you'll see them all over the place they're companies that have built their entire business models on web books uh web books are in kubernetes web hooks are in ci systems web hook on in marketing automation systems web hooks are pretty ubiquitous they're all over the place you will see them in um pretty much any app that you're dealing with uh in a lot of these especially cicd systems if you're if you're working with

ci cd and marketing automation where books are literally an extremely critical part of these these applications right so this is it's not something that is a very small um you know technology integration that we have now web books typically work like this they're event driven we've already covered the event driven part once they are that event gets generated they make a post request right they make a post request to the bunch of json that says hey a new user was created this is the user id um email blah blah blah blah blah all of that jazz right it generates a post request for uh you generally it's a poster it's not always a post request but in most cases

it's a post request that hits the consumer now the consumer once it receives that json the consumer can do something now it's not always json it can be xml it can be aml but most of the time it's json right so the consumer receives json and from there the consumer can do something with that data now additionally uh good web hook providers also give you the ability to configure additional security for your web hook right they give you hmac and api keys that allow you to verify whether it's coming in and they also make sure that you have the ability to verify them when a request is coming from their end so again when i say sometimes most of the

time you would see that this is the case especially with applications that are especially sas applications most of them give you the ability to do this and the other thing is also that in some cases these providers also allow you to set custom headers how many of you have seen this you can set up custom headers on a web hook right you can say that to access my application you will need to provide this particular header and that also is part of that particular request that comes in from the provider and in i've not mentioned it here specifically but there is another feature that a lot of web hook providers provide which is they have the traces of

the http request and the response as well right it's not just that it's a fire and forget system they have the traces of the http request and response that they send and if they fail they retry they have the traces of the failure they have a lot of telemetry related to every single time they trigger a web hook so this is also a common trait of a lot of web hook systems right so you'll see this with a lot of web book systems so if you look at the natural focus uh the attack focus at least um is that a lot of us would think when you're looking at attacking web hooks a lot of

us the thing that immediately comes to mind is how do i compromise a consumer right that's the is that the large scope of thinking most of the time we think that how do we compromise the consumer because we think of it as one-way communication we think of it as this request comes to the consumer and that's really it we don't really think about it too much from a provider's perspective right because we think that okay this is hitting the consumer and that's it we're done we don't really care about it from that point in time so most of the attack focus that most of us think about is from the point of view of the consumer

so can i do some kind of a deserialization attack against the consumer can i tamper with the payload can i do replay all of that is targeting the consumer right most of these attacks target the consumer that's really one of the common set of assumptions that we see when we're trying to attack web hooks however in our attack we're going to be attacking the provider we're going to shift our lens a little bit and we're going to be attacking the provider we're not going to be attacking the consumer in this case we are going to behave as a malicious consumer and through our malicious consumer we're going to be attacking the provider so that's basically what we're trying to do

here so our focus is can i compromise the provider so that's really what we're going to be trying to do here now can i do something like this so this is the provider this is my consumer the provider makes a http post request with json to my consumer and the consumer finds a way of sending that request or using some technique to attack the consumer or the provider as part of that same transaction right as part of that same transaction the idea here is that the consumer tries to attack the provider and is of course the idea here is that they are successful at attacking that provider so that's basically what we are planning to do

here that's the goal of our attack today right so this can only mean of course when you look at the world of requests and responses the attack that most favors this kind of construct is ssrf right because ssrf is all about redirecting requests or misdirecting requests so whatever you want to call it ssrf is the naturally suited attack vector for this sort of thing so how do we make ssrf happen here right so let's talk about ssr of a little bit i'm not going to spend too much time on ssrf because i'm assuming most of you know about ssrf but i am um i'm just going to focus a little bit on what ssrf is before i'm getting into uh

the actual attack itself so ssrf is basically where you are through uh through the victim application you are getting the victim application to make requests to an internal application and get your response from that internal application that could be a metadata url right very classic aws metadata url where somebody can access 169.254.169.254 metadata credentials dump all the aws credentials as part of that response that is ssrf right or ssrf could mean i'm running this nosql database on 10.10.10.10 internal network and i use my application to this victim application to make a request to come back with a you know chock-a-block full of database responses so the idea here is that ssrf is used by the attacker to get

the victim application to get responses from an internal uh in most cases http uh request but it need not always be it can be ssh it can be go for it can be a whole bunch of things any url uh pro scheme that is supported by the client on the victim's end can be used in that case even a file so for instance let's say in a kubernetes cluster you have uh the service account token mounted at var run service account there's a seek i don't remember the exact directory but that's there so you can use a file call and slash and then get access to the kubernetes service account token on that pod using

ssrf as well depending on the client of course all of this depending on the client so ssrf is not new we've seen this it's been used in several attacks we've seen this and it continues to happen ci systems have been vulnerable to it uh payment systems have been vulnerable to it pretty much any uh major provider you've seen some kind of vulnerability in terms of ssrf it's been used in massive attack chains as well so ssrf is not really new right now the problem though however uh of course ssrf like i said can be used for a lot of these things you can uh cloud has become the natural target of course because with the cloud you have the ability to

access metadata and through that you can steal credentials so that's become a big part of ssrf exploitation but you can do remote file rate local file read you can do a denial of service you can do remote code execution and so on now what we want however is that in our case we have the provider we have the consumer we want to redirect our consumer driven request back to the provider so what we want to do is instead of doing an ssrf to uh to our network or we are obviously the consumer we are outside we want to redirect a request back to the provider so that we can attack something internal to the provider we

want to attack an internal url or an internal web server that the provider is hosting that we can compromise through this particular webhook attack transaction that's what we want to do here right so that is why we're using ssr so in this case we have a web web book request that comes in and then the consumer redirects it back to the provider and says hey you know what make a request to 169.254.169.254 and your provider just makes that request and comes back with a response which has credentials right but there is a problem here obviously you will be thinking you probably have guessed some of those problems which is this is you're talking about a post

request right you are dealing when a provider sends you a webhook response or a json payload they are sending a post request but ssrf largely relies on get requests right ssrf relies on get requests how how do you do that how does that work when a post request comes in how do we transform it into a get request so that we can actually successfully execute that ssrf right so the idea here is that you have all of these things there is a get when there's a get request involved it's pretty straightforward but in most cases web book makes post requests and put requests or what have you and how do we do the ssrf how do we manage the ssrf

in that case basically we have the http 303 redirect now the 303 redirect has a quirky capability the way it works is that you may you you get a request which is a post or a put but if you do a 303 redirect it makes that a get and redirects it to that new location right so your web server if you uh set up a 303 redirect on a post request you you get a post or a put but that's automatically converted to a get request and of course it sets a new location header and that's where it redirects to so that's what we're going to be using to convert that typical post request back into a get request and then attack

our consumer that's basically what we're trying to do here so the idea of a 303 is it's it's called the see other where we say that hey you're sending me a post request but i'm going to redirect you here which is a get request so the end uh request that goes to the consumer or sorry to that goes back to the provider is a get request so in that case the get request could redirect to 169.254 or an internal ip address or what have you right so all of those things are possible because of the http 303. so what we want here we want the provider again the same thing we have the provider makes a

web hook request to the consumer the consumer http 303 is that stuff back to the provider and points to the internal metadata service which in this case is 169.254 whatever it is and then we get access to that internal metadata url and credentials and all of that stuff so that's basically what we're doing here so this is the crux of the attack now we use this on docker right now docker uh i'm sure all of you have heard of this small company we know as docker now docker uh obviously hosts a pretty large docker hub right so docker hub is in most cases our source of uh you know getting container images and we use container images from docker hub

all the time so when we do docker pull uh couchdb or mongodb or nginx or whatever it is the it the request goes to docker hub and docker hub pulls uh pulls down this container image and of course your you can start using it at that point in time so docker hub had this func i mean still has of course i mean but this functionality where you can create a web hook you can set up a webhook that says as soon as an image is pushed into my docker hub trigger a web hook to my consumer url so in this case we set up a nok i o ur url and we set up a redirect

a request a web hook to this particular consumer uh url now we start we pushed and we set up a small application that does this right so this is basically the application that we used it says whenever of course you can the redirect url was set to be different i've just set it to localhost but it's not localhost so the redirect code is 303. as soon as a request is received on the slash route in this case it's going to 303 redirect that request to this particular location which is 169.254.169.254 slash latest metadata and of course you can add additional things as you start to enumerate uh the credentials and the different paths in that particular

internal url right so you can with this what we were able to do is we pushed a container image to our docker hub and once we did we started looking for web hook traces now we knew docker captured the web book traces but the web book traces were not available on screen we were not seeing that come up on screen in some cases it's available on screen some cases it's not in some cases it's not so in that case you're typically flying blind you might have to try a blind ssrf type of thing but in this case it was we thought it was blind but it was actually not so when we actually looked at the network request

we saw where book trace is being downloaded and we were able to enumerate the role that was being used by docker's dockers this was docker's eks cluster that we learned later on and we enumerated that particular role from docker zks cluster and from there we were able to dump the aws credentials as part of this attack right so as part of this attack we were able to dump the aws credentials and turns out those credentials were for their eks cluster that they were using for their entire docker hub operations for docker hub right so this was uh the credentials for that um eks cluster and of course we immediately wrote to them we verified that this was a finding we

didn't do anything beyond this uh we verified this was a finding we wrote to them and they fixed it in a couple of hours thankfully they were pretty quick about it they fixed it in a couple of hours and we verified it and of course uh that was basically it so we were able to identify this particular flaw so i'm going to demo this right now i'm going to do a live demo like i promised and i'll show you how this works so i've just i've already set up most of the demo i'm just going to quickly run through this now um i am a user so this is me as a attacker slash user on an application

i'm gonna set up a web hook and that web hook is gonna be a malicious consumer web hook right so this is basically me doing the setup i'm just going to quickly run through the instructions so i'm going to log in as the user so i'm going to log into the user like so let's hope i have a jwt yes i have a jot so i have a jot and i'm going to list all my web hooks so in this case as the user i am going to list all my web hooks and you will see that i don't have any web hooks as of them as of right now i'm going to set up a

web hook for a post request so i'm going to quickly set up a web book for a post request that says hey whenever a user is created or whenever a book event is triggered you need to send your so i'm setting up a consumer url right now you need to send your webhook request to some url port 8088 in this case method is a post method so in this case you'll see that i've set up my web hook and if i pull down the number of workbooks i have i have one webhook consumer that i've set up right so now i'm going to trigger now the way this demo works i forgot to mention that so we have our victim

application and we have a couchdb database running as an internal database service on our internal network so imagine that there's a coachdb running on an internal network the the nature of couchdb is that you can directly access coachdb over http request you don't need to use a special protocol or something like that you can directly access it using http request like post get and that kind of stuff it's a nosql database that's very easy to work with in terms of http requests and responses so this is this is the example here so the idea here is that we want to steal data from our couchdb database in the back end right so the idea here is that when we set up a

webhook and we're using that webhook we're subjecting it to the whole boomerang thing that we've seen so far and it's going to come back with traces it's going to store the traces and we're going to see the traces in real time so we're going to initiate a webhook event like so so it says it comes back with a trace now let's look at what this trace is so if i do echo this it looks kind of weird let's see what the trace is it says users so we were able to enumerate a couchdb database called users because my hook essentially was configured to get this right it was configured to redirect to uh this particular url

underscore all underscore dbs right so it's going to list all the databases in that particular couchdb so now that we know that there's a database called users we're going to use users and say get me all the docs of that particular user db and i'm going to rerun this let me just check i need probably need to restart oh it's restarted automatically great perfect so it should work let's rerun this request and this time it should dump all my users from that couchdb user table right it should dump all the users of course in this case it dumps the keys and the values i can start enumerating based on each key and things things like that so if i take this and

if i jq this dot content base 64 dash dash d so you'll see that it dumped my user's keys and yeah the values and stuff like that but of course not the full value it just dumped the key keys i can enumerate and get into it with uh by accessing individual keys and ids and values and so on right so this is a very simple demonstration of this again you can use this on aws on an imds v1 configured aws service which is a lot of services still unfortunately and you can do this right you can actually pull off this attack if you have a web hook that is vulnerable to this right so let's get to defense really

quick and uh some other sub variants of this so one of the things is that a lot of people as soon as i come up with this they say yeah aws is fine but what about if it's gcp what about if it's azure how does that work how does metadata access on those clouds work now remember this was something i we mentioned in the beginning a lot of web hooks allow you to configure custom headers right they allow you to configure custom headers and what do you need to access the metadata urls in azure and gcp anyone you need a metadata header right so if you have a web book that allows custom headers to be configured

you can attack you can use the same attack against gcp and azure because all gcp and azure need to access that metadata url or the metadata endpoint is literally just a metadata header right so in this case we compromised the ci system by adding the google metadata header and they were running their services on gcp and we were able to get access to their entire internal infrastructure because we were able to access their runners metadata gcp metadata service and through that we were able to potentially escalate privileges across the board to their entire infrastructure that was running a whole bunch of runners on gcp right so this is also possible so when you have custom headers that's another thing that

you have to be watchful off because those custom headers can be used to host your own service if you are not very careful right so that is one of the things that you have to keep in mind the other thing of course is a lot of people uh you set up a lot deny lists for 169 254 or 127.0.0.1 they said our localhost they set up deny list so that if you use that in a web hook when you're setting up a web hook it would just not allow that to happen however remember our old friend dns rebind i can set up something.something.com and i can set that to 127.0.0.1 or 169.254 or 169.254. and

it'll still work so unless you have resolution level security where it doesn't it blocks the resolution when uh it encounters a local host or a deny listed uh uh then i listed ip address or host it would won't work but in in the cases where you're just doing a basic deny list it will not work it will still work because the attacker can find ways around this pretty easily as well so this is uh this is the whole scope of what you can do now defense the number one defense and literally the easiest defense for this is do not follow the redirect now a lot of http clients that you're using by default are configured to follow

redirects i know for a fact that python and i think some java very popular java clients do that by default they follow redirects you need to turn off that functionality right so you need to ensure that the client you're using does not follow redirects or you have to turn off the redirect when you are processing any urls especially as part of the web hooks right so whenever you're using a client to process web hooks always turn off allow redirects because this this attack works entirely because redirects are allowed right because redirects work this attack works if the redirects don't the attack just doesn't work so you have to ensure that that happens the second thing you want to do is i mean this is

slightly complicated stuff i would not recommend this but if you are not able to do the allow redirect deny redirect part of it some some some of our customers said you know what we can't do that we have to allow it because many times our customers need that to work so i in general i found that to be a little flimsy as an excuse but nevertheless if you can't for some reason not able to do that then you can set up the other stuff but believe me the other stuff is a lot more complicated than the first one if you can do the first one i highly recommend you do that because network security policy would mean you'd cut off

access to the metadata endpoint cut off access to any localhost services cut off access to internal network endpoints which is hard to do right because that service is probably meant to access those endpoints so you may not be able to do this as well but if you can great if you if you can't you might have to think of other options dns proxy or dns allow list when you're resolving dns to prevent against dns rebind that's something that you can potentially do as well you can add of course input validation to your webhook url and you can do an ip deny list based on these controls but remember the number one and the most effective way of

preventing this attack is the first one if you can do the first one i highly recommend you do it but if you can't then all the others are options but they are not very good options but they're imperfect options but they're options nevertheless right so that is basically what i have so conclusions web hooks are a really powerful way to integrate also a very powerful way to get attacked because of these chain of attacks look at it in your threat model because web hooks are seldom threat modeled have you i've not seen too many threat models that encounter or take into account web hook functionality everybody assumes that web hooks are just fire and forget mechanisms that

they'll never they'll never have to bother about later on but that's not true because you are still part of an http request chain and when that request chain can perform all these kind of shenanigans that's when you want to be a little careful right so uh think about it as part of your threat model and then incorporate it ssrf of course we've seen what ssrf can do so you definitely have to think about how ssrf can impact this and look at defense holistically and from multiple layers like i said the number one defense for this flaw is the redirecting i highly encourage you to do the other things as well if you can but remember ssrf is

harder to prevent because it depends on your use case depends on where you're redirecting depends on the scope of the redirect there are lots of depend situations that you'll encounter with ssrf so definitely the first one if you can do it that would be uh really uh the best way to defend against this sort of attack with that i come to the end of my talk the slides for this talk are in the bitly link and uh yeah this is uh you know my talk let me know if you do have any questions for me thank you sorry oh really okay i finished it quicker than i thought then okay all right all right thank you so much and thanks

for attending