Precision Munitions For Denial Of Service by Gerald Benischke

hello good afternoon um so as it's left for me to introduce myself so I quickly whipped up the slides there and well not really but essentially um I'd like to think of myself as an old kid mainly because I remember Jabba 1.0 um a sort of journey into security was through a long time of developing software I made software for a lot of companies a lot of banks a lot of payment systems a lot of front-ends middleware and it gets boring um I'm sure there's a good metaphor to be had in in banking um but I never forget the stories that a friend of mine told me about a router that was hung in mid-air and secured literally by gaffer tape now we might think that this sort of literal gaffer tape is something that doesn't happen all that often but trust me it does and the more critical the infrastructure the more gaffer tape is used at simpler times so more recently I've moved into the appsec area um and at the moment I work for a for the UK's leading tax collection or authority and helping them to secure over a thousand microservices and that becomes important because when we talk about it at scale so in this talk I'd like to talk about denial of service but not the kind of denial of service attacks where you get just get lots of clients and lots of packets and things stop working so I'm not talking about the the low orbit ion cannon to overwhelm the system with brute force and ignorance but more about sort of targeted strikes more cruise missiles than carpet bombing um I'll also be using the play framework as an example where there's a specific vulnerability um and I I'll talk about how modern infrastructure can actually amplify this type of attack and they'll have a few more interesting denial of service fixes so um on the previous slide I mentioned the low orbit ion candle so often used in denials distributed denial of service attacks but like I said this talk isn't about that so I like to stick with Star Wars um here I'm sort of likening the distributed denial of service attacks to the Imperial Fleet and of heavy lumbering Star Destroyers that will fire lots of shots and highlighting everything in front of them but not necessarily hitting anything and the the sort of precision pack is like your neighborhood friendly gang of rebels with their x-wings taking out the Death Star in a single shot so and this is my last bit of of Star Wars metaphor I promise and when you look at the architecture of modern systems um a lot of the time there's a Content delivery Network at the front then there's like a web application firewall and countless loads balances and proxies before a hdp request actually hits an application the kind of thing that the developer writes so the attack has to navigate this particular trench and get to that exhaust port on the hound before it can do any damage in my opinion that's why distributed denial of services attacks are often limited because all these systems in front of it and they're there to prevent bad things to get through so let's look at a specific example of how a relatively simple payload can arrive at its targets and do some serious damage so I've I've picked out this particular cve because I found it um and it's in the play framework and it's triggered by passing a payload like this through it's essentially valid Json it's a single element that has an array which contains an array which contains an array and then you do it at 30 000 times and you put a zero in it and you close them up so it's valid to Json um but it's the pausing of it actually caused the play framework some problems it specifically it ran out of memory and then because it ran out of memory it's um it basically fell over it sort of crashed and died in the Heap and returned to Center 503 so that's that's kind of neat um the interesting thing about that was that it happened in where the the passing of the payload happened before it even did any authentication past the payload fell over not really much that you can do to secure it except if you have a have something in front of it but we'll come on to that later the other interesting thing was that because the crashing and dying application instance was returning a HTTP 503 and the proxies and loads balances in front of it we're actually going oh something went wrong with that instance at best send it on to the next one and then the next one so one single payload that was sent in was killing one two three instances which is quite cool um now that has got a number of sort of impacts so first question is will will our infrastructure save us in this case because you know as we all know we've got lovely infrastructure that is self-healing and auto scaling and automatically restart things you know what's the what's the worst that can happen if um if you kill an instance of of your app server um so the first thing is of course you can keep doing it it's only a single request um your web application firewall is unlikely to pick that up because it's a single request and it might look a bit odd in the logs but it's a single request and this little one liner um we're talking about one line as earlier in the um in Tom's presentation and that one sends 50 requests and one every 10 seconds to to that particular payload so effectively what I'm doing is playing a game of whack-a-mole you know I can I can an instance comes up it down another one another one another one and when that is used in in a scenario where you might have you know 10 instances that provide a particular service then very quickly those 10 instances go down and if that is embedded in a hundred thousand requests it might not be easy to pick out well which ones are the ones that causing it [Music] um so it's it's a very interesting sort of vector this particular thing so then the question is what well if the infrastructure hasn't saved does a WAAF certainly will save us stands for web application firewall um you buy them from Amazon you buy them from cloudflare and they go They're going to inspect the payloads that is being sent through and depending on some rules they will deny a particular request so that good requests and the bad requests and the wife's will sort of try and filter out the bad requests the problem with every application firewalls is there's still use computes and typically they will only examine the first 8 or 16 kilobytes of a request so if you send a payload like this where you fill the first um eight kilobytes with something innocuous looking and then fill the rest of it through the payload will still get through the waft will be successfully bypassed and I still have my um denial of service now you could obviously restrict the size of the the payloads but then that gets to be very difficult because different end points want different payload sizes and you don't really know what the application is doing you don't have to keep in constant touch with the application developers to to ensure that they're not exceeding the payload sizes and and still something might get through um so while we're at the play Frame Works the play framework by default um allows 100 kilobyte payload so you deploy your application something like this um will be about 30 60 kilobyte um and it still gets through it's still damages your application so that there's going to be different things that will save us I'm guessing um so let's talk about um uh the software composition analysis or or just saying well you know we've identified a particular vulnerability and it's been given a CD it has been mitigated there is a new version out for play well there was last year's um and it's been given a score of 7.5 and hands up if you're checking every single CV in your organization who does that I didn't think so um it's a problem um because especially as um uh CDs with their scoring if it's not a remote code execution vulnerability it doesn't get more than 7.5 so this had 7.5 as the uh the CVSs score but that's the highest that you will get and whenever you're in a vendor presentation about the uh about a nice tool that that allows you to prevent your build pipelines to let you through bad stuff they talk about having policies that sets a maximum CV that it will ban well that will typically get through because otherwise you end up in a in a situation where you have got so many things to mitigate that all you'll be doing is mitigating and mitigating and and this so it's difficult to to for for uh software compositionalize your vulnerability scanners such as catch everything Because unless you're at a stage where you only got a very simple service where you only have got and have got the capacity to update all the dependencies all the time um it's not going to catch it and normally you don't have that capacity you will have like hundreds or thousands microservices that all need to be uh running through the build power plans tested ensured that um it's safe to to release so if you have if you have to deal with lots of CVS every week that's not going to happen right um so on to my next example of something bad that can happen um go to the go through the history books now this is a CD from despite the number it's from 2000 2013. again it has a has a score of 7.5 it's a it's an interesting um vulnerability because if you look at the XML payloads that I've sort of mentioned on the on the right hand side there and you've got basically a lot of elements and all these elements are empty but they have a name and that name has got the exact same Java hash code and you can generate them and if you have like a hundred thousand of them and then the in this particular instance the Xerxes passer tries to pass then even though it doesn't crash it just spends time and time um trying to do it because what happens is that because Xerxes uses a hash table internally it uses that to optimize the access to the storing the elements but because all of these elements have got the same hash code it acts more like a list and and when I tried this out I was able to keep a single request was able to keep an application busy for 10 15 20 minutes before it timed out and that then pegs that application at a 100 CPU now one time might be okay but then you do it another time and then you do it another time and then um the CPU is so busy passing the XML that's the end point that you've got for the health check doesn't return in time so your infrastructure starts killing the thing and again if you keep that running only a handful of requests can have a big impact to and to the things that that you're running so there's more of those of course you know you can find denial of service payloads um in lots and lots and lots of places and that's not even the code that you write yourself that is the code that you bring in through libraries and even harder I think is the code uh you write yourself ordered your developers right because as we all know um when software is developed there are lots of deadlines and pressure and stuff the product owner wants to have a feature delivered and can we just refactor this to deal with the technical that no there is in time can we put it on the backlog and it's never happen so a lot of these things might be hiding away the problem is that a yamo bomb is not much show up in your dependencies because your configuration is done using yaml but you're not very unlikely to to use yaml as user-controlled input so are you sure that that's the case so you're gonna have to look um a redos uh a regular expression uh denial of service uh I think Tom mentioned regular Expressions you know if you try to solve a problem using regular Expressions you know I have two problems and and that is one of them if you if you write it wrong then you could end up with something that again can let your CPU spin in for a good couple of minutes and the the old classic from I think 2003 the billion laugh attack um where this little um snippet of XML creates a gigabytes worth of memory now a lot of the time the services that you're running don't have a gigabytes worth of Mac Ram so the question here is um Addy's a problem if you're in a position where your service is used for authentication say it deals with saml you know I'll pity you if it does but um if it is if it is a component of your whole website or your your whole offering where a single service say the service deals with logging users in if somebody finds a denial of service vulnerability that they can trigger with a few requests and if there are lots and lots of requests going to that service but then suddenly no longer work because you've killed somebody's instances could be could be quite traumatic for your service you know if you if you imagine the services that are that are used for keeping borders and trade off open where calls need to be made and if that stops suddenly we've got lorries at Kent um yeah so the problem obviously and I it's CVS now this this was where I was working at one client I did a I was using a tool called x-ray which listed all the CVS and all the artifacts that we had and it said I had just shy of a Million worth of occurrences of CDs yeah nobody's gonna have the ability to look at those individually and unfortunately the problem is that a lot of the time cdes uh only valid in certain contexts say for example the there's a there's a CD in in log4j not the famous one another one where if you use a JMS surpender you can achieve an rce nobody I've ever seen no I've never seen a JMS offender used anywhere but it's still something that comes up in your uh in your list of CVS so you're going to have to have a look at all of them but who's going to have a look at at 800 000 different occurrences because a cve depends on the context that it's in so I actually did that I took those 800 000 and and I ended up creating some aggregation to sort of try and find well what are the ones that I'm interested in and then I started knocking them down and the actual number of CVS the different types of CVS was only 800. once I read through them all and understood how they worked because I understood how all the services worked I was able to say yeah that's applicable no that's not applicable yeah that's it but no that needs a bit more investigation um so it's it's about sort of creating a pipeline to look at your availabilities unfortunately the tooling that you'll get from the vendor you know it's great at bringing you lots of data but a lot of the time you still have to put in the work to to to make use of that data the fact that you know that you've got 700 000 cves is useless even if you know that you've got oh I've got 700 000 with a score of 9.8 [Music] because a lot of the time as an example the CVS with 9.8 are because they use something called polymorphic desalinization and if you don't use that you're not vulnerable so I will be would you be interested so you're gonna have to work out what your environment looks like what your services use is it going to need some kind of catalog a catalog can be a full blown tool it can be a spreadsheet and foreign but you're going to have to sort of automate some of this aggregation which sounds awfully like devops to me um which is good because that's where I work um right I'm coming to the end now I don't actually know what time it is but um the lessons for me here are this it is worth taking denial of service vulnerabilities seriously because they can be exploited and and sometimes when they are exploited you can do some real damage with it it is worth examining your content types and because if you don't use XML and payloads then why would you let it through um because a lot of Frameworks automatically accept different types of payloads and then converts between them and it is important to curate your own abilities to do that you're going to need somebody to look at them um which I think you're gonna have to do some sort of aggregation of the data that you're getting and and of course it's important to to have a lot of defenses and and layer them and I think through that you can get ahead of the these denial of services right and that just leaves me with questions it's always good when that happens it either means that I've talked I've mumbled or it's or have not had a point