Precision Munitions For Denial Of Service - Gerald Benischke

hello um I'm Gerald um you are and you John very good hi John hi gr um my talk is um preposterously titled uh Precision Munitions for denial of service um okay sides stop working um so I'm old and grumpy I remember Java 1.0 coded in it I was amazed by Java 1.1 I've um had a long career of developing software making software um I all you through all the all the details um and it's led me to finding problems breaking software and now I do it do it sort of professionally um the apps that LE for hmlc digits um but I've got many hats so this talk is about denial of service but not the distributed kind I'm

not interested in a lot of bots overwhelming your your website so Precision strikes over C bombing um I'm going to use the play framework as a an example uh mainly because I work with it found some problems with it uh then I'm going to talk through something I called automatic amplification uh and maybe some some lessons and and what we can do to to prevent it so like I said it's denial of service not distributed denial of service so I'm not the low orbit iron Cannon of anonymous Fame I'm not the this Imperial Star Fleet just firing lots of things and things and hoping that stuff breaks it's all about being small targeted precise having a massive

effect so uh to do that um I'll sort of jump in and don't worry the Star Wars metaphors will stop uh stop soon but if we have at modern infrastructure typically your service um is behind a number of layers of infrastructure you've got your content delivery networks you've got your web application firewalls you've got your load balancers and all of those um essentially you have to navigate in order to deliver um denial of service attack and a lot of these bits of infrastructure are precisely designed to stop lots of brute force and ignorance from getting through now this is the vulnerability that that I found and it's quite an interesting one because when I send a Json payload which

is what you see in the uh the little thing here which is a valid piece of Json uh but I just have an array inside an array inside an array inside an array and I've got about 30,000 open square brackets and 30,000 closing Square brackets and what happened was that this payload when that got passed by the play framework which is an an application framework it it blew up because the way that it was passing was not um clever enough to sort of say yeah if you if you have a nested array in an array in an array of like 30,000 things it's it's going to it's not really ideal so um so I took that

and uh I found locally when I tried it out and was playing with it it it it blew up the the the the system the application instance just crashed and stopped it's quite an efficient way of denial of service if you can shut somebody's service down now if we think back to the modern infrastructure usually you don't have just a single instance room uh and this is where this comes into effect so when I put my payload and deployed it against um an actual test system which was on test infrastructure which had got the uh CD and the web application firewall the loer balances in front of it what happened is I blew up three instead of one by by sending a

single payload and what happens is that the load balancer goes and says right I'm going to send that payload to the application the application blows up and returns uh an http503 which basically means some things went wrong and in this case the load balancer by default was configured to say well this is a something went horribly wrong some just going to send the requests to another instance of the application blew that up and then the next one um and then it kind of went yeah all right I'm going to give up but it's it's a single request that blows up three application instances um so that's quite quite cool so then I thought well does it

actually matter single request blowing up three instances typically we run in kubernetes clusters or Cloud infrastructure and that kind of stuff is selfhealing uh you know something blows up it spins up another instance that blows up it spins up another instance we've got you know selfhealing infrastructure we've got automatic restarts we've got Al scaling groups but the the interesting thing is those bringing up another instance isn't an instant thing it takes a little while you know you have to provision a new bit of um a container has to be deployed um and it has to come up now where I was that sort of took a minute and you also have to think that a container is not

always detected to be nonfunctioning straight away typically what you've got is you send sort of little pings and if three of them fail in a row then it goes yeah that's dead I'm going to replace it and what what I then did I I wrote this little bash script which just basically send 50 requests uh one every 10 seconds and if you think that each one of those request takes down three instances over a minute I can kill 150 instances which typically you don't have that many instances and I can bring instances down quicker than you can bring them back up again so it's it's game of whacko so infrastructure isn't going to help us

here um but what about the W um you know surely if I've got a malicious payload like that I can I can protect myself against it well one of the things about waps is they need to inspect every request and in order to do that they've only got a limited amount of computational power so what they tend to do is come in uh what the tend to do yes please sit down uh what they tend to do is they take only the first few kilobytes I think in awf by default it's the first 8 kilobytes uh in aai it's the first 16 kilobytes and look at the first thing and the rest of the request is not

inspected so if I modify my malicious payload and and take that Jason and put something an innocuous looking string that's about 8 Koby or 16 Koby long at the front of it and then have my malicious Jason it'll go straight past the wff so I'm not saying that um a WF is something that is not sorry all right um so I'm not saying that a web application pable is something that won't help but it won't help on its own it's a it's a nice layer of fence so going back the the sort of vulnerability that I found um got it cve it got assigned a uh CVSs score and obviously you know we're all running

proper dependency scanners um and we're all looking at all of the uh cves that are being yeah well no not the scale of looking at every single CV is a bit of a problem and the fact is because it's a denial of service vulnerability the maximum CVSs score that it can get is 7.5 on a scale of 0 to 10 and a lot of the dependency scanners that sort of proport to say well you know we we we do coding that secure by Design um they sort of advocate for sort of saying okay well we know that there are lots of vulnerabilities and you can't look at all of them so how about you put in a policy to say that your

build gets broken if there's a vulnerability of score higher than eight well that'll get through so the question is as a as a Defender do you look at every single CD now that was one example there are other fun examples um and this was one that that I quite liked now this is an XML vulnerability and it's allows the remote attack to cause a denial of service via crafted messages and that's what a message looks like you basically send through lots of XML uh tags that all turn out to be the same hash code um which then means that underly the processing it kind of uses more and more and more time to process this and if you make this

request big enough say approaching a megabyte then you can kill something quite efficiently and the interesting thing was so this is a vulnerability from the year 2012 so you know more than 10 years ago uh and then you look at how many things still use it and there's like so this was the vulnerable version and then if you look at the number of downloads lots of software still uses that old thing because this had a rating of 7.5 again so lots of people kind of go I'm not worried too worried about that remember if you can kill somebody's instances if you can completely deny let's say your login service and nobody can log into your system anymore well maybe may maybe

something to to look at um so this is what um the CPU usage looked like by by sending 10 of those malicious hash collisions it just eats up the thing and if I keep sending it a relatively low volume can kill the Surs now there are other fun things in there um there is things like a a yaml bomb or the billion laugh attack or zip forms or probably one of my favorites the regular expression denial of service reason why they're one of my favorites is because regular expression denial of service um gets usually gets a science a score like 5.3 so way down and a lot of people just don't look at it but the impact can be the same thing

if you can deny I assistant from running for long enough then the infrastructure will go oh that's dead because the pings that are being made will no longer respond in time so the instance will get killed now the other problem is um is a problem of scale typically when you've got um uh Services you don't have one or two or three 20 Services you've got hundreds or thousands of services now this is a intentionally Badly Drawn diagram of of some infrastructure that I that I'm working with at a at a well-known tax collecting agency of the UK which shall the name remain nameless p m um and these are all the the systems that talk to each other and

this is only in in like the the front end bit now all all of these have got potential vulnerabilities in their dependency chain um and that's what makes it tricky can you actually look at all of them um if you maybe think of your organizations or uh places that you you know of Is it feasible to look at all of those vulnerabilities and and mitigate them all um so when I did that that's the said unnamed tax agency the first time I ran uh a vulnerability scan it came back with 836 uh 486 results so this these of course were a lot of the same [Music] ones but a lot of the problem is that things

depend on context when there's something that's a vulnerability and it said for example a an XML one well if I don't use XML then I can ignore that but am I really going to be able to to look at all of those and I've come to the conclusion that unfortunately um the answer will have to be yes because in order to assess whether something is vulnerable or not the only way that you can do is eyeball it can say well how is the particular thing being used um now what helps if the services that you deploy are sort of using this idea of paved Road I think that was that term was initially coined in sort of Netflix

and essentially the idea is that you uh put in ways that you build your services and that are fairly opinionated and follow the same patterns so that rather than having you know five different languages and 20 into different Frameworks uh in a plethora of of sort of services you'd never be able to get a handle on what kind of uh Deni of service vulnerabilities are hiding there but if you have an opinionated platform a way of Building Services that is fairly consistent then I think youve got a chance and um got no idea how we're doing for time Lots okay on um so that's that's what I ended up doing and one of the most fun things is as a

as a developer as a security person I found that my favorite tool are spreadsheets essentially to have those 800,000 things I couldn't look at them all obviously but if I build myself a pipeline that's sort of looks at well what are the common ones how are they being used where are they being used and create a big spreadsheet I have a chance to actually look at all of them and that's what I ended up doing I think over the last in the last year I looked at about 700 vulnerabilities uh assessed them and checked how they would fit in and the interesting thing is a great many of them were just out of scope but just not

applicable a great many of the ones with sex cve scores as well the ones that have got 9.8 and I looked at it and okay we're not using it in that way so I can ignore it um I don't have to update that dependency which is all right I could do I could just keep all dependencies up to date um if I if I had 10 services or 20 but if I'm talking a thousand services that becomes more and more complicated yes maybe maybe I'm not the best at graphic design but um so what we do at said unnamed uh UK tax collecting agency is we've come to the realization that um application security or a security team

and the team that is developing Services they can't do the job on their own there's no security team big enough that can go around an organization fix all the problems or analyze all the problems so it needs cooperation and it's it's one of my other passions one is absc obviously that's why I'm here talking the other one is is agile software development and I think that's where absc if it turns agile which means that you you uh look at uh collaborating with people that you look at fast feedback loops that you look at iterating um makes a difference here and that's where I think a security team can specialize into analyzing vulnerabilities making recommendations the service teams can

provide context um and fix the issues and you know ask questions question s so that's I think a lot of leg work it's one things where AI won't save us um but working together is the way out of the denial of service so I'm quite aware that about most people have missed the first third of the talk um so I don't know whether you want me to go back a bit or whether I'll just finish up deal

Choice okay I might I might just go back and just um say that this this was the the the the attack initially the idea was that this is a valid Json payload it's got an array and an array and an array and array if I've got 30,000 Open brackets 30,000 closing brackets um I can blow up a single system um modern infrastructure can amplify this so rather than blowing up one system the load balancer will resend a request which then engages in the Giant game of wack and ball and that's how you can deny a service quite efficiently so back to the lessons I think far too often denial of service is not taking seriously enough because it's

like well I can't achieve remote codes uh execution so why should I worry about it but we we spend a huge amount of money throwing it at web application firewalls throwing it at dos protection to protect us from denial of service and I think we need to do the same thing for for those Precision strikes for those targeted denial of service attacks which um I think will mean that we'll have to look at all of our vulnerabilities in order to to sort of get a clear picture of what's vulnerable um scale is always a problem and the solution I think is mainly collaboration making sure that a security team knows what the developers doing and the

developers do uh know what the security team is doing that there isn't like um you don't want a development process where the Gods in this in the ivory security Tower come down right at the end of the life cycle and said no you have got a vulnerability you can't go to production and then they go yeah but we need to cuz there's a deadline and we we'll just sign off the risk uh a paved Road and layer defenses are are also something really useful and that brings me to the end of my talk that I've given it one and a half times um I'll um I'm very happy to to give the talk again and again and again um outside and

I'll I'll talk you through it um other than that any questions the CV that you found yeah was there anything that would have um help to prevent it so let's say I'm using that framework at a minute I it what if anything could help um so the interesting thing was and I probably should have mentioned this um the the interesting thing when using that framework was that the the processing happened before it even reaches your own code um and the framework had some logic in there to sort of say depending on what content type to send um if I don't process uh Json and I'm expecting um form encoded URLs um I try to convert between the two

so the problem was that there was a a website that accepts post requests of forms and you send it Jason and the framework goes away and says oh I'm going to try to convert this oh oh and there looks to be an array and that that's when it sort of started to blow up because it tried to convert that Json into a query parameter name and so it's it's it is about this idea of being aware of what what content types you accept because if um if you put some logic in your load balancer or your CDN to say I don't want to deal with Jason requests then that would protect you because for for that particular thing

but it's always difficult because there's so many different variations what was the uh what was the back end toist that's just play play framework oh understood understood so is that his own sort of language or of sorts or is that Scala oh understood okay play framework is Scala yes got uh which is a jbm based uh thing got it got it and you other questions brilliant well um uh John and Dr I I introduced myself personally for two people so I hope to catch up with you with all of you in the bit