You don't have to patch!

alrighty uh always good to start uh the talk immediately after the keynote with a slightly provocative title but guess what you don't need to patch uh hey Pedro and I are going to be talking about patching today um my name is jasphere I go by Jazz that's what you should call me I've worked with and for a large number of small and large tech companies around the Bay Area uh all the security all the time Scrambler I do appsec sometimes I do talks let me start with a question to you Jazz don't you think that some companies are obsessed with patching like that's the only thing they care about I know it just feels that way doesn't it like that

the only thing that one needs to do is to be patchfully and then everything else in life is going to be fine this talk was motivated by a um by a ciso that I worked with at a large tech company and he believed that every security decision should have a cost benefit analysis every decision should have a cost benefit analysis except patching patching is sacrosanct uh it not only has to be done it has to be done as soon as possible and you understand where he's coming from so the talk today is going to talk about why patching is hard why it's not always possible but also what else you can do in order to make it make the blast

radius from unpatched software be slightly smaller so that whilst you are racing to patch software um you're not vulnerable um I'm going to start with talking about the limitations of patching patching is hard right we all agree patching is hard not only does it take a long time to prepare a patch the tasks involved with testing the patch making sure that it's functional it doesn't break your software then it's performant and then after you've done all of that you still need to roll it out and sometimes you have a Reliance on vendors or third parties or sometimes even teams in your organization that you don't control and they don't believe the same way that you

do that this patch rolling out is super critical not only that once you've done all of these things after you've done all of the hard work you roll this patch into production it turns out the patch does not even fix the security vulnerability that you had in the first place right I am sure all of you will like not in smile yep you've all seen this happen um which brings me to the question like why do we end up patching and like every single talk and besides this year uh I turned to chat GPT and uh this was my attempt I was like hey can you just write this talk for me and even in AI

even in AI was like no dude the premise you're pushing is immoral um and and and and that makes sense right what Chad GPT is doing here is summarizing essentially the consensus view of the internet and the consensus view of the internet is that if you're not patching your software uh you're doing something immoral or or wrong and the reality I think you will realize by the at least by the end of the talk but maybe you already are there is that the reality is a lot more nuanced than that um so this brings me to the question like why do we think that we need to patch and the premise in security and in

Engineering in general is that the only way you can be secure is if you know what's happening in production and the idea is that if your software is unpatched if some attacker has gotten in there then the outcomes the actions that your production system is taking is unpredictable okay that makes sense but the reverse of this the thing that we extend too far is to say like oh okay that means that if if you're fully patched it's fully predictable if it's fully patched it's fully predictable and the more you think about this the more you realize how much of a fallacy that is right um I'm sure all of you have stories uh like this this is my story uh I used to

work at UH at Google where I was responsible for uh integrating a Dom parser an HTML browser uh called Xerxes and and Xerxes is amazing Google like many organizations that you guys work for has a one Library one version of a library uh is is allowed in all of your applications okay this worked brilliantly for the longest time uh unfortunately it turned out that in the same version that Xerxes fixed a very severe security vulnerability they also happened to remove functionality that I was relying on and this put me in a very difficult position right I had a choice to make either I uh allow the software that I know to be vulnerable to continue to run

in production whilst I'm chasing up and trying to re-implement the functionality that they re that they removed or I uh patch the software and then break my application and then my users have no access to the application whilst I'm racing to fix up this particular uh this particular case now in in my case uh somebody very senior at Google emailed my manager to help me make time to prioritize this which was very awkward for a very young Jazz but but the point here is that forced patching forces these kinds of very difficult choices on you even our friends in compliance understand that this is difficult companies companies have limited resources and uh even the compliance folk recognize that

in order to fix up the software patches need to be prioritized and another way of saying that is to say well we know that they are going to be patches that you know you need to apply that you have not got the limited that you have not got the resources to apply what that means is that sooner or later you're going you're going to have software running in production that is unpatched and that you know is vulnerable which which is which paints a very Bleak picture right this is a this is a very Bleak situation because it means that not because you're a bad security engineer not because your organization is neglectful but because you have

limited resources software is going to be unpatched what can you do about it what can we do what tricks can we have in our tool chest so that the extent to which we are vulnerable whilst we are unpatched is reduced and that's what we're going to be talking about today um if you think about it what is the point of patching the point of patching is to build this beautiful basket of uh of fruit right um and and and the problem is that every once in a while you have a rotten apple in the basket and that's going to spoil your entire basket of fruit and patching is supposed to fix up that one apple but

if you think about it in those terms you come to realize well there's actually a whole bunch of other things that we could be doing too right uh it turns out I don't like apples I just don't like apples and so you can remove the fruit from your basket that you don't like or don't want this is attack surface reduction not only that the fruit that you do like they can also spoil and you know spread like contagion throughout your entire fruit basket what you can do there is compartmentalize and isolate these fruits so that yes the fruit that are going to go bad are going to go bad but they don't end up spoiling the

entire batch the concrete example uh here is a tech surface reduction we have things like s-bomb that has been all the rage I think that that's all I've heard about for the last three years of b-sides uh software bill of materials basically says what it says that as long as I understand what software is going in to my application I can choose to remove the software that I don't need or remove the software that I don't want we have content security policy CSP for web applications does a very similar thing what does it say it says that I only want these libraries loaded from this location to be used in my application right just just the mangoes and bananas

for me uh nothing else right and the equivalent for that in web applications means that you can you don't need your web application to suddenly load the entire internet it just says that the only things that you can have are loaded from particular locations um the other thing to recognize is software processes software is not the only solution we can have processes as well I like to call this a CSO with a stick right if you have a person in charge who decides like no this is the quality of software these are the organizations this is the kind of a development process that she wants to use to load software into our application like that makes a dramatic

difference in reducing the attack surface area on the other hand you can do isolation you can do Network segmentation where you don't need to have every node in your Production Service connect to every other node you can use things like iptables to separate out systems similarly you can have system level sandboxing you can have things like SEC comp and what is it called app armor app armor to say what services an application can use or call the the thing to remember there is even though you don't have a perfect white list every time you remove a system call you remove some part of the vulnerable applications that a vulnerable system calls that an application can call and

then very close to my heart is a thing called virtualization virtualization means that you lift an application up and you replace functions that the application is calling with other functions where the effect of the function is the same but the function additionally goes ahead and logs blocks or attenuates the effect of the call of the functions that you're calling the thing to remember here is that in all of these cases you are reducing the blast radius of an attack of your vulnerable software but you're not necessarily having to immediately patch software we're going to go through a couple of practical examples uh uh starting with network isolation all of you are already familiar with this so I'm going to blaze

through this you have a load balancer that's able to access a web server or an API server each of these are able to connect to database uh if you see the load balancer directly accessing one of your databases you know something has gone wrong you've probably been popped you've probably been popped because you're running some unpatched software the ways to fix this you can either patch your software which is absolutely the thing that you should do but in the meantime you can also use Network segmentation you can use iptables to prevent Pathways throughout this entire process that are unexpected so the load balancer should only be able to talk to the web server or the API server and not

directly to the database in in these cases the software remains unpatched it remains vulnerable but the but the level of attack that the attacker can carry out is much more difficult so our second example is a system level isolation and for that we chosen image magic which is widely used by web apps and notoriously plagued by vulnerabilities so specifically selected a command injection vulnerability that was branded with the image tragic by the way don't you think that security people come up with the best name puns we really do right so image magic uses external programs to do some chores called delegates so delegates are configured with templates that tells what to do so in this example

curled is being used to fetch an image that is stored in the https address but what they forgot is the basic rule of web app security that you should never use and sanitize user controlled inputs in your application so this vulnerability allows someone to supply a command and the the result of executing that command will be returned to the attacker so let's see that in action we have a video so I'll be using the same example to list the current directory okay now let's run image magic and there you go you have an interesting file called secret text let's see if we can look inside by changing the command running that again and you can now let's find a more

interesting targets say Etc password yeah yeah all right I'm sorry you have to see me type very slowly okay there you go so it's not a surprise that it worked so this vulnerability was quickly patched and back then the recommended actions were to verify the output if it contained like the magic bytes known to be generated by the most common image file formats to disable unnecessary delegates and to patch and what are these These are stronger validation attack surface reduction and security updates so what else what else could you do it turns out that even then elkantov published a blog post saying that you should sandbox as well so we really meant SEC comp which is only like it

requires changing the source code so if you're just like a third party using image magic you wouldn't go that far but there's another option on the table it's up armor so if you don't know up armor it's an open source solution that allows you to basically block out access to files to network to other capabilities by default and then you have to explicitly re-enable the things that the application is expected to use this is the app armor profile that we'll be using to mitigate that vulnerability I'm not going over this whole configuration I'm just pointing out that it is very restrictive in what it will allow uh to read from the ETC directory okay so this is the demo so let's play

that out with the up armor profile in so we weren't able to see the contents of the the password file and you can see because it was too fast but basically the log file was showing that there was an attempt but it was denied the third and final example application isolation and for this we'll we crafted a very small web app it's basically a payment page that contains built-in access vulnerability it allows you to pass an xss Vector using a get parameter and it reflects that parameter back to the page so let's exploit this by supplying a multi-stage xss payload so the first stage is very simple it's just a script Source elements adding a script for

loading a script from baddomain.com badscrip.js and this allows the attacker to change using the second stage to change what gets injected dynamically and this is the second stage so it's very simple basically it sets an event handler for the on submit of the payment form so when the form is submitted iterates and collects every form field it outputs that to the console to prove that it got access and then it actually charts out the data to a malicious domain now I'm not showing you this attack you have to trust my word that this works well we're straight we're going straight to Mitigation Of This attack and for that we're using a web page Integrity or WPI which is a

client-side sandboxing solution that we have developed WPI allows the application owner to control what scripts see and do very much like app armor but for the browser it offers an expressive rules engine that allows granular control over a wide variety of behaviors you can do attack surface reduction by blocking out like everything that Europe patient will not use and you at the same time it allows to set boundaries using isolation rules to control or to prevent any scripts from touching critical parts of the application so next we're rolling out the rule that for bits or to blocks access to the form by any file but the process form.js which is not the attacking code at the

beginning you will see some data from a previous run where we were running in alert mode only never mind that okay so this is the video so first we have to fill in the form okay and remember like the rule is already running on the website and it failed to to leak the credit card holder data we jump to the drop server we don't have any new entry so it means that it failed to collect the data and in wpis dashboard we see five new alerts the first one is the addition of the second stage script and then four additional ones that represent the blocking of the access to the forms so the first one wasn't block we're just

seeing that it was added to the page let's look into the details you can see it's uh the credit card number from the credit card form we can see the source that initiated the action and you can send you can see details on the sensitive data so so in the previous example stage one and stage two were still being added to the page so we can do better we can roll out a new rule to forbid the addition of script Source elements to the Page by any script and doing that we will be able to block out the addition of the second stage script to the page okay so we have to do another run we'll have

to fill in the form again all right never mind the error we didn't bother to implement the post page and you will have a new alert here that confirms that we have successfully blocked the addition of the second stage so if we look the details you can see that you can see the the scripts the the domain where it would be loaded and the script that initiated that action okay so you have to remember we weren't we didn't patch the application right so just to summarize uh what you had here is uh using WPI and virtualization what you were able to do is prevent the exfiltration of data actually you to stop the attack entirely whilst the

application was unpatched presumably the application developers were racing to develop a patch but in the interim you were able to mitigate and stop the attack right that's what we exactly exactly so we're reaching the end of our talk and so patching remains an important part of every company's cyber security strategy but it's not always the best solution or the most practical there are alternatives like reducing the attack surface or isolation that can complement patching and sometimes save the day when patching will not you can think of other options I'm sure but hopefully by now we have convinced you that these two can be highly effective so if by leveraging these Alternatives and Implement them where appropriate

companies can be can have a various better security posture and be better prepared to to against like new threats coming in um anything that you want to add just these people are 25 minutes late to their patching [Laughter] he said oh your title of your talk is clickbait but this is not true right like if the if there's one takeaway that you have uh from this talk hopefully it's not that you don't need to patch but if you know you're talking to your friends and your peers over the next week and they're arguing that you absolutely have to patch now hopefully what you have arguments to show that uh using attack surface reduction and using isolation uh to reduce the blast

radius what you can do is recognize that there is a cost associated with patching and that blasts area reduction allows you to have the breathing room to be able to patch your software in time we are done thank you

we have time for a couple questions from the audience if you've got questions raise your hand and I will come to you

this is very Rock Style by the way I can't see any of you okay uh middle uh hi there so um I work at a large company and I'm in what we call the security embedded team which is the team that goes out and makes security nice and pleasant for the app teams that don't want to hear from us so what I'm wondering about is have you thought is it is this stuff about putting like app armor and Jace scrambler have you thought about like how we can deploy it at a global Enterprise level without pissing off everybody who's whose application doesn't quite conform and still have useful rules well actually there are vetted up armor

profiles that you can use out of the box that will not for sure like break and piss up other people yes and similarly for Jason yeah so if they are vetted if they are well tested like if it breaks maybe you want to check that out because it might be an exploitation or something camera thank you for the wonderful presentation I had a question about the the example the second example where all the form fields are stolen couldn't that be easier just be solved by implementing a Content security policy well the the CSP actually it's it's kind of a binary in that sense like if you accept a script to be included in that page it's you are giving out a blank

check because that script was allowed you didn't know that you would try to access the form Fields now that you gave access to that to that script it can collect data and we know of many examples where either an attack or just abusing their access scripts were caught like grabbing data when they shouldn't and this is not to dismiss CSP right so all of this talk is that what you can do is reduce the attack surface area or isolate and CSP does do a very good job in many cases of preventing arbitrary scripts from loading but the scripts that do load you require something more nuanced in order to say not only do I want to not want scripts to load from

this area but I want it to be the case that having loaded only the following apis uh uh are allowed just a quick compliment iccsp is removing the apples that you don't use so you don't like include the whole web but the ones that you do can still spoil the thing so it does not create isolation in that sense you know awesome thank you very much to both of you foreign