You Don't Have to Patch!

so let's move forward with the next session by Pedro and jazir uh one one small note I was uh during the opening session I was saying that the final keyn note unfortunately Adrien cannot be here he was for personal reasons couldn't travel um but he just confirmed that he will deliver it we'll do it remotely and hopefully it will work okay uh without further Ado so ped and J so you don't have to patch which is great news thank you okay all right um guess what everyone you don't have to [Applause] patch that's it we can go so today we're going to be talking uh about patching um my name is uh Jazz spere most people call me Jazz you

should too I've worked at a wide variety of small work for and with a large variety of small and large uh tech companies and startups uh always in security all the time and I am pet Fortuna I'm CTO and co-founder of J Scrambler I also co-lead the local OAS Lisbon chapter um I've been working in security for many years I do absc mostly and sometimes I do talks so so J don't you think that some companies are like obsessed with patching like that's the only thing they care about it does feel like that sometimes doesn't it as if the only thing that if we got right if we were fully patched life would be beautiful in this talk I want to talk a

little bit about why patching is not the panace that it is hard to do um and that even when we are fully patched they are not only is it hard to do sometimes it is really difficult to roll out in time and what else we can do what other tools we can have in our tool chest to mitigate the risk of running unpatched software but before I do that uh if I can entice you to to hit that QR code so that we can have a survey of how you feel about um how we go about protecting software today uh uninfluenced by the talk so far well that that sounds a little bit pretentious you know but yeah

please please help us out here can take a couple of seconds yeah it's just one question and I'm I'm sure you thinking like oh I probably would take all of them with me to a desert island Island but that's those are not the rules of the game so if you're going to a desert island you are definitely taking patching with you and there's one other thing just one other thing just one other thing that you need to take so you have to make a hard choice and by the end of the talk we'll we'll revisit the results with you and we'll see what most of the audience thinks about this all right on with the show

um patching is hard right we all agree it takes a long time to create a patch to test it to make sure that it is functional to make sure that it's performance um to go through quality assurance and then after you've done all of this hard work you still have to roll the patch out uh and sometimes you're dependent on other teams sometimes you're dependent on vendors and then you go through all of this negotiation and then you finally roll the patch out into production and even then after you've done all of that hard work sometimes the patch does not even fix the security bug that you have had have you had that situation oh okay only a couple of you

maybe you guys work with much better team than I do uh I have some numbers to share with you so um on average uh it takes 215 days uh for a patch to be rolled out 215 days it gets better if the patch is security critical it only takes 181 days a reduction of like almost nothing right 30 or 40 days it still takes such a long time for patches to be rolled into production it's interesting to note why the criticality of a patch does not seem to make a huge difference on how long a patch takes to land a couple of nights ago I was talking to denish Cruz and he pointed out something

that I had not thought about uh which was he uses how long it takes for a patch a security patch to land as a bellweather for uh how good a how good an engineering team is and not all engineering teams are a teams sometimes we have to work with uh whoever we are working with and not everybody is able to roll patches out uh quickly so why do we patch this is a a a question to start with and like many of the talks here U today and actually over the last year I turned to chat GPT I asked it hey can you write this talk for me and even an AI said that the premise

of this talk is too immoral for it to deliver uh too immoral even for an AI but you can understand why right uh what chat GP PT is doing is giving the consensus view of the internet and the consensus view of the internet is that patching is sacent I used to work with a ciso uh a little while ago and he used to say that you have to do cost benefit analysis of every security decision you take but not patching because patching is a must I think that that is a mistake that I would like to think uh I would like for us to think a little bit harder about so why do we patch um um there's a

concept in engineering that suggests that if if your software running in production is unpatched then it could be doing anything then you have no idea what is happening in production that makes sense but the converse of it is something that we sometimes implicitly come to believe too it is that if your software is fully patched in production then you know everything that is happening in production and if I say it like this and if you think about it you obviously know that that cannot possibly be true right but it feels true I'm sure all of you have a story like this this is my story um back when I worked at Google I used to be responsible for I used to own

the integration of a Dom ping Library called lib Xerxes and uh Xerxes One Fine Monday rolled out a security patch but in the same in the same update that they rolled out a security patch they also removed functionality that my software was reliant on and now I was stuck either I had to allow the vulnerable software software I knew that was vulnerable to continue to run in production while I raced along trying to reimplement the missing functionality or I updated the software I fixed the vulnerability and then left my software broken again until I uh implemented the missing piece forced software upgrades forced software patching forces you to make these kinds of choices in my particular

case a senior exec at uh at Google emailed my manager to help me make time uh to implement the fix yeah we know how that ended yeah uh all right even even our friends in compliance understand that patching can be difficult and takes time uh the idea is that software resource it takes software resources it takes human resources in order to create and roll out a patch and our resources are limited it doesn't mean just because so it just means that there will be times when software you know is vulnerable continues to be running in production whilst you are building or testing or rolling out a patch not because you're a bad security engineer not because your organization

is neglectful but because software takes time the question is what can you do to mitigate the blast radius of continuing to run vulnerable software whilst you are doing this this all seems pretty bleak uh so what can we do about it uh I think about I think about patching as creating this beautiful basket of fruit you have your apples your mangoes your bananas but sometimes an apple can go bad and then lead to the entire basket of fruit spoiling in this case what would you do about it well it turns out I don't even like apples I don't know about you it tastes kind of weird uh so we don't even have to have them in the basket right and

this is the equivalent of ax surface reduction what you can do is remove those parts of our software those parts of our fruit basket that we don't need or want necessarily you can use the help of esbal for that that's a good point uh the other thing that can happen is fruit can spoil even the fruit that you like can spoil and then they can result in Contagion throughout the uh fruit basket in this particular case you can compartmentalize your fruit and separate them away the fruit are still going to go bad but because they are isolated the Badness is contained to one particular fruit or one particular compartment now I'm going to give you concrete examples

in software for what this looks like as uh Pedro said uh attack surface reduction one of the mechanisms that we have is es bomb we had the daddy of es bomb give us uh a keynote uh today what does esom say esom is software bill of materials it says if I understand the software that that is going into my application I can choose to remove software that I don't need or want there's content security policy used by web applications uh CSP allows you to say I just want these particular libraries just the mangoes and bananas for me and nothing else there's no need for your application to be allowed to load up the entire internet and then the third thing

that we can do is a non-technology solution we can have processes uh like uh well what we call ceso with a stick uh where we come up with rules about H what kinds of software we can choose to have or policies about how code review occurs in my particular case back at Google we had a policy of only one Library one version of a library uh in our third party at a time the other technique that we have is uh is isolation uh isolation just means separating out different parts uh of our application uh Network segmentation is something I'm sure you all are familiar with where not all nodes in your in your production system need to be able to

talk to every other node in your production system only the ones that they need to get their job done similarly at the system level we can make sure that we use things like SE comp and app armor to prevent arbitrary system calls from being callable only the functions that an application needs in order to run and finally kind of close to our heart is virtualization I don't want you to think about virtual machines what virtualization does is it lifts up the application and then replaces each individual function or import functions with equivalent functions that do exactly the same thing but in addition are able to do logging or prevent inputs or attenuate the outputs so that you

control how much access these particular functions have these things together mean that the extent of the blast radius of a vulnerability of a unpatched piece of software remains as contained as possible I'm going to shoot through a couple of practical examples with concrete examples starting with network isolation I'm going to blaze through this because most of you are already kind of familiar with this uh ordinarily you have a an an an access that is occurring it connects to a load balancer and the load balancer in normal behavior either connects to a web server or an API server and that web server connects to respective databases sounds good pretty good if one day you come in and

you see logs indicating that your load balancer is suddenly talking to your individual databases you know you've probably been popped right uh probably because you were running some unpatched software now you could p and you could and you should patch that software but if you have Network rules IP table rules that make sure that the load balancer is only able to talk to the uh to the nodes that it needs in order to get the job done the work that an attacker needs to do to take advantage to exploit their vulnerability becomes a lot harder on them ped the second example is a system level isolation example and for that oh uh and for that

we picked uh image magic which is um a widely used software to make to automate image edits mostly used by websites but because it's not notoriously plagued by vulnerabilities so we specifically picked one called image tragic by the way don't you think that security people come up with the best name puns they really do I I do think that uh this is a command injection but before that let me explain how image magic works Works uh it uses external programs called delegates to do its chores it's basically a template so you you see this uh tiny example here it's using curl to fetch any image stored in the https location but what they forgot was the

basic rule of web application security that you should never use and sanitized user provided input in your application so they forgot about that and it turns out like people can feed a command and whatever the the result of that command will be returned to the attacker okay so this was a big one so let's see that in in action okay so a little time for a demo I'm I'm showing ex the exact same example so I'm I'm just like listing the contents of One Directory and just running convert and as you see you see the like the listing oh secret. text we might want to look inside tasty things all right okay it's just one line

of text we can probably find something more appealing like u center something password Etc password Ah that's a good idea uh I'm sorry you have to watch me type very slowly and here you are okay so this is a like most of you are pretty familiar with command injection so this should be easy all right so obviously uh it was quickly patched um or it was a a fix was quickly issued it's not the same thing um and apart from that they recommended that you need to verify if the output looks like the image type that you're were expecting you should disable unnecessary delegations those like you can think of those like plugins H and

Patch all right and what are these These are stronger validation um ATT tax surface reduction and security updates what else what else can we do here it turns out like even then elant um published this blog posts um advising people to to use boxing now he really meant SEC comp uh if you know SE comp you have to recompile the software so if you're using um image magic as a third party um dependency you will probably not go that far of recompiling all of that uh but there's another option in the table on the table it's called app armor it's um open source Linux um control uh solution that basically take takes out all of the the

capabilities um made available to to to software like uh the use of the network access to files other types of of um devices on on the Linux system and then you have to explicitly reenable the things and only the things that the application will actually need when running normally not in exploitation mode of course so you have a you have a small example here if you can see it it's it's enabling the use of the network and read and write permission to slome slfu for Firefox so if you run that app armor profile Firefox will actually not run properly because it needs more things but it's a small enough example for you to to follow okay

I'll give you a slightly more complicated one so we rolled out this up armor profile to wrap image magic around I'm not not going over like all of this I'll just point out that it it is indeed very restrictive in regards to what what image magic can access under the ETC directory and it it actually needs a lot of stuff I was surprised with that but so let's see that demo again this time around with the up armor uh enabled okay so we just run the program again and as you can see uh here in down in in the right it was Deni so the image magic asked for permission to the ATC password and it

was denied so it's not a surprise that we see permission denied over there the exploitation was not possible due to the uh isolation provided by app armor you should not try to write your own app armor profiles that they are difficult to get uh you never know what the application really needs it might not be showing today it might show tomorrow you might break software so you should use uh industry vetted app armor profiles and a good place to start is go to the official repo on gitlab and and start from there okay they have app armor profiles for most things that people use these days okay so there's it's there's no reason why you shouldn't

wrap everything that you're running inside an N AR profile even if you're running those inside Docker containers or whatever like it should still use up armor profiles if you can the third example is an application isolation example and for that we built like this very small web app which is vulnerable to an xss basically takes an xss Vector uh using a gap parameter and then that gets reflected to the body of the p page so obviously this is a pretty simple xss and to exploit it we uh we are using a multi-stage uh xss payload so the first stage is just this script element Source uh we're just adding uh and loading uh the second stage from bom.com

script.js uh but so this way the attacker can change the the injection the second stage at any given moment because they they control the the bat domain.com and also they limit the amount of characters that are being passed to the G parameter so as as you know uh if if the get if the the URL gets like too big it kind of sticks out in the in the application locks so it might might might be easier for people to to discover the exploitation the second stage is oh okay is this one so essentially it adds a listener to the form submission so that page you have you saw it's a payment page so it's uh hooking into the

onsubmit event of that form it's iterating every form field collecting whatever the user has typed into the that form then it's pasting what it got to the console just to Pro prove that it had successfully accessed the information and and last it xhrs out information to uh an attacker control dashboard now before I I'll I'll show you the attack playing out in a second but before that let me talk about how we are going to mitigate it and for that we're using web page Integrity or WPI uh which is a client side sandboxing solution that we have developed uh it allows um application owners to control what scripts see and what they can do uh

much like app armor but for the browser okay it offers an expressive rules engine um with which um basically you can control a wide variety of behaviors like the ones that you see here uh with it you can you can do ATT tax surface reduction because you can block out like the behaviors that are not needed or will not be needed by the application and you can also set boundaries or isolation rules to to basically prevent scripts from ever touching critical parts of your application so we're going for this one we're we we we we broke the demo in three small sets the first one is really easy what we're doing is we're rolling out this rule that it's not blocking

anything right now it's just like monitoring the network triggered events related with script Source elements and any type of networking event related with xhr Okay so let's see that this is the xss vector I've load the page with it and now I need to fill the the the form we're already seeing one thing being alerted to the WPI dashboard this is the loading of the second stage scripts which is already already hit the page we can see the domain we can see like the the signature of the the get URL and obviously the injection is there so and we're not blocking anything so I will be filling out the form and then we'll see if the second stage was able

to collect the data from the forms and as you can see in the bottom it was which is not surprising we were expecting this the drop server the attacker dashboard has one new entry okay so exfiltration was successful as well and we have one new alert related with the exfiltration so we see we can see the end point we can see the method xhr and we can see also the the the script that trigger the action the specific action that led to the exfiltration all right this one is done so the next one we are rolling out all right there's some delay between my screen and that screen we're rolling out uh a second rule that will start

blocking we'll start blocking access to the form okay so this is one of the code behaviors that we're seeing uh we're not blocking the script from being loaded what we're blocking is access to the form actually what we're doing is only allowing access to the form to a very like specific script that we're expecting to be like dealing with the data being entered on that script which is process form okay everything else will be blocked by default let's see how that works let's continue obviously you have there's a boring part here which I have to fill the form again maybe I can jump a little bit let's see okay I'm sure you appreciate and as you

can see like it wasn't able to capture the data on the form logically we don't have a new entry on the drop server and here on the dashboard we have five new alerts the first one is the addition of the second stage script and four more each per form field that was um try that that the script try to access but couldn't so we have confirmation that it was blocked let's look inside one of these you have information of what form uh the form field the specific type of information and again what was the script that initiated this particular action but the bad script is still loading right yeah so that's the problem that that's what we are blocking here uh

so we'll specifically Rule and new rule that will block any networking event caused by adding a script Source element to the page okay so in this particular example this the page is so simple that no valid reason to do that exists so we can just make the rule generic we could make it more specific if we need to on a real case but in uh here it's not needed just kept it simple for you guys I'm not all right so we need to fill in the form again let me do the same thing all right and what let me pause here um so this is the page that comes after uh the form post of the payment form uh we're

sorry we didn't implement this page because this is a demo okay don't judge please Jud we're lazy people um but it's enough to show you what what what is going on so we have we see the requests of the second stage and we successfully blocked that request all right let's take a peek inside it's uh very much like the previous examples so you have some data on the event that was uh blocked and um and yeah so this is the whole so so just so that I can summarize uh what you were able to do is you were able to set up rules to alert when a bad thing was happening to stop the exfiltration of

data and uh to eventually just block entirely the loading of those particular scripts on a vulnerable page whilst the software remained unpatched presumably whilst the team would be racing to patch up that software so this is a mitigation this is a way of mitigating the effect of that unpatched software and if we think about it the same thing is true in the network case and in the app armmer case as well where what we're doing is yes the software remains vulnerable but the the blast radius of uh of the software is uh reduced whilst the team is trying to patch the software cool thank you for that Jaz and we are breaching the end of our talk before

that we promised you to show you the results all of of this survey let's see how many of you actually responded it all right so it seems like we have six responses oh you might be lazier than we are um I wasn't expecting that anyway uh we have six responses we have um all right so so the the one that people picked the most was the tax surface production but let's start from the top we have strong authentication which obviously it's it's very important for security in general but in regards to preventing exploitation it it will not add as much as other options that we see here um then we have uh in the bottom

vulnerability scans s and dust those are great to find vulnerabilities but uh fully automated like vulnerability scanning uh they will not find like the the hard bug usually so it's it's hard for fully automated to work that way so you will not find all the bugs okay so you're still like putting out oh another one thank you whoever like just responded to this thank you um you're not catching everything uh then you have attack surface reduction which is great like you can remove uh things that you are not using and therefore any vulnerability existing in those parts will not affect you because they are not in production this is great but again the the parts that you you do need might

have uh vulnerabilities they might be exploitable I'm sure so it just helps but not much and we believe that the one that we would take to a desert island along if it was the only thing we could take if if if it was the only thing uh apart from along with with a along along with um with patching would be the isolation mechanisms which uh is getting fractioned how much time do we have we might be able to okay okay now it's a game um so so for us it would be those two and and and uh we could live um happy ever uh after in the desert island just mangoes and bananas for us you know

um or right you you can still keep going we need we need one more all right so while while patching remains an important part of every cyber security um security strategy is not always the best option or the most practical um so there are alternatives like attack surface reduction and isolation that can complement patching uh when patching will not save the day they might uh you can think of other Alternatives too I I'm I'm sure but uh by now hopefully we have convinced you that these two can be highly uh effective so when by leveraging these Alternatives and implementing them where appropriate uh companies can have a better security posture and be better prepared against cyber security uh

threats anything that you'll like to add here because these people are already like 40 minutes late to their

patching it's funny to stand in front of a security conference and tell people you don't need to patch I will tell you that um I hope that that's not the message that you take away from today's talk we are pro patching I believe in patching in fact last last night Joshua our keynote speaker cornered me and said I hope you make an exception for critical infrastructure people die if software is unpatched and that sort of hits you right but that's not what I'm saying that's not what we are saying we're saying that patching is hard and that rolling it out is hard and then there's gaps there's periods of time whilst your software will be unpatched

again as I say not because you're a bad security engineer not because your organization is neglectful but because rolling out software is difficult particularly if lives are at stake but actually for all of us we need other tools in our toolbox to make sure that the blast radius of software remains contained and we believe that a tax surface reduction and isolation as well as other kinds of processes are a necessary part patching is great but if but but you do not need to patch immediately if you take all of these other steps um in addition thank [Applause] you any questions we have a few minutes

hello and thank you for your talk um I'm sorry if I'm making any confusion here but the web integrity that you are proposing is it to same that Google is proposing and Mozilla is strongly against because it is a gatekeeper for the web or it is a different mechanism I couldn't understand if it is the same or if it is different well I can answer then I'll I'll throw it to you uh so we're not proposing a specific uh mechanism for let's say isolation Okay so we're proposing generically the use of isolation Google or Firefox or mozzilla they control the browser so there's definitely some amount of isolation that they can bring and bake in into the browser and this guy

actually knows a lot about that uh but for instance we talked about application Level isolation that could never come from Google or browser makers no we talked about uh SE comp uh could could could be delivered by by the software uh maker uh we talked about app armor uh probably the secops is the best point like those are the guys that will decide to use that type of tool so in general we our our message is more generic than that not a particular solution that you should use but to use isolation to use attack surface reduction I I I think that that's exactly where I would go we have solution we have specific examples of solutions in particular spaces but

the you know there are other mechanisms with uh with other parts of software where a similar kind of idea can be applied anytime we can make it so that uh a uh what used to be a monolithic application is componentized did I pronounce that right uh then you know you can make sure that contagion is limited but for the case of the web it was web integrity from the Google right web page Integrity the one that you show for the forums so that that's an application Level isolation so it's needs to be shipped by the application owner not by the browser in fact Google had a proposal that could be confused with this type of

approach called uh web integrity web integrity yes um and I think it just went away so they dropped it and there's some some things there are some things that the browser uh will be reluctant to do on the browser level because it might be perisis coming from the browser maker that you can do on the application Level and that's like the solution that we showcase is one example of that anyone

else okay Pedro okay Jess thank you so much thank [Applause] you