Patching Monthly IMPOSSIBLE, Maintaining Compliance Is Possible

[Music] higher performance communication resources and faster and more reliable technology higher performance communication resources and fting to talking about patch management woohoo uh think some of you would rather be going to a dentist or or a root canal sounds pretty appealing at this point but I am going to talk about patch management and a new way that that a new methodology that I've used in the past that's been very successful on how to do patch management a little bit

different uh so here's a kind of an agenda of what we're going to walk through little bit about myself okay uh I work for protiv and protiv offers uh we the old Arthur Anderson we we traditionally do audit and Consulting Services we have an it uh part that uh handles it security so I a qsa auditor I go out and do security assessments we do n cyber security framework uh and then we also help you with remediation

so I come from I'm kind of new to the world of a Consulting and in my role in past organizations I was over patch management and some of the organizations I had over 6,000 servers that I had to that I was in charge of keeping of running a patch and vulnerability Management program for as you can imagine patching becomes a full-time job and when youever you say the word patching to assist admin all of a sudden he's going to have flashbacks of late nights energy drinks at 3 a.m. and a nightmare of having to roll patches back because stuff broke so commonly when I go out to see see organizations organizations I would say typically they will patch every

single month but typically once or twice a year they have to roll patches back um because of compatibility issues so according to uh hp's brand new paper their cyber security paper that HP just released they don't like the current state of patch management either I think it's funny that they would use such strong words like while it is laudable with laudable that adobe and Microsoft both repes patches at any point in their history so basically it remains unclear if this level of patching can be continued can be substained so Microsoft on average right now is averaging about 100 to 150 security patches per year per year and from a testing and a deployment standpoint this is time consuming this

is expensive to organizations and also risky because when we whenever we make changes it introduces risk to the organization also we find out that the most commonly breached security hole today is the same hole that it has been for the last five years we are not patching and because we're not patching hackers are lazy we're going to use the same tricks that worked yesterday we're going to try them again today because of that it's still working today because companies find that patching is too difficult it's too complicated and it's too costly so when I go out and do an assessment against an organization I find very levels of patching maturity from a walk into an organization and they do

absolutely no patching and what I commonly here is we can't patch here because it breaks stuff then I'll walk into an organization at the next maturity level and they'll they'll basically say hey I'm going to turn on auto updates I'm not going to do any any type of patching testing or anything I'm just turn on auto updates but there's no management of that there's no reporting there's no maturity around that and it's typically only around OS patches it's not around third party patches it's not around any of that then we get to a level where I'm going to use W sus I'm going to have this centralized uh infrastructure now and I'm going to control patching but this

doesn't address third- party patching this like Java and Flash all these other products that have to be patched so then I'll see a maturity level of well we're going to have we're actually going to patch and we're also going to purchase a vulnerability scanner like nesus or nexos and we're actually going to scan our entire Enterprise and keep track of our missing patches and our missing vulnerabilities little something I've learned over the years when you run Windows update and Windows update runs and it says I installed this patch all it does is it actually when it says it install the patch all it does is it goes and flips a couple registry key values it doesn't

actually validate that the install was correct and and the the dll versions and the libraries were actually updated to the right version so commonly when I see an organization go out and do a vulnerability scan for the first time the the vulnerability scanner will come back and say hey these servers aren't patched and the admin's like well we applied the patch to it well it doesn't not necessarily because it doesn't mean that the the patch was installed correctly so it's important that you do validation with a a vulnerability scan scanner or using Microsoft mbsa to actually validate the patches are installed correctly now lastly we I'll see this complete program okay where we're going to patch

every month we're very mature we have a process down what I'm introducing today is is a sixth bullet point at the very bottom uh when we talking about evolving to the program to a point where you only have to patch a couple times a year instead of every month so we talked about there's several risks just just by applying a patch okay so typically why do p patches fail when I deploy a patch out why does that fail the number one reason is the lack of testing so typically it departments go out they they go out and see ah we need to deploy these patches uh I'm going to deploy it to a to a test server or a couple test

machines today and we'll see if it really breaks anything okay didn't break anything let's go ahead and roll it out to production well that doesn't mean that they they thoroughly tested all the features and compatibility of the entire application also the second most common reason I see is when we when we do reboots when you reboot a box it doesn't always come back and even more so when it's a physical server if you when you reboot a physical server it's 10 times more likely not to come back then you know if you didn't reboot it so that's when fell occur is is on reboot so you know we talked about that it we lack the skill set and the tools

to adequately test patches so to properly test a patch most organizations have a a software development department and in that software development department typically they'll have a QA department and the QA Department was really good at writing test case and they're really they have great tools to actually test the features of of an application but it usually handles our own testing of a patch we don't have the skill set in the IT department we need to to mature our it departments to have the these the availability to tools to not just be able to be able to test every feature of a box to simulate what a user does simulate uh an average user opening

applications actually loading a file making sure that the file loads correctly all this can be scripted and it can be automated it doesn't have to be time consuming I find that that with through automation you can reduce the amount of of testing that it takes by 70 to to 85% if you do automation the other re common reason that I see patches fail is you deploy the patch out to all these servers it worked great in testing I had no problems but the second put load on that server all of a sudden the CPU spiked after I applied that patch we don't test we typically do not load test when we deploy a a patch so we need to

mature our our programs to the point where hey we're g to when we test a patch we're going to simulate actual load so we can really understand what this is going to do the problem though is if we increase our testing it's it's going to increase our cost to deploy the patch and it's also going to increase the time for us to deploy the patch as well so cost and time to deployment increases but on the other side it reduces risks so risk to the organization for business disruption so com we talk about cost and risk should should the business make this investment and should what is the risk the organization if we don't we talk about this all day in

cissp classes risk to the business you have to make patching a a risk to the business so most organizations I go into I ask them I say okay for you to to patch once a month what how much does it cost your organization to deploy a patch every month and most companies cannot tell me the cost to patch if I'm going to write a business case to a organization whether I should patch or not I should understand what the cost is to the organization to patch okay so so typically I need to to understand these variables of typically uh I keep track of these Matrix every month how many hours did it take me to test how many

hours did it take to deploy how many hours from various business business units did it take for them to validate traditionally I also track the number of times we have to roll back p Pates every year so typically in most organizations I would uh patch failure rates are between 5 and 10% if they patch every month one to two times a year they're going to have to roll back some patches track that information you need you need to understand that information so that you can build the business CA business cases of when to patch now when I look at when I'm trying to assess whether I should patch or not we need need to understand the severity

of the patch okay and we we're going to we're going to talk a lot about using compensating controls to remediate Patches okay but when I look at compensating controls like say a network IPS or a firewall when I look at these things I don't care about them anymore I don't care about a firewall I don't care about an IP IPS anymore because the most common way breaches are occurring today is through pivot attacks so we look at the Target incident that that occurred at Target they they were able to breach a heating and air conditioning desktop machine and then they used that to to attack another desktop and then they use that desktop to attack a server and then

a cash register a point of sales unit so all of a sudden you have these pivot attacks that are coming in these attacks are coming from the inide side of the network not from the outside so when I'm assessing the security of of a server I do not take into account perimeter security because most attacks now are coming from the inside so let's talk about this concept of risk-based patch management most compliance programs like PCI compliance Regulatory Compliance programs like high Trust Hippa if you read through the programs they all will say a lot of them will say do you apply patches monthly what is your what what is your policy around patch management most companies say

monthly but they also will say do you patch monthly or do you have a risk-based patch Management program when I ask people what what risk-based patch Management Programs me and I get a hundred different answers so let me show you what my definition of a risk-based patch Management program is that i' that I've deployed before so we're going to talk about comp compensating controls on a server so a server will typically have antivirus it might have memory protection like Microsoft emit it it could have a host firewall it could have a host IPS these are all secure it might have applicational whitelisting these are all security layers that are on the server itself okay and so if I tried to exploit that

server it would have to go through every one of these layers to be able to exploit and as that exploit travels through these different layers of security it's going to interact and it's going to change the behavior of that exploit so even though Microsoft comes in and says hey this is a high vulner this is this is high severity but but when I actually try to exploit it on that machine it's not a high severity because my my host IPS picked it up and it stopped that exploit from occurring and so since it stopped it it's not a high severity anymore I can re-evaluate the risk because I'm going to assign I'm going to now track

hey this exploit can be stopped by this uh compensating control this this this security product here stopped this exploit so when we when we do assessing on a box we we're going to talk about two different things we're talk about vulnerability scanning we're going to talk about penetration testing and exploitation okay so typically vulnerability scanners that you use today like nessus nexos these are all vulnerability scanners what they do is they they look at a box and they what they're looking for is like dlll versions uh they don't actually try the ACT actual exploit they say based on this dll version your server is vulnerable to this exploit it doesn't try the actual exploit it just

it just evaluates what you could be susceptible for okay now a penetration test or an exploitation software like like metas or core impact that's going to go out and actually try the actual XO big difference there so penetration test actually tries the actual exploit and sees what I can actually do with that exploit C can I get a shell access can I take control of the whole desktop can I pull data out it's just I need to I need to use both tools so my approach to to patch management is there's varying phases to to the approach to a risk-based patch Management program the first phase is the inventory phase I need to understand what Hardware is out there I need to

understand what software is out there I need to understand where the sensitive data in the organization exists on the network and then I also need to understand the impact of that system if that system goes down what is the impact of the organization because not all servers are created equal if I bring down a database server it's more than likely going to have a much larger ramification than bringing down a like an app server the next phase when you we've talked about the importance of testing and how important it is to test but if if you work in an environment that has 5,000 servers you cannot do 5,000 test case that's too many things to test against it would

would take you literally months to to deploy patches because you would have so many test case so you want to be able to reduce your test case down so what you want to do is you take your servers that have the same footprint the same applications loaded on them so you have a bunch of web servers they they might be different web servers but they have the same version of Tomcat on them they have the same exact version of of of all these different applications so they're they're configured identically I mean they they have the same vulnerabilities on them if I if I conduct a vulnerability scan on these two different systems and they're exactly

the same they're configured the same I'm going to Lump Lump them into my into their own test group so I can reduce the number of test case so I've been able to to to successfully go from over 5,000 servers to 50 test case so I only have to test 50 test cases now I don't have to do 6,000 now the next thing I do is I perform that vulnerability scan we talked about earlier and what I'm looking for is not just miss missing patches when Microsoft releases a patch it doesn't it it can contain multiple exploits so exploits are usually tracked by cve numbers so a common vulnerability exploit will be an actual number for it

so if you look at a Microsoft uh patch it might contain 10 different cve numbers so what we what we want to do is we want to look at the server and say okay tell me what what cve is does this server is it vulnerable to I need to know a list of those exact CVS because what I'm going to do next is I'm going to take that that list of cves now and plug it into my penetration testing software and I'm going to point my my penetration test Ser my penetration test tool to that server and say okay the vulnerability scan said it's it's it's vulnerable to these things I want you to actually to to try

to exploit that that small list I'm not I'm not talking about testing hundreds of thousands of different exploits I'm I'm taking a small list of maybe a couple hundred exploits that I'm testing it againsts the goal here is to be able to automate this and to automate your testing so you're going to go out and actually exploit that the next step is what you're going to do now is when that explo when when those exploits ran you need to capture the log files from your very various compensating controls on that server so you need to look at this the AV logs you need to look at the the log files from your host IPS the log files from your

application wh listing and look in those log files and see where it stopped that vulnerability which which of those caught that vulnerability and stopped it or which one changed Its Behavior so for example I I I only get partial access to a box instead of full access to a box I I no longer can get root because one of my one of my compensating controls changed Its Behavior you need to be able to create a an in a a list I I track a an Excel list of cve for each server each test group I keep a list of cve numbers and then exactly that are that the vulnerability scanner says I'm vulnerable to and then I keep a list of

exactly which compensating control caught them because the end of the day I need to be able to go to an auditor like me I'm a PCI auditor I'm a qsa and when I come in and do a Qs when I come in and do an assessment at your organization I'm going to ask you show me a I want I need to see a list of of vulnerabilities that your box is currently vulnerable to I need to see the outputs of a nessus scan so I'm going to take a look at that nessus scan and what you can do is you can come to me and say okay here's here's a list of vulnerabilities that this box that nessus showed or expose

showed here's a list of the exact compensating controls that stopped each one of those cves you have to be able to demonstrate to me the auditor that those compensating controls worked so you have to keep this evidence for an audit now there's always a big question who should decide when to patch typically the IT department or the information security department will will say hey we need to patch now but should we this is we're talking about spending money we're talking about spending in resources in most organizations they they holistically will track the the risk of an entire organization so they're out there they're going to uh the and typically it's the risk committee is is is fronted by the uh CFO

and the CFO will look at the the risk to the business and what it costs to fix those particular risks to remediate those risks we when we take this approach of understanding what the cost is to patch and we understand what the risk is to the business we can have discussions with people like the CFO and say okay here is the actual cost here is the actual risk here is the percentage we can look and see what the percentage of the the likelihood that it's going to be uh exploited we can tell you exactly what the impact is if you exploit these particular servers okay so typically I will write now a business case to the

business saying why we need to make the investment to patch it's fascinating when you when you change the methodology of of going from hey this is just regular maintenance to this is a business decision that the business has decided to deploy this patch this is a business decision all of a sudden I was it became a hundred times easier for me to go to the business and get support from the business to patch because this was a business decision not an IT decision uh it was phenomenal phenomenal change in the in in the way that the business perceived patching because this this was a this was a business decision to to reduce this risk so we talked about this treat it as

a project when you treat patching as just regular maintenance I find that organizations they they get halfway way through their patching cycle they never really finished it and so now they they really have these these clusters of servers that could never be patched and so and we lose sight of these patches on these these special case servers so typically that's where the the the the hackers are going after they're going after these systems that for some reason or another have never been patched but they don't we don't even track the compensating controls on these servers either as well so even if they have vulnerabilities we're not tracking how how we're going to even though we've made a bit a decision as an

IT department not to Patchy servers we're not thinking of ways to to compensate for for not patching so one of the big challenges that I've that I had when I used this method was coming up with an En a test environment that accurately represented my production environment and it can't just be well it's a VM okay because we're going to do load testing and when we understand load testing we need physical we need Hardware that can have the same capacity of a production even in the cloud environment okay if I'm if I'm going to spin up a cloud environment uh a cloud instance in AWS I need to make sure it's spec the same so

that I can put the same amount of load and make sure that I I have all the same dependencies I can't just spin up a local copy of the database on that web server I need to actually have a database server that I can I can load test

against so we talked about the importance of automated testing so I've been able to like I talked about before I've been able to reduce the the number of hours that it took to patch so I worked in this organization it was it was costing us $100,000 every month for for us to patch that's $1.2 million every year that sounds like a huge number but this wasn't a huge organization this it is not uncommon for for companies between two one and 5,000 employees to be spending this kind of money and labor every month on patching and support calls and and help Des technicians to support this effort it adds up quickly and so the more you can

you can reduce the number of times you have the patch and reduce the time it takes to test the patch and the number of times you have to roll back from the patch is going to save you money I got a little little excited I talked a little fast um you know so just to wrap up some key takeaways of of what we talked about today uh once again my name is Adam steeve uh if you saw my talk last year on on why I love LinkedIn and using why I think LinkedIn is is a better pin testing tool than than inmap go ahead please go ahead and Link me in uh my name's Adam Ste on LinkedIn go ahead and

Link me in I have a bit really large Network I love hard questions uh if you have questions around PCI or coming up with with a remediation for this audit finding feel free to hit me up I love hard questions does anyone have any questions around this

go so what I did is says okay if we're going to change our methodology we need to increase when we do patch we need to dramatically increase the amount of of testing so I took them the number of hours for for testing that we were spending per month and I doubled it okay and then I also doubled the amount of documentation so from a monthly cost perspective when we did patch it was costing us 40 40% more money for us to do a patch uh so we were investing more money when we did we were just doing it less often but the let's go ahead

you have to pay someone eventually you're going to have to pay but can you reduce the number of times you you have to pay it the end of the year I I can tell you flat out I was spending a lot less the end of the year but you bring up a great Point I've increased my dependency now on my tools I need to make sure my tools are healthy and they're running my because now I'm very tool dependent

so you know the point I want to make is this doesn't work in every organization I I think there's use cases that it is better for the most part there are it's probably tra it's for most organizations patching monthly is a better better way to do it I'm just saying that in certain certain use cases it it it it doesn't work for example I've been doing a lot of work out at a hospital and they have all these medical devices and and the medical device manufacturers will only approve they have to prove every OS patch that is applied onto a medical device and they they will only approve those patches a couple times a year and so I physically

I can't patch those devices every single month because the the manufa facturer says I can't and I and I often will see that on appliances as well when when a vendor stps you at Appliance their Appliance uh they will only allow you to patch it when they when they bless it so any other questions cool thanks guys appreciate it