PG - Vulnerability Management 101: Practical Experience and Recommendations - Eric Bryan

all right so again good evening I'm Eric Braun if you saw my talk on the schedule or saw my name on the schedule and came anyway I appreciate that very much so anyway I'm here to talk to you about vulnerability management and there's quite a bit to get into so we'll just jump into it I want to say a quick word of thanks to besides for inviting me to speak and to my mentor to make it up here for mentoring me and hopefully we've got something special for you and I hope that everybody can take something from this talk back home and use it in your organization all right so who am i Erik Bryan security engineer at North

State Technology Solutions I have been an IT about 18 years and security for about 10 of those years back then I worked for a private investigator doing some digital forensics and that's what really got me hot and heavy into the security side of things so now I do risk management and compliance consulting and of course vulnerability management so what is vulnerability management vulnerability management is defined as the cyclical process of identifying classifying or mediating and/or mitigating vulnerabilities and a vulnerability in this sense I'm sure you all know a weakness in an information system security design implementation procedures that can be exploited to gain unauthorized access to information or an information system so why is vulnerability management important

because the Center for Internet Security says so it's not a bad reason the CIS maintains a document called the top 20 critical security controls and number three formally number four but it's so important they moved it up to number three continuous vulnerability assessment and remediation the goal of which is to continuously acquire assess and take action on new information in order to identify vulnerabilities or mediate and minimize the window of opportunity for attackers it's also important because around us we have a constant stream of threat intelligence this comes to us from pretty much everywhere from vendors like Microsoft and Cisco security researchers like Bruce Schneier and Brian Krebs lots of other sources that we'll get to in a

minute and failing to keep pace with this just allows for easier exploitation so example here you have a vulnerability that's published and not responsibly published so at that time the information becomes available to everybody it's available to the bad guys to weaponize an exploit to the vendors so that they can develop and deploy Patch and to us or I guess I good guys to remediate and apply those patches and other defensive measures so this isn't so much a how-to on vulnerability management as it is about my experience implementing this at a client so this client back in Charlotte North Carolina that we have and this client has twenty five thousand devices and these are workstation and servers and

point-of-sale IOT is in there and they're running cisco and windows and AIX ESX island they're in countless applications so when we started this program with them they had pretty much nothing and what they had was always reactionary so here's our scenario there's an existing vulnerability discovered so I mean what do you think happened from experience you know what's happened in your organization's this was ours all the time every time so what would happen is everybody's hair was on fire and we had to run this thing down until we figured out what we're going to do about it if we were going to remediate it if it needed to be remediated and how to address it from

there so here's our next scenario here's a possibly applicable vulnerability we might have this it might be out here it might be something we need to look at it's on Windows we know we got Windows that's about it so what happened this every single time again everybody's all crazy everybody's hair is on fire or running around trying to figure out how to address this vulnerability and this process it was ultimately effective we eventually got patches applied eventually got updates done but it was incredibly inefficient and just exhausting to everybody who had to facilitate it namely me so for my experience here we have three primary components of vulnerability management we have number one vulnerability discovery so finding out the information

that's out there secondly we have vulnerability notification taking the information we got from our sources and relaying it to whoever can remediate those vulnerabilities and finally verification verification so taking what they said they've done and just verifying that it's complete alright so the first goal that we wanted to accomplish here was to improve our vulnerability discovery our awareness of what was out there so our first step here we deployed a vulnerability scanner it happened to be rapid7 expose and the first thing we did was run discovery skin now this client runs a slash eight IP address so in case you're ridiculously good at math or just happen to know this that's almost 17 million IP addresses

and this took weeks and weeks to complete but it was minimally invasive didn't crash anything thank God and it gathered basic information now every address an expose gave us an IP address every asset we got an IP address now if it could be fingerprinted then we would get an operating system and sometimes the host name as well so that would give us the basic building blocks that we needed to scan further and to do that we would do credential scans so we had an Administrative Service account that we would plug into next pose and then it would scan those devices with the service account now if the credentials were valid it would bring back a lot

more information and since most of what we were using was Windows that was very valuable information for figuring out what kind of software we had out there so the next aspect for us improving our own ability discovery was a daily or weekly review depending on the source of information so to do this we had lots of sources first of all the national vulnerability database the NVD and this is a multi-format downloadable searchable database that let you look and see everything that's out there this week we could compare it against last week's and see what was new or it was especially fun the first time we looked at it and had to sort through 25 years

of vulnerabilities and figure out what we might need to address secondly the United States computer emergency response team or u.s. cert and they provide notification services they have a weekly security bulletins for us to look at next up vendor specific alerts so here's your Patch Tuesday and your regular updates from Cisco and VMware and Oracle and whatever vendors whatever other vendors that you're using so the next source of security research and aggregation sources so these include threat posts CSO online and cyber wire also we had penetration test done as part of a compliance process so after going through this for a little bit we learned to incorporate the findings from the penetration test as well it's

very valuable source of information and finally social media so like Brian Krebs Kadima sorest security week Megan woo has a Twitter account I guess and then paying Twitter accounts they're constantly updated so if you're looking for the latest and greatest vulnerability and Mason Twitter's where you go so thanks to these sources we've identified some vulnerabilities that we might need to address so what now the next aspect we had to consider was resource allocation how do we prioritize these vulnerabilities in a way the specific to our organization so first we looked at how vulnerabilities are scored by the industry the common vulnerability scoring system or cbss and this is an open framework for communicating the characteristics and severity of

vulnerabilities and the cbss uses three primary metrics base the intrinsic qualities of the vulnerability temporal aspect specific to an organization's I'm sorry temporal aspects that may change over time and finally environmental assets or aspects specific to the organization's environment so and that uses a ten-point scale as we can see here low medium high and critical now these environmental factors that are built-in in the cbss to help organizations tried and rate these based on what they're using so they use these dynamic components and here's what they look like right and so we had a problem with this because it's just too subjective and depending on who was reviewing what vulnerability it was just a mess because you'd have different

people coming up with different results and you know look it's a lot like this this is what it felt like trying to figure out those critical components yeah math am i right so we devised an alternate and simplified calculation scheme the adjusted scoring system so my acronym needs some work I've been I've been looking at it for a while now so the purpose behind this system is to take into account the industry scoring the cbss and combine lots of institutional specific factors so there were three potential factors here with a maximum score of 10 the cbss has a maximum score of 10 so we would add those together and we would average those to get us new enterprise

specific score to help us prioritize so the first and largest factor we looked at was external accessibility if any asset with this vulnerability was accessible from outside the organization for points if there were no assets that were vulnerable they got zero so the next factor we have was data sensitivity in our organization we had PCI data of course because we're a retail this client also operated a pharmacy so we had personal healthcare information to consider as well and personal personally identifiable information so those rated like this and they are mutually exclusive so if you had a system that had all the data types in it then they wouldn't get 6 points from this aspect they would just get the

highest one and the final factor was prevalence and after playing around with it a little bit for our organization this is what we came out for the sweet spot of where we felt prevalence would make the most sense here if you're going to take this and try it elsewhere you want to adjust these numbers based on the size of an organization so by implementing the adjusted scoring system we go from this down to this so it's much simpler right so let's look at a practical example have the using Cisco iseñor organization and the supplicant for that is anyconnect so we had this on just about every single Windows Device out there so quite a few devices so it

was rated a high severity a 7.8 so we had to look at it and figure out okay we know this is a problem how do we address it so we took into account our different factors internal only within PCI scope way more than 50 devices so we did our math down here and came out with a justice score of 6.9 still important still needs to be addressed just you know not right now in this chart by the PCI DSS shows us this as well so it's the same scale we talked about a minute ago but for low remediation is not required so we had a lot of those that it just booted it right out of the park

nothing to worry about there for medium it has to be addressed within within 90 days and so in the previous example it knocked down from we have to fix it in 30 days to now we have to fix it within 90 days and that made a little bit simpler and we could prioritize anything above that that was more important according to our system so we scored two vulnerabilities and we prioritized them for our organization and then throw back Mubi reference for you know so we get to the second component vulnerability notification taking all this information and relaying it to the persons responsible for mediation so this client had about half a dozen administrative teams to

address these vulnerabilities on different level for example we may have a Windows server that's running Oracle so we had the windows team that patched windows and the database team that passed Oracle so we'd have to have those teams working together to remediate that and that's just one example there was several vulnerabilities that pass through every single team just just because of the nature of the vulnerability so we would limit the number of vulnerabilities to 10 per team per month we tried giving them all the vulnerabilities but some of them had hundreds and they didn't take that too well yeah so to do this we would start with an expose report top-25 remediations with details very important

gave us a good jumping-off point and based on that we would create two documents the first one we give them a spreadsheet overview so they could see at a glance here's your ten here's what you need to look at and they give them all the information about our external factors the CVE etc and then we do a second one a detailed document that would have that information plus any links that they might need to to remediate this and there were quite a few it took some practice to get this going the right direction but if we didn't do this they would bug the snot out of us trying to figure out how to get us to tell them how to do their jobs

which is fun of course so we had the reports assembled and we met with the teams and so we had two meetings per team per month and in these meetings we had the team of course we had the security team of course and we tried to involve their management as much as possible they needed some some encouragement let's say let's put it that way so the initial meeting the documents are presented to him and we tried this an email at first but we found that they would either ignore the email or what kind of give us half answers so it was important for us to get out in front of these teams and give them the

opportunity to respond to look us in the eye and understand that this is important it's something you need to look at and do something with it so we had the first meeting and then they'd come back for a second meeting and they'd have our documents that were marked up and annotated with what they had done or were doing or what they had plan to do so in these documents that we got back from them we saw there were four responses the first response not applicable don't have it not using that component no problem secondly false positive can be disregarded also throw it right out no problem next it can be remediated and here's where we come back to our timeline for a

mediation within 30 days for critic or high fourth in 90 days for medium that's great that's the best case scenario or finally it cannot be remediated in the predetermined time frame or at all so we had a problem on our hands so when we got the response from the team that they were unable to remediated it remediated on time or at all then we would generate a risk acceptance plan so why do we need a risk acceptance plan it's very simple because at first we would try to just Slough it off you know just kind of shrug our shoulders no it's not that big a deal but we came to realize that these things have to be run by and approved by

senior management because it's not up to the security team to accept risk it's not up to the security team to accept risk I can't say that loudly or frequently enough but instead it's the job of the security team to come in to inform and advise and what constitutes senior management so we thought about doing this at different levels at first so having just is manager sign off on some of them based on severity or the VP of is or to see so based on what it was we ended up having everybody go through the VP of is I just seem to make the most sense for us but senior management for your organization a good rule of

thumb is whoever's most likely to get fired if it goes sideways that's a pretty good rule of thumb to decide who needs to sign off on it so after we would put together so what goes into a risk acceptance plan so we would lay out our documentation so we would give a detailed description everything that management needed to know about the vulnerability to make an informed decision including the scope so how many devices were affected the platform's so what type of devices were affected are we talking about our point of sale here are we talking about our servers are we just talking about workstations it's important to get this information across and also in here you want to lay out

your worst case scenario so that since they're signing off on it you want them to have a good understanding of what they're signing off on and what the full implications of it could be so you list your worst case scenario you go through all the bad and then you can come in and list the good and here's where our vulnerabilities some of them just weren't that big a deal because we had mitigating controls around him and then so because of these mitigating controls that didn't really matter so in our mitigating controls included your ids/ips so your Cisco firepower your secure works firewall Cisco a.s.a.p A's pal Altos your antivirus your network segmentation we had some where

the device that was vulnerable completely segments and from everything else what kind of risk is that almost zero and application whitelisting I'm telling you that's the way to go if you haven't heard of application whitelisting what are you doing at this conference if you have heard and are look you should look into it if you've looked into it and implemented it good for you ghost are you are on the road to success and including here anything that might mitigate the risk no the vulnerability so now we get to our third component remediation verification ensuring that remediation is complete because teams will tell you that it is when there's really not so we discover vulnerability has been remediated and

the remediation has been reported all right so that's what we'd hope for from them in this case it's up to the security team to go out and collect evidence so what constitutes evidence well if you found out about it through your vulnerability scanner it's pretty simple you just rerun the scan on the devices if it shows clean you're in good shape also configuration and patch level screenshots for vulnerabilities that were discovered a house out of an expose those are a little tougher also vendor reports so your Microsoft baseline secured analyzer MBSA could independently verify this as well as we used f5 load balancers so they have their own tool where you can import your configuration and it'll come back and

tell you all the patches and configuration issues with it it's pretty handy so now that I've discussed my experiences when Bowman really vulnerable 'ti management I wanted to provide some suggestions so if you want to get this started up at your organization or improve your stance on this as a company you know here's some things you can look at so the first recommendation bring in a consulting firm to perform a security audit based on those top 20 critical security controls and these security controls include number one inventory of all authorized an unauthorized devices so there's your an expose discovery scam number two inventory of authorized and unauthorized software so there's your credential scans that give you that information and number

three the owner ability management which is why I'm standing up here so our second second recommendation stand up with vulnerability scanner compatible with most if not all devices in your organization a simple discovery scan the first pass they're relatively quick mentally invasive it might take a while but it's going to help you a lot just do you understand the assets you have out there secondly run your credential scans that we talked about a minute ago and based on the size of your organization you want to break these down by IP range or geographical location and organization unit or asset type next create device hardening standards a good way to do that do this and here we're back to the Center for

Internet Security see is they have to see is marks these are super handy and they allow they provide advice for hardening a device based on this operating system and those are all included here cisco router switch firewall all the operating system and even multi-function devices even printers there's covered there I thought that was neat some vulnerability scanners you have an expose and necess or two examples I'm going to use can run a specific type of scan against a device and provide a CIS benchmark report which is very handy when you're trying to put out some documentation for your auditors I know necess has a really cool capability where you can export your router switch config into necess and

then we're do on offline scan come back and tell you everything the front it's really cool especially since next pose wouldn't do what we needed to do there no offense to next pose it's our next recommendation hire professional penetration testing firm and you want them to simulate attacks on any devices that are external outside your organization in that external space anything with sensitive data if you're doing PCI compliance or HIPAA compliance they're probably going to force you to do this anyway and finally anything that's compliance related so if these recommendations are a little bit too resource-intensive and they can be especially if you're a small shop look it's a software-as-a-service and these solutions are great because they require

minimal infrastructure cost and it allows a person or organization to focus on using the application instead of trying to administer or deploy and maintain it very handy so our key concept here if you missed everything else I said tune in for this one the key concept is to base an information security program around goals and strategies not a tool or a set of tools all the vulnerability sources we went through and the vulnerability scanner all that is really handy but no combination of those things can tell you everything you need to know about your environment it was very important you based around goals and strategies so your approach to vulnerability management has to be comprehensive has to cover as many areas

as many device types as possible next it's got to be concerted have all your teams working together towards this common goal of eradicating vulnerabilities because that's going to happen it's really not customized so here's where we get back to our adjusted scoring system you can have a customized for your organization consistent the same for all teams all vulnerabilities and all circumstances this was important we had those vulnerabilities I talked about it we just wanted to write off because it's no big deal you know shoulder shrug quick email from our manager we don't need to worry about this but it was important we found out to be consistent and to hold ourselves to that standard of addressing all the

vulnerabilities we came across you know top ten at a time and finally changeable so initially you'll need constant changes to your program we spent close to six months as a team working on our vulnerability management program before we presented it to the first team and we thought it was great we loved it we had celebratory lunches and dinners over it because it was so good we presented the FIR to the first team and it fell apart so just keep that in mind don't get discouraged when you have to change your program 111 two times so in conclusion vulnerability management can be a powerful tool for reshaping and information services department engaging in a cyclical process of identifying vulnerabilities

notifying the persons responsible for addressing them and tracking the vulnerabilities complete remediation has the potential to vastly improve and Information Systems department any questions

and one quote one thing I was wondering is that you described you dead you had quite a big network yes but I mean what a big network but a bigger pond abilities comes a lot of work right very much and I guess the I see people has had it had a lot of things and I paid already so so how do you manage to do to do to do that in practice because that's what I've I mean it all looks good and great and paper and all that it is great and paper yeah excited but I mean I mean what what you mediate and then and then new new vulnerabilities around you again I mean do it in practice could has got

that piece yes so that's why we wanted to limit ourselves to ten per team per month because if we came out and wanted you to fix everything and sometimes we could lump them together like if there was your Patch Tuesday and you had two patches that were within that patch Tuesday you could just tell them hey here's all your patch tuesday findings you can go patch those and by limiting it to ten per team per month we could let them take small bites and small chunks and pretend that those other ones didn't exist for 30 days it would get better it would get better teams usually after we had implemented it for about six to eight months we found that the

teams only had two or three items per month it had worked its way down that far and improved their business process at the same time to where it was just easier as an organization to deal with vulnerability even outside of our top ten process

how frequently do you run vulnerability scans and do you get pushback from asset owners who say that's an out-of-date scan how do you deal with that so we run them monthly the on a monthly basis and it was a rotating schedule so we didn't boner ability scan everything on the first Tuesday of every month for example and it was broken down by IP range device type you know like I talked about a minute ago and we usually would have to do a change request like get it approved by different people but if the asset owner was like we fixed it run another scan we could just do it right then and there on demand you know assuming that we had

time to to run that scan for them and that it wasn't gonna take their system offline which usually was not the case one more anyone all right thanks so much guys appreciate ur error [Applause]