Extinguishing the Vulnerability Management Dumpster Fire

I'm Rebecca deck so welcome to extinguishing the vulnerability management dumpster fire so as Susan said I'm a knapsack person and this one time I got stuck doing vulnerability management so this is the story about me getting stuck and doing vulnerability management and also I get to use a lot of these old pictures that I had from my time in the Army here so this is an actual dumpster fire that myself and my friend sergeant dentist set so you'll see a lot of stuff that I set on fire during this talk so I do app sex stuff right now at a Valero I just started there I do a lot of manual testing work pretty much anything

anybody's willing to pay me to do that's really what I'm gonna do I've been focusing kind of on DevOps and IOT work and cloud testing the last last couple of years I enjoy automating myself straight into unemployment so I think like being a parent working in security like your ultimate goal should be to not have a job anymore in like 18 years I'm somewhat of a hatchet woman like I said I'd do anything anybody's gonna pay me to do so I've done sysadmin work I our work forensics vulnerability management and soldier and I was a software developer for like eight years so here's the plan so I'm this is just kind of a general idea of what I'm gonna be

talking about so first the set up me kind of getting some friends in order to actually do this work making a plan executing the plan the frustrated cargo that we had along the way and the epilogue where I talk about what happened after I left this company so setup we had a vulnerability management person and this person just quit like two weeks before Christmas so it was kind of a big problem right we had about half a million known vulnerabilities in our enterprises identified by our vulnerability scanner which as we found out was not scanning the entire network anyway so actual numbers probably much higher that was spread out across 5,000 system so you're already talking like a hundred

vulnerabilities per system but we had a gold image that was updated every two months so a lot of systems only had like three or four vulnerabilities which also meant that other systems necessarily had like a thousand vulnerabilities it was pretty interesting to see what had happened over the years we had 26 ad domains nobody knew what they did and we discovered new do new ad domains throughout the process and I was separate from IT so I didn't even have admin rights to my own computer let alone all the servers and desktops in the enterprise and we did own a vulnerability scanner that's probably a prerequisite for doing this kind of work so really everybody was just happy the

way it was until I came along and ruined it for everyone so really nobody cared about any of this for a long time right it was just this thing it's like the broken stair it had always been that way in the organization since the day they started scanning the network and found out they had this big load of crap that they had to deal with and nobody wanted to touch it my real job was way cooler than vulnerability management but this is a lot more important so I'll talk a little bit about why it was a lot more important during the talk so in the past we would get awesome quality push back like the vulnerability scanner is just

lying which turned out not to be the case most of the time right the scanner had admin rights to log into all the desktops and all the servers in the company so it wasn't wrong it was looking at your windows patches and saying that they weren't there right pretty cut-and-dry people made the argument you couldn't patch anything until you knew where all the important data lived in your environment which is in my opinion not the best way to deal with it right so since they could never manage to prioritize anything they also never managed to patch anything so every month they would just stick the patches out there and whatever stick or whatever stuck they were happy with right that

was life the client decide what patches it needs so we would push it out from our Windows Update server and they were like well the clients they know what patches they need and I was like no this kind of makes me think of my kids and what would happen if I just let them decide what they need to do they would like pass out from dehydration after like a bender of playing video games for three days my kids are six and eight so that's about all you can expect from them and that's really all you can expect from these servers right they're gonna be out on a bender scanning is gonna break stuff and that one is kind of legitimate

especially if your network is made of toilet paper so we had like one and a half megabit Internet pipes to some of our like we had a hub-and-spoke model and when you went on the spokes it was like these one and a half megabit MPLS circuits so it was conceivable to Dass each of these branches on the MPLS circuits so we had to be super super careful one time this isn't really related but this one time I was doing testing there and I looked at a server that was in a that was out on one of the spokes and I was cutting off traffic like I did it in a recurring fashion and the phones went down there

every Wednesday like at this specific time and nobody knew why for months and that was why so yeah sometimes that is a legitimate concern but if you do good root cause analysis you can fix these problems and stop knocking networks over that domain doesn't talk to anything and like yeah that's that's a terrible reason to not patch things we actually had firewall rules that prevented scanners from looking at certain domains and we had firewall rules that prevented patches from going to those same domains and our security tools were gonna save us we actually owned everything fire I made which is an achievement in and of itself right even their packet capture solutions I didn't even realize fire I

did packet capture like that's kind of worse to breed but we had it right and people were like well we don't need to apply this patch because fire is our god right so first I had to make people care this is pretty much what our network looked like when we started and I've critical security controls to kind of drive prioritization the critical security controls we intuitively know this as security professionals right you probably don't need to go do red teaming if you can't patch a system but that was what we were doing and the critical security controls make it not my stupid opinion but it was the opinion of many security professionals which carries a

lot more weight than my opinion so I use that as kind of the vehicle right so I was hired to do apps at work and at the time that was like critical control number 18 and critical control 20 is red teaming which is the other thing that I was doing well this is critical control number four right so the only things you should be doing before vulnerability management figuring out where your systems are figuring out what's installed on them configuring it correctly and then you start doing vulnerability management right and but if you've got the vulnerability management stuff go ahead and start doing what you can where you can so looking at the Yap sec findings it was

kind of like this Venn diagram right so I was finding this amount of stuff the vulnerability management scanner is finding this amount and a lot of what I found was part of the stuff found in vulnerability management scanners anyway right so yeah if they had MSOE dos 67 and this was in 2015 and yes we did have MS o 8 o 67 in several places but the odds of an attacker sitting there using ms are using like some exploit that they write custom for your network when you're vulnerable to MS o 8 o 67 that's probably not gonna happen right they're gonna use the b-team before they send in the a-team they're not gonna waste their

time if you haven't even passed your systems and all these findings that the vulnerability management stuff finds those are publicly known vulnerabilities there's probably exploits poor them out there even if a public exploit isn't there people can reverse engineer patches and do work right again someone is not gonna spend time on your bespoke application if they can avoid it and then you look at things like this so I started giving executives graphs like this of the findings that we found it was like as you can see here yeah yeah so my guys you see down here this is what I found and they're like well that looks like zero and I'm like no no it's not zero but if you put it on

scale it is effectively zero right so don't waste your time doing this this is a more important task so that got a lot of traction with the executives and I was able to kind of weasel my way in and get some kind of resources assigned to me cuz again I didn't have access to do any of this so this is me trying to lift up a couch and this is the rest of my team those are my kids sitting there playing switch while I tried to do work right you can't lift up a couch with people sitting on it doing nothing so I had to go find friends to help me this is my good friend sergeant dentist and

if you know me at all you can tell he and I got along very famously he helped me start a lot of the fires that I have in this presentation but yeah so I got one person from the server team and one person from the desktop team I had my one vulnerability management analyst who was a fantastic person and working way below his pay grade at the time which has since been corrected at other other places but still so this is what I kind of went to war with right and we had a saying in the army you don't go to war with the team you want you go to war with the team you've got so this is kind

of what I could wrangle up and this is what we ran with so before we could do any patches we had what I would call a very stout change control process right and if you've worked in large enterprises that's probably not very shocking so we had two weeks between the time we submitted a change and the time you could actually make a change so during that time people could sit there and pick it apart and try to cancel your change but that was kind of a blocker in the past they would be like hey we need to patch these systems they were like oh it's gonna take two weeks we got to go to the change control just never mind

right so instead I was like hey server team and desktop team you know change control works I don't I need you to do run to change control for this and really the two weeks wasn't a big deal all because we started a pipeline and every week we put stuff in the pipeline so there was always stuff happening right so yeah we had a two-week ramp-up period but after that we were off to the races and then it really didn't matter if I had to change control process or not that I had to deal with because we had this pipeline and the work just kept flowing so I made these weekly vulnerability management meetings so that was just the four of us

I didn't invite managers on purpose there was one particular person that I was trying to avoid throughout the entire process because it was fairly toxic so we didn't invite any managers I didn't want anybody asking questions that were bothersome because I was trying to avoid analysis paralysis right the idea that we want to do everything so perfectly we can always do a better job of looking at this well at some point just have to suck it up and start pushing patches to servers and see what happens and then handle it from there this is another army army thing that we had who is at the 80% solution on time is always better than the hundred-percent solution either late or

never so just start doing something even if it's not perfect it can still make a huge difference and I made sure to keep this meeting to only thirty minutes because people start like tapering off and they stop paying attention what you're saying which is incidentally why this talk is also thirty minutes so people can actually listen but yeah so it took a few weeks but then we had things going so here's sergeant Dennis finally lighting this dumpster fire so we had four remediation techniques that we used we did patching config stuff that were things that I would push out at these weekly meetings so I would say look here's the vulnerabilities we're gonna handle this

week I need you to make these exact changes so I did most of the work for the vulnerability management for the server and desktop teams saying here or the windows registries you need two entries you need to change there is going to be a file located here on this device you need to change it to say this right very specific instructions because they had other jobs right it's not that they were incompetent but patching these systems was not all they did so you got to be respectful of these people right and help them along a lot of them especially like in desktop what I've seen or that people don't have the technical expertise to do all this right

they're not gonna run these things down and figure out why a patch failed to apply so you kind of have to do it for them as the senior person on the security team because you care about this they may not we created projects so some things are gonna take forever to do I was talking to Ray about SSL and TLS earlier and those are great examples of things you cannot just push patches for right you start changing cipher Suites without coordinating across the entire enterprise everything is going to go up in a flaming dumpster so some things recognize those and then make projects for those those are the things were yeah you need to involve management on those

kind of decisions pushing pushing a patch not so much old systems so remember I said some systems we're gonna have like a thousand vulnerabilities on them the windows 98 system that ran some of our critical infrastructure and again this is 2015 yeah and honestly Windows 98 surprisingly resilient from a networking perspective if the ports are restricted because there were no good debuggers at the time and it was like predated Metasploit and all these other public vulnerabilities like exploit systems so finding an exploit that would work on like the one exposed port was really not as trivial as you might sound and the VNC that was running on it was so old it was before they had that preauth bug in

VNC like in 2000 or whenever that happens so and then but those need to be projects right I'm not just gonna go shut down this critical windows 98 says server right that's something that I'm going to involve other people in and then risk acceptance some things you're just not gonna fix maybe you have a third party integration and if you apply this patch or you change these cipher suites you're going to break that third party in raishin you don't have any choice in the matter right so some things you just have to let go and I'm not above that I could still pull them off my numbers and drop my metrics I formed a secret

remediation cabal which was really just my friend and I from the the vulnerability analyst but we would get together and talk about what we wanted to do I would keep this meeting super short like 15 minutes and we would do it two days before we had the other meeting with other people so that way we could get all our ducks in a row make sure we were squared away so pick out what vulnerabilities we wanted to remediate next so he would show up and he would have a ordered list of vulnerabilities based on one based on severity and one based on volume of vulnerabilities right so if you had the same vulnerability 50,000 times in your network which is

shocking with only 5,000 systems but there you go so you had things like that were you like alright let's find the one that were the most common so we picked out the three most common by volume the three most severe so I would kind of make the judgment call on those where I was like alright MSOE Doe 67 needs to be fixed Tomcat Manager with a username of Tomcat and a password of Tomcat that needs to be fixed right yeah it's only on a hundred systems but it's still important right even though it's not on 5,000 systems and then we would pick three things that we're going to require projects because I knew the projects

were going to take forever so I like to throw a lot of those over and I didn't have to do them right I just had to tell people to do it and that wasn't included in my area and then we picked the top stupid system so we'd have the top offender and we were like alright this system has a thousand vulnerabilities on it that one just needs to go go I was like Corey go find this person tell me where he lives and we're going to figure out what this system does who owns it and how to make it either die or get patched to one or the other I don't care then we had our patch meeting so these

are the five things and I'm gonna talk a little bit more about these so first thing we do is review our previous work so this is my Bradley Fighting Vehicle and it actually caught on fire when it ran over an ie D and it was so hot the aluminum burst into flames on it so that's why I say the previous work so we look through what happened the previous week and we say look we meant to apply this patch we had 5,000 affected systems we remediated 4,800 of them why did these other 200 systems fail we wouldn't solve it in that meeting but that was a take away from me that became frustrated cargo that we'll talk about in a minute then

we reviewed upcoming changes and you might be confused by the picture of weeds but this was actually a swamp and people used to hide in the swamp and they'd had bombs and they'd go and push them out into the road from the swamp so I was tasked with removing the swamp which was pretty exciting so we know I didn't have a flamethrower but so we had to review upcoming changes right so we said all right what do we plan on doing in the next week right what's coming up what happened to the ones we submitted last week what's going in this weekend is everything on track for these sometimes the answer was no and we had

to pull these back out and resubmit them it's fine again stuff is going to go wrong it's gonna be okay just deal with it when it comes up we had our risk management committee and they would they were the ones who were sitting there just saying yes this is on fire and these are things we don't care about right which is a valid thing that needs to be done right some things you work in a fix I just wanted them off my books so I would say look the risk management committee said no they will not accept this risk we have to patch it we have to overcome whatever thing we thought was a problem why we asked for that exception

or yeah we have three months we're gonna kick that can down the road and deal with it later the secret cabal that was me sacrificing some chicken nuggets that are in dinosaur shapes it was the best cabal picture I could come up with so we just sit there and say look these are the vulnerabilities we want you to patch so everything that I said we came up with in the cabal I told them about it I would ask them if that was a good idea had somebody tried this kind of thing before the desktop and server admins had that tribal knowledge that I didn't have I'd only been at the company for like three or four months when this happened

to me so I didn't have that background they were kind of someone that I had to to lean on in those cases I'd ask them what are the blockers what's going on with these other things that we have to fix we had to confirm action items this is one of our Iraqi friends and he was trying to put out this fire with that bottle of water so you have to know what you're doing and make sure everybody has the right tools right say look what is your job for this week I need to do the following things you need to do the following things so make sure you confirm those action items sometimes my wife will tell

me to do things and I have no idea what she just said and so I'll like walk out of the room the thing never gets done and then later I'm like oh my god I was totally supposed to do that and I just dropped the ball so make sure everybody knows what they're supposed to do get confirmation they have to say yes I understand I am doing these things and usually I don't tell them what they're supposed to do here I say you tell me what it is you plan on doing in the next week right that way if they're active participants in this part of the meeting and then frustrated car go this is a

turret stand it weights like 5,000 pounds somebody put it on a forklift that held about six thousand pounds back the forklift onto a bed and then lifted the bed onto a truck so that was what I call frustrated cargo right is something somebody comes up with you're like the you know what no this isn't gonna work the way that we planned on it we've got to back up and do this again a better way so things didn't always work as as we plan right sometimes one of the things I learned was Windows patches have dependencies on previous Windows patches so you'd be trying to fix one vulnerability but because there was a configuration change a missing service

pack a previous patch that had not been applied then this patch wouldn't apply so this was the chief cause of people saying you know the client just rejected the patch it said it didn't need it so this is why that was not always true to dig through windows logs to figure this thing out and you can't just kick that over to somebody who doesn't understand the intricacies of Windows logs it required a lot of research on our parts so this was stuff that I did if something didn't apply I owned it right it was my idea for how to apply the patch if it didn't work that was my job and I think that was where a lot of

times these things these kind of initiatives had fallen flat in the past as people said well it didn't apply I don't know that's why we still had a hundred systems vulnerable them SOE no 67 in 2015 right people applied the patch it went out it just didn't go to these systems why I don't know but yeh like mission accomplished we can all go home not so much but then we just rolled those back into the next iteration so that would come right back in in two more weeks and we try again maybe we had to maybe we figured out that it was the thing that we had to have a project for maybe we figured out it was a piece of

the network that we weren't even scanning right these kinds of things so in the end Here I am I finally got this swamp to burn right and there it is the swamp burning down it turns out it took about 2,000 gallons of diesel fuel and a smoke grenade but yes you can set water on fire so we fixed a lot of stuff right we went from about 500,000 vulnerabilities to about 80,000 vulnerabilities over a period of four months which I consider to be a really great achievement considering they had been sitting there forever then the specific broken stair I was trying to avoid found out what I was doing when my slide started making it up

to quarterly executive briefings and and he was like well why in the world did all these vulnerabilities drop off they're like oh yeah Rebecca did that and then they were like oh my god stop doing other people's jobs for them right so the decision they made was get your butt back on application of security we don't care so that's what I did I went back over to app sack and they were like well we're not gonna do this stupid meeting anymore and they were gonna say they said well the server team they should just hire more competent people right and you can imagine what a kind individual this person was when he was like screw these

other desktop and server people they're just dumb right that's the wrong answer you really need to help people out right don't just kick them to the curb when they're having trouble if people never input can improve if that's kind of the the stance you take so they hired an entire vulnerability management team to do this with like 10 people or something reporting the person that I did not like and the vulnerabilities were right back up there to five hundred thousand several months after I left the company so if that gives you any idea like these things work you can do it without expensive expending a ton of resources so don't just sit there and suck it up

when people make these kind of silly arguments figure out ways around them and figure out how to get things done but do your best to avoid the analysis paralysis that will kill you and I've seen it kill other vulnerability management programs at other places I've been you're better off just starting to spray patches around the enterprise then you are just sitting there doing nothing right don't just roll over when people say that something can't be done it's totally possible you may have to do somebody else's job for them right and maybe it's in their job description and you're not specifically tasked for it but if you actually care about accomplishing the sometimes you have to do somebody else's

job for them and the scanner is probably not wrong at least not if it has admin rights if it's a store dad's tools the Apple talk but not so much with the vulnerability scanners with admin rights and that's it this is my cat Buttercup and my kids birthday balloons so yay

so what questions do people have

so the question was did we ever apply were there specific patches that we saw that just started wiping out other vulnerabilities when we applied them I don't know and honestly with my method I don't know that I would have realized it because I only looked at the top ones because I was trying to keep my time investment at a minimum and so I wouldn't have noticed small changes down at the bottom if something else entirely disappeared I wouldn't have even looked at I just didn't care I know I mean they were just everywhere yeah when there was so the question was how did I resolve friction between myself and are you talking about system owners yeah so the system owners

honestly most of the time I didn't cause very many problems and I was like look you can back out my patches I don't care like just go ahead it doesn't really matter to me I'm fine with it and I didn't really cover it but our rollout process it was kind of a phase thing right so you started out small and it rolled out to like lower environments and then rolled up so people who had good test environments these things got figured out early so that might be something that that we were able to catch early on in the process right when they start pushing the patches someone raises their hand and I would just back everything

out I didn't care I would rather lose an entire two weeks worth of work then sit there and make somebody lose their lose operability availability was definitely king at that organization any other questions Oh up there yeah so the cause so the question was why did the number of skyrocket back up and the reason that they stopped going back up was they stopped doing root cause analysis that was something that this organization was not good at right so sitting there and figuring out why something didn't apply whether there was a patch whether it was the network going down you have to do the five why's I thought about doing entire or CA class but figuring out the root cause of your

problems is so important in everything in IT because otherwise things breaking you just don't know why yeah last one I think and then I probably have to wrap it up yeah okay yep so how did I keep up with public disclosures of vulnerabilities I didn't care honestly I just had I had numbers and I was making those numbers go down that's all I cared about so when we in the past I have worked at places where I did have to care about that I use a lot of Twitter as some people who follow me on Twitter know I use a lot of Twitter so I use that to figure out when big things are coming out I also used to do a daily

threat intelligence update so in part of that my job was to explain to executives what things were that we're coming out that day so I look at news articles the internet storm center does some work but but yeah I'll be happy to talk to the offline if there's anybody else with questions I can definitely get with me after I'm super easy to find I'll be wearing an orange a Valera jacket so I'm like a giant walking traffic cone all right thanks everyone