Vulnerability Management? Completed It, Mate! by Glenn Pegden

um hello everyone um brilliant management told partly through the the medium of the now 15 year old UK sitcom the in-betweeners if you're familiar with the in-betweeners this may make a very very dull boring topic a little bit more entertaining if you're not well a why would you come to something with such a pompous and stupid title as that and B I do apologize this won't make a lot of sense um to make it interesting start with the application who are my stuff I'm Glenn I'm very old um certainly compared to a lot of you I've done virtually every job in per second IIT over the last many years but for the last seven eight of those I've

been specializing in vulnerability management um I work for a company called flutter that you probably haven't heard of but most of this is about the time of the now flutter-owned division of Sky Bet and it was lovely to hear Richard thought before I actually call out the gambling industry for Action being at the Leading Edge getting stuff like because I feel that we are generally on the front edge yeah we I get a lot of horror stories for how bad everyone finding you up the keynote from Holly at the beginning you know her Horror Story is you know she still picks an SQL so it's the other end of that Spectrum actually we're getting things mostly right

um one of the reasons I'm doing this talk is a major vulnerability vendor um holds our program up as one of their examples for their customers of how to do it right so the fact that I'm touted out regularly at conferences by by a vendor to say go talk to These Guys these guys have actually nailed this I'll take that as a credit I know what I'm talking about roughly outside of a day job um um I am a big confidence junkie um if I'm not talking I'm attending if I'm not attending I'm volunteering and later this year we're doing resize leads we're bringing that back Mr Carney formerly of business leads is doesn't really need a round of applause

for for essentially what was even Sam's not in the room is she what started at Defcon I want to bring B size leads back I knew that I wasn't sensible and grown up you know to do it I got drunk at Defcon with Sam um actually then also in my in in the room here was was unintentionally part of that I said to Simon's like some I'm I'm bad I I I can do excitement I can do from I can do all the silly stuff but I need the grown up to actually organize me I I need I am Ben of new prison Newcastle I need a Sam Humphries and somewhere off next morning woke up to a

text saying hi I'm Rosie at the moment I'm organizing please I'd like to see but I believe you want somebody for leads I can be your grown-ups I already can't fit this in Liam is following me up and I already said I'm going to overrun I've done actually two minutes I didn't need to so um yeah so I do an awful lot of conferences I do love talking and organizing stuff like that but this place is special to me I grew up in Lancaster um I'm from the area I didn't attend the University here but this this is my real Hometown because I live in Leeds nowadays but this this was kind of special to me I often sort of joke about

the fact that I where I grew up was like grown up in a Peter K sketch but certainly in my sixth form years were very much like the in-betweeners um again if you're familiar with the show I grew up there was four of us four close mates first day of high school at a very rough comprehensive be slightly better off of offense did turn up with a Blazer and a briefcase on his first day we did have a little Tatty yellow car that we drove around in and shouted I'll send it to the people out the windows and I absolutely genuinely did spend my weekends at Caravan Club so as you in between the speaks to me on a

very very personal level so um anyway before I talk about how we do vulnerability management how we do it differently let's talk about traditional vulnerability management the way most places do it the way everyone starts off so they go down the CVSs route I'm too Fair we're just putting this slide deck together I realized why this was a really stupid idea because I can't use in between the memes because very few of them aren't offensive in some way or others so if you're familiar with the show you can work out what's under that sensor so anyway traditional vulnerability management what generally happens penalties realize that we need to do something like this they go out

and buy one of usually one of the three big tools out there they scan all the things they get all the vulnerabilities about and they get this unimaginably huge pile of vulnerabilities back you know a number so big it doesn't make any sense to them it's like you know a normal sort of medium Pleasant um organization it can be tens of thousands of vulnerabilities it's like where you start with that so they do the only logical thing they can they panic and then if you get the option they run away um for most of us nowadays the job market isn't that point that running away is an option so somebody actually ends up having to try and fix that so

what they do right we're going to look at that we're going to fix this and they throw absolutely loads of resources at fixing this they go out to the you know they're system owners their operations teams they're devs whoever is responsible and they they play on town they take up more and more of their time they're expected to dedicate more and more of their time to actually patching to keep these systems up to date and it very quickly feels like you're not making any probes it feels like you're putting huge amounts of effort and not getting any real game for it that's just treading work that's just keeping in the same place and as a

result you've underwear management team or the manager becomes one of the most hated person in the organization because all they're doing is sucking everybody's time and not providing any real obvious benefit um and that seems to go on forever no matter how much resourcing no matter how much you try you always say always well it's not great sort of stage forever um so yeah analysis of that way of doing it not great um so we always come up with something different back then it didn't have a name this was a thing we started sort of six years ago I can't take all the credit I've been built standing on the shoulders of giants but the last sort of 18 months it's got

a name it's now called risk-based relevant management we didn't come up with the idea well we did come up with the idea but lots of other people came up with it simultaneously we certainly didn't give it a name but it was only as this started to emerge and things like hey we already do that that's that's literally what we came up with so what is it what what what we actually got in there well we worked out the traditional approach wasn't working we had to go and sell something we had to try and get the system owners to to work with us in a different way we have to come up with a new way of doing this so

we came up with this pitch that was kind of different what we actually wanted from the team was far less patching than they were already doing we were doing away with unobtainable license for anyone that's sort of not already SLA service level agreement basically an agreement from system owners for how how quickly they'd pass certain types of vulnerability we we um we essentially let them set their own sis we said but this sort of thing how long do you think is reasonable yeah it is um you reckon you can fix them in a ways oh no yeah reckon you can do a year and they're like that's sure you're not gonna do a year it's like that's what

you're going to tell it and we let them set their own size which again it seems very mad we um discourage whack-a-mole patching to hit deadlines so welcome if you don't know what that is it's literally unprivatized patches but basically see how much patching we can do in one go we'll try and make these big men within the Little Numbers we'll take these hundred thousand vulnerabilities and see if we can get it down to 80 000 by the end of the month we did away with all that we stopped reporting big scary numbers in reports and we stopped sort of saying we have this number of vulnerabilities we did away with all that um um we prioritize the automation of

tooling over resource intensity patterns so if the team will say well we're gonna we've got two choices we're either going to miss our SLA this month or we're going to actually spend that same time building some automation that we can use in the future with a built the automation Sonic will cover up you know we'll explain away your si breaches if you're doing if you're if your behavior is correct if you're doing the right thing and we pretty much did away with punitive measures we're missing slas um you know it was genuinely um all carrier very little steak [Music] um and we changed the focus not from patching deficiency or patching efficacy well actually on risk and

Remediation so we stopped trying to say oh you know um this team has passed this number of things this team has it it was much more around this behavior is is they from the right thing rather than they were very good at patching this one so better definitely better now most of you in this room I'm guessing I'm home pursuing would be security people and you're probably sitting there thinking this sounds absolutely insane why would you do this why would you take an area that's already pretty bad and essentially take away all the controls take away all the or you know all the force you've got to make this better why we're looking to do that

well first of all nobody cares about vulnerability management I care about it very few people actually care about one of William and certainly outside vulnerability management yourself but they do care about risk management modern businesses understand risk really really well um you know it's a proper business thing now that risk management is fine so um we started switching to you know doing things in terms of risk rather than vulnerability um we moved to this sort of model because most vulnerabilities that your scanning tools will find in a lot of cases could never be used in your specific organization your specific setup in any part of valid attack chain yes the software is still vulnerable but

there's a good chance that you're not using it in conjunction with a certain module or a certain configuration setting or in a certain you know there are lots of wise and Wares for that your scanning tools won't be able to detect won't be able to know and if in a lot of cases they're just looking and going all right that's version 1.36 we know that's vulnerable we'll plug it up as wonderful even if it's not actually in a way that could ever be part of an attack chain um another reason that we switch it is that optimal isn't fixing the most vulnerabilities it's fixing the ones most likely to get your organized compromised first so it all comes down

to prioritization sometimes if you've got a choice between fixing 10 000 vulnerabilities or fixing five those five are the five you should go for because those are the five are going to get you owned they look bad on reports but if they're not really ever want you know like to contribute to that compromise why the hell are you spending time on them um on the SLA thing the fact that we we people set their own slas that was actually a little bit Smoky Moon you let people set their own slas they're much more invested in them they can't use the argument of that's unreasonable you know you're asking the impossible is that well you set these this was your thing

it's like if you now think they're unreasonable let's talk about that we may be our change in the middle but by letting them set their own they were much more invested in hitting them whereas if you just give them some arbitrary number they'll just miss it and go well I couldn't do it mate you're asking you you're asking too much um I mentioned mentioned on on our pitch that we we didn't really um penalize people for missing patching slas that doesn't mean that we just ignore it we do report upwards this sort of thing but we don't report oh they didn't hit their SLA they were meant to patch all they didn't what we report up is the risk of

that action the fact that you know this stuff hasn't been patched within a reasonable window the risk to the company is this it could be the company is insignificant it could be the fact that the rest of the company is is huge but it takes away from this we're judging them on their ability to patch and moves it on to we're telling you about the risk to the company um another reason that this is the thing to move for Google is that when I first started in this industry time to exploit was huge slas around patching were sort of 30 60 90 days and that was quite reasonable you know that that was you know generally a vulnerability to come

out and it take a while before it got weaponized and you know it was actually a real threat to organizations and over time that's been coming down and they're like the big Watershed for all that where everybody suddenly got look this was an entirely new game we were playing with local J because I remember um sort of the night before it really broke read about it online it was just a Minecraft thing you know it's a bunch of kids suddenly it was on Minecraft nobody gives nobody cared um doing the school run walking us into school a couple of people that I knew on Twitter can we take a look at this because this looks a little bit bigger

than Minecraft and by lunchtime we were getting weaponized tax balancing off while up at the outer edge and it's like hang on that's gone from this is some kids annoying each other on Minecraft to we're we're actually seeing weaponized attacks of this in 12 hours there is no point having a 90 day SLA if actually that means that you've got 89 and a half days where you're um so moving to this sort of model where it's like actually the stuff that we need to act on now will act on now the stuff that actually isn't really much of a risk will deprioritize that and act on it when things are less busy um so um

I'm about to say we've we've got the the actual system owners on board they think they're being sold a great new thing we've got security teams are all on um on board because you know hopefully you you people will now realize why we're doing this made lots of friends um the big question on this is that how on Earth do we actually pull this off how do we actually get to this mythical thing where everyone's doing less patching and generating less Risk by doing less patching well actually most of the secrets of most of the cleverness for this is in the report thing the actual mechanics of it don't change really at all you still go out from

there and buy your Coalition this is your um your next poll whatever um you still scan it in the same way you still get this absolutely huge pile of vulnerabilities it's all what you do with that data and we're generally you know most of the audience will consider themselves techies techies love numbers but actually numbers can be quite destructive if you don't understand what they're really telling you so as part of that we we stopped reporting on total number of vulnerabilities total numbers of vulnerabilities is a meaningless metric or not mean it's a very destructive and very um sort of um distracting metric big number must be bad not necessarily the truth and a lot

of the reason for that is churn and the Improvement management churn is essentially the number of vulnerabilities you get closed off each period so each week each month that then another equal number of that come back the next month in the new place give you an idea of how quickly that builds up something like a Windows roll-up patch you can easily patch you know several hundred vulnerabilities so your tools are going to tell you you've got several hundred vulnerabilities on something several thousand machines potentially said in the size of your estate and even if you've got nice automate patches if W sources kicking and patching those within two or three days of them coming out if you happen to you

know run your reports just after patch Tuesday it's going to go well you've got thousands of vulnerabilities in the two people you have um they're going to go away and then by the time next time you're on the report a whole new pile will come back so by ignoring That Tune and just reporting on things that weren't patched within the agreed SLA it allows you to deal with much more manageable numbers it allows you to focus on the problem areas not the areas that are actually already being taken care of um we then decided you know we did it much more prioritization and top down now if this was a longer time slot and if um I talk a lot more about how to

prioritize your vulnerabilities there are lots of clever ways of prioritization out there now we've built some absolutely glorious stuff that takes into account you know hundreds of different um metrics in doing that but we haven't got time to talk about that today so we'll just assume that all we're using is the cbss score on its own it's a really crap metric but it's the best we've got so um we start by saying okay everybody just focus on your criticals we want to get to a point where all criticals are managed within SLA so that everything we don't we don't care about everything just just put you know just focus on the credits and then you know like you get

on the control quite quickly because hopefully you don't have very many of those and then you move it onto the highs and you slowly keep moving that barrier um I think I'll and sorry just on that again this goes back to reporting there is if you're saying to the organization like just focus on your critics in your eyes there is no point reporting on those moderates and those lows because all they're doing is confusing numbers because the number of vulnerabilities versus the severity is kind of a logarithm thing you may have like five criticals 50 highs 5 000 mediums somebody look at our data your eye goes up five thousand that's the big scary number whereas in terms of

risk it's not it's the fact that it's the five criticals at the other end of the the actual real scary bit of that um focusing on automation means that actually you end up remediating a lot of the lower stuff for free so when people you know building things for um patching say all the critical well you know we can do this we can do this within seven days however problems this and if Bob goes on holidays or snoopers so we've automated it so that you know anyone can now press the button that does all the criticals and it's like well the tech you're using for doing those criticals looks just as well that has you yeah which is a lot of money but

it's kind of harder to not just use that same tech for everything so whilst at the start of this process they go into thinking oh my God we've got 100 000 loads to deal with I mean hey don't bother but by the time he gets that one because we've only got three thousand how's that happen it's like well I'll bring up your automating patchy new built at the beginning of it he's packing all the loads just as easily um risk acceptance again if if I got a longer slot this is something that I would talk about a lot more in a lot more depth risk accept technical risk acceptance is something I feel very strongly about business risk acceptance

fine I get it there's a good point for it it's useful technical risk acceptance especially for vulnerabilities is a nonsense it all is is transference of ownership transference of guilt sort of transition risk and you know sort of appeasing people's guilt we did away with that a vulnerability is it's either there or it's not if somebody can't if for whatever reason can't remediate that within their degree time scale right we'll write it we'll document it we'll say why but ultimately we're not going to go oh yeah security you know we get that we know you're trying really hard we'll risk accept that for you it's a nonsense it's like it doesn't the risk isn't accepted the

risk is still there all we're doing by accepting is make it a serious risk it's like what's your risk it's your stack it's your stuff so I've um very much against the idea of risk caps absence I do support the concept of risk resource sometimes you're um you know the prioritization you've given to a vulnerability that maybe all the mitigations you you haven't taken into account it may be it goes all the way down to well that thing over there you're saying hi um is high we can't you know it will never happen for us because we're never using that particular configuration like okay I'll buy that if that's never going to happen we'll rescore that all the way

down to professional it's still there the other your software is still vulnerable but it's now down at score zero it's information you don't need to do anything with it but it's still you know it's still there um again on the report by reporting on risk not the efficacy of patching it nullifies all those hard to patch arguments when people go oh it's a legacy system oh we're young we can't get patches for that we've got that you know it's like doesn't stop the risk being there so I'm not going to throw you under the bus because you can't work out how I patch yet but I will happily say this area you know this estate they stay

over here it's a risk it's a genuine risk we yeah there are probably good reasons why it's a risk and it doesn't stop it being a risk just because we understand why it's hard to mitigate it in other ways um getting teams to um to document why they have persistent SRE failures is actually risk acceptance or should I say fulfills the same role as risk acceptance without that transfer of risk so assistant SI failure things like things like why aren't you patching we can't patch this because we'll document that we'll write the risk up but it's ultimately it's all your problem it's still your risk not securities and it doesn't you know you don't get

this whole Catholic church I will absolve you of your sins sort of nonsense level it's your risk you want to do um and in the reporting we also very much focus on the right we we focus on people with the right behaviors not not the bad looking numbers so if you know people have built an awesome thing that hasn't really actually had any real benefit but open next year it will make it easier to build an auto patching thing at the moment doesn't really give the benefits but it'll show we report that because by shouting out about the right behaviors other bits of the company go oh well they've done a cool thing we should do a cool thing and he

has a little bit of games in there a lot of people try to gamify their numbers it's like oh they've patched more than you nonsense it's a stupid thing you end up with people patching to make Reports look good rather than actually reduce risk but if you focus on the behavior everyone's well we should probably do the right thing as well and it actually that perpetuates itself um there was one more the final bit with this is um everything I've said there make it all flexible at the beginning of this journey people will think it's impossible people think it's such a hard thing to do they'll remember these huge numbers so you set the bars very very

loose you know if people want a 90-day window for patching criticals suicidal but if that's what they want start them there because that's still better than not having one and when they go actually we can we can do that really easily moving down to 30 then then the seven and get where we are nowhere critical than generally patched under incident conditions rather than actually to win SLA but you people are scared of that big hit but bringing it down slowly over and over time makes it a lot more comfortable Journey for you and time-wise actually a little bit ahead of what I want so ladies and gentlemen that is kind of my little guy to wrist base Management in a nutshell I

for once have got to the end and we have time for questions um if you are familiar with the in-betweeners you're not asking that question well you can ask that question but I'm not answering that question and um I've got one or two Sitters in the audience so one or two people understand what's underneath that redaction I'm glad that's used this a bit because I can guarantee um you'd be asking questions about Lego yeah um so realistically actually any questions yes

I definitely encourage you um we very much have a policy that the how part of remediation is entirely up for teams if they want to hire an army of interns and have them manually log on the boxes and update stuff they can't we know it's stupid we're equipped into thrives on automation so we know that they'll probably do it manually for a week or two usage altitude they'll do it manually for a week or two and then um you know they'll look for automated because in our place everything ends up automated um however I mean I I do similar thoughts on this one's a lot of our app set program and stuff like that that a

little bit more but yes the more automation we can push back to them and not just vulnerabilities but as you say config management um deployment management the more of that they can do it it all feeds into each other it all gets you to that right that end point quicker um just probably you know basically I could do another two or three hours on that so grab me in the bar let's save a house party and I'll happily talk to you more about some of the stuff we did in that end of our questions

[Laughter] okay the reason I'm giggling that is I used to work with Gareth Andy he knows that I love PCI like I spend my I spend my time with me um how to be honest actually I how do I manage our expectations I found to be the best with you when you're absolutely honestly bored with them um one of the reasons that PCI was always a big struggle for us is that everyone assumed that we've got this lovely sort of pen machine pumpkin eyes tiny CD no we've got two slash 23 domains literally 8 000 machines worth of CDE so we have to be absolutely on the ball on everything and it forced us to to get it right and thankfully I mean

when when we started down this road certainly the case opening I'm also familiar with it was very prescriptive it always you must do this you must do that that's got to be exactly done this way thankfully over time it's got a lot more based around are your behaviors right not just how do you tick this box so nowadays it's actually a lot easier I get on really well with our qsa he he's brought into respective management um to be fair yeah five years ago I couldn't have sold this for an idea as an ideal PC I would have been no all but certainly all the highest critical he didn't think it was really doing it

would have been all of it nowadays they get this approach nowadays they understand it they're not asking for compensating controls on this they they they're caught with the times so yeah how do I manage that it's like by living now not living five years ago and I'm sure at some point I'll end up with an old-fashioned qsa that'll argue this with me five years arguments approaching used to it now done oh [Applause]