Solving the People Problems in Your VMDR Program

okay uh good morning everybody um glad to see so many people that showed up here for this first uh this first event that we have in southern Utah real excited about it um so I'm going to be talking about solving the people problems in your vmdr program because believe it or not the technology is not your issue um so I want to give just a little bit of background about me for those of you who don't know me uh my name is Jesse Harris I am a cyber security engineer at company called Dental intelligence they make software for dentist offices which is you know doesn't sound super exciting but I I actually have a pretty good time there

and I work with some pretty fantastic people um this uh is a slightly older picture of me uh it's about 1985 uh the older people in this room will probably recognize the commodor 64 that was the first computer I used uh we got our first family PC in ' 89 uh it was a glorious 286 from Packard Bell uh before they were really terrible uh and I got a hand me down for my grandfather about two years later so I've been I've been doing technology for a very long time uh I got my uh start in cyber security um at sanch back in 2011 uh supporting pgp software moved to RSA was RSA for eight years uh did a couple

years at microfocus uh you I imagine here it's probably a little less but many people in Utah have some Nexus to microfocus Noel or attach mate uh usually it's about half the room uh which is pretty funny um I my first cybercity incident was actually back in 1999 we busted a credit card fraud ring that was operating out of Phoenix that was that was really exciting that's how I got my first taste of working on that kind of stuff uh I do wear a lot of hats um mostly I've been working on vulnerability management this year is it's something that we didn't really have in place and I had to make from scratch and I thought it' just be a

good idea to share some of the stuff that I've learned here now I do want to make uh make an observation here this is something very insightful insight that was shared with me last week all advice comes with a free side of survivorship bias so I can tell you what's working for me and that does not mean it will work for you it might it might not so it's free take it for what that's worth so I have had to recently Implement a vmd vmdr program from scratch and I don't mean just get some software install it College day um the software is pretty easy we had to build a lot of processes around it because the

software will give you a lot of great information someone still got to do something with it there's no magic button that says yes patch please uh prior to this patching was pretty sporadic it was it was not really on a scheduled Cadence um it was usually no more than pressing the Windows update button which you know that updates windows but there's a lot more to patching than just windows out there what really drove this is we had a push for PCI compliance and any of you who've been through PCI compliance process you know exactly what that's like um the easy part was actually picking the technology uh I went out just you know picked top three vmdr

products uh tenable qualus rapid 7 quickly eliminated rapid 7 because it didn't meet our needs did a bake off between the other two that took a couple weeks that was that was the easy bit and then we' had you know about another couple of weeks for implement M ation so it's pretty pretty easy to do that the tricky part's acting on what we found so there are a lot of things that your technology can and will solve technology is great at automating processes for doing things way faster than humans can um works well for well-defined problems problems that don't have a lot of fuzziness or need a lot of judgment calls so if you think

about what a vulnerability management process would look like if you were doing it manually you would log into a computer you'd go look at what software is installed you check the patches you do that on every single computer compiled in a lovely spreadsheet because in information security we love our spreadsheets you would go see if there's patches for those products you would read all the stuff this sounds like a lot of work doesn't it but the technology is really great at going in scanning the system saying here's what you've got here are the patches that are available for it here's links to all the notes to read so that's taking care of a ton of the work right now you're still

obviously going to need to do some work at figuring out you know the priority of patches if there's patches that simply do not apply to you or ones that you've already mitigated for um I I've got to say that the the status of uh of vmdr program vmdr software is pretty good you you get a really comprehensive view of what's going on out there um you know patching is even not really all that bad anymore either we've got great technology to automate that because when you think about when you're installing a patch what are you doing you keep clicking next until you click click finish a computer can do that right and so we have a lot of great tools to

automate that there's platforms like InTune Jam manage engine jump Cloud there's dozens of them now um it can even be as simple as a scheduled task of you have Windows automatically downloading and installing the updates for you you can schedule apt on your Debian builds to automatically install for you even do the scheduled updates uh you know this is this is really great it's not like the '90s anymore where you have to handhold everything and these are all problems that are fixable by technology because they're predictable they're repetitive tasks and it's and you could have a human do it but it's way faster cheaper to have a computer doing it but there are a lot of problems in

your vmdr program that your technology is not going to solve and the problems are always your people this is not what this is not going where you think it's going so let's think about how this program is impacting your it team so I I think it's I think we'd all agree that software's gotten a lot better than it used to be right like 20 years ago you wouldn't install patches because you're like oh no that's going to just wreck all my stuff I'm going to lose data it's I'm going to be spending three days fixing it and patches they still sometimes go sideways but they don't do it quite as much as they used to but those of us

who've been around for a while and that's pretty much everyone over 40 You Know Who You Are um we've seen the patch has gone bad we've seen oh no my operating system went down my data is all corrupted I've got to pull backups from tapes this is going to be my entire weekend and probably most of Monday morning before people open the doors and when you go to your it teams they always come up with a lot of what ifs what if a patch fails to install and we can't get the application running again what if it corrupts the data what if we have to spend the whole weekend back backing out our changes what if we

have to rebuild the boxes what if the snapshots got corrupted what if our backups are bad what if what if what if the wha ifs come from a place where people have had these bad experiences but part of the problem is that those of us who've had these bad experiences we keep passing on these bad experiences anyone knew coming into the field we set these expectations of patching is a nightmare and patching breaks things even though it is better we like to throw punches at Windows all the time and make fun of Microsoft but when was the last time you had a Windows box just fall over dead and it wasn't Hardware or a bad

driver it it's been a long time I I can't think of the last time I've had that problem happen you know but we still end up you know talking trash about it and you know we need we need to have a different mindset there of what kind of impressions are we passing on to our colleagues and co-workers what kind of mind chair are we creating here that creates this barrier to patching we we shouldn't do that and a lot of the reason why it is in this scared of patches mindset is because they're getting blamed all the time when anything goes wrong it's the joke everything's working why are we paying you wait a minute now

everything's broken why are we paying you you can't win you don't get the credit but you do get the blame and so this is a problem within your this is if you have this mindset in your organization at something that has to be fixed like it needs to be able to get the kudos for things working smoothly not just all the blame for when things go wrong this can this can lead to some this blame it leads to long hours it leads to weekends and evenings doing this work it leads to burning your people out and a cynical work environment it's not healthy this is a people problem your technology will not solve this problem and it is not the only party

that hates patching your users don't like it either got a challenge for you log into your MDM platform and see when the last time your users restarted their system was it's been weeks or months I can guarantee you that Mac users you're the worst you won't reboot unless someone puts a gun to your head I don't know what to do about that but it's not like this is unfounded like when we think about it again going back to software quality the system still running even though it hasn't been rebooted in months so user is like well I don't I don't need my my system is working fine oh wait you're asking me to reboot I got a

zoom meeting in three minutes I can't reboot right now you want me to reboot now I I'm leaving to go pick up my kid from a soccer game I can't reboot right now your reboots are not happening because they're the users are being asked to do it at a time that's inconvenient to them there's no incentive for them we'll we'll get back to incentives in a minute Dev team teams are also resistant to patching and the de Dev teams are resistant to patching for very good reasons in this case a lot of them have intense pressure for features features features that's what the product team is usually pushing on them get more features get more features

you can address the bugs later we need features features features I got to sell to this other organization that wants the software to do this thing and they wanted it done yesterday and they aren't really given the resources and time to go into that backlog they are not given the resources now what's important to remember is that we are trying to fix problems we're not trying to go and fix the people the people are not I say that the people are your problem but I don't mean that that you need to go you know bust out your lar and go hit the users until they do what you want them to that's that's not how this will work we can't go blaming

them for getting in our way because we as security people we are enablers we are trying to make sure the technology is serving business needs right we want that's the whole reason it's there we're not given this playground of technology to make this perfect little Minecraft castle are we the technology actually has to do something it has to generate revenue and so we need to take the time to listen to these user concerns like users are telling us even if they're not verbalizing it what's wrong with our patching program and we need to take the time to sit down and listen to them and say say give me your concerns let me know how this isn't working for you

let's work together to find a solution you would be amazed at how much more progress you will get if you just talk to people and I know a lot of people go into technology thinking well you know I I just want to work with machines all day I don't I don't like people I don't want to work with people well you have to work with people the machines are not are not what you do all day the machines help people do stuff and you can fix machines easily that's why we like working with machines they we can we know oh this failed I can fix that but with people it is a lot harder that's a

lot harder nut to crack one of the things you really need to think about is you need to think in terms of incentives for a patching program and not punishments we um we get in this mindset of we need to create disincentives um if you haven't had a chance to watch it you should jump on YouTube and look up uh Sean Price's talk from say KH about shuhari yes this this is exactly what I'm what I'm talking about here so you've got to think what incentivizes people to get this work done and what you what you may have noticed from you know the things I've brought up from it and Dev is that they don't feel like they have the incentives

from the top to do it and so you really need to get that buy in what have you done to make senior leadership think this is important because if as someone on masteron put it very succinctly they don't want us sitting there talking about bits and bites they want us talking about dollars and cents what's the dollar value of patching I know we hate to think that way because we think oh that's what the suits think well you got to think suit you got to talk suit if you want to get things done if you are sitting there going blah blah blah about firewall rules and patching and cves your SE suite's going to glaze their eyes over

and they're going to be like shut up nerd and so that's that's a very important skill you should think about developing now once once a sea Suite sees that a patching program is financially beneficial you won't believe how quickly they buy into that right now we have been in the process of doing a PCI project and we are looking forward to so to getting in compliance with gdpr and we have full buyin and you know why because we are point we have got customers saying once you got sock to we're signing six figure annual contracts now that if you're if you're if you're the CFO you're like yeah yeah yeah yeah dock 2 what do you need tell

me I'll I'll cut the check today and that's the kind of that's the kind of incentive you need to have there you need to have that buyin from the top because now they're saying okay I'm going to resource the IT team to get all this done oh we need to do these things for the dev team okay let's think about getting those resources for the dev team because that's going to align with the business you might even get some you know additional headcount by doing that whereas previously people see that you're a cost center and that's really what we need to do is we need to make sure that we are making our value case

up there so hopefully this will play through yes just go ahead and play the all right it's gonna problem motiv I don't think I've got audio so it's not going to let me yeah I've got it plugged in but sound doesn't seem uh hold on this thing is tricky today well that's probably not going to play but um hold on anyway you probably have all seen office space right and so it's this this is the scene where Peter is sitting in with the Bobs and he's talking about having eight different managers to report to and they are just absolutely flumix at the idea that he's reporting to eight different managers and he says well you know really my only

drive is to not get hassled that or lose my job but you know that will only make someone work just hard enough enough to not get fired and so we have to think about our motivations here we need to think about what the incentives are for everyone to do it and that's like I said that comes from the top if we create the incentive of you know if we do this in our security program then this is where it creates dollars for us this is where we're reducing risk you know that's the kind of thing where the C Su says well how can we make this happen for you and these incentives matter incentives work a lot better than punishments do

there's there have been so many psychological studies on this that positive incentives will get people to do things in ways that negative punishments will not absolutely will not and it's important to understand that that you need to have those incentives in place and that has to come from the top once everyone sees that there's a benefit to the business they will work with you and create those incentives incentives you know those incentives for the IT team may be hey maybe we work on building a more robust infrastructure so that you can just patch any time during the day during your normal work hours rather than we have to wait for you know off hours in the evening or the weekend

that's a great incentive for the dev teams it's hey you know we will give you the time to work on these on this backlog of stuff and maybe that makes it easier for you to develop the features so that you're spending less time doing your work that's really what it's about is we need to make sure that we are enablers in the organization now this is going to be a really I'm going to say that vmdr is tough to do um I I haven't yet achieved everything I've talked about today uh we've only been doing this for about four or five months um and you know people often teach things that they wish that they were

doing uh we we like to present the idealized version of of ourselves when we talk about this and I know that I'm doing that here but you do need to lay the foundation one piece at a time I think the IT team is always a good place to start because these are people of where you can talk the technology they understand this is where the risk is and not patching here's what we need to do to get the patching done and that is a good way to start laying that that foundation and you need to make sure that while you're doing this you're documenting your work you're documenting your results because you can take these

results to other to to leadership and to other teams and say hey look at what we've accomplished let's do the same for you and now we've got it to a point where in just four months we've burned down thousands of vulnerabilities that tenable is yelling at us about we have got patching on a regular Cadence now we're using our maintenance window every Sunday to make sure stuff gets done we're starting automation of where I click a button and just roll stuff out rather than having to wait uh we we have achieved a lot in that short time and it's been fast because we've had that executive buyin and this is the most important part because we have

really good working relationships with each other you've got to establish those working relationships because again it's the people people who don't like you won't do what you ask them to that is a fundamental truth something that's uh something that I've learned in politics which applies in office politics too is it's the art of getting people to like you so that they do what you want them to do and I have had to be very intentional there and make sure that we are doing those small steps uh I've started making some inroads with the development teams of you know introducing myself to them and talking about you know hey what are our security goals how do these align

with what you want to do you know how can we how can I help you don't go to somebody with a security issue and say you need need to do this that is a way to make sure that somebody's going to be like I'm not doing that now because you told me to you need to go to them and say how can I help that mindset that language changes the entire relationship and we I have had you know great conversations with our Devon QA teams of they're excited about patching which you know that's usually people are like oh I'm I'm terrified of patching I want I think things will break but they're seeing that we are trying to

make sure that things work well and we're trying to help them do their job and get stuff off their plates um I will warn you that we're a small organization we can we've been able to move pretty fast if you're in a larger organization you're looking at multi-year projects don't let that get you discouraged think of every break everything down into small steps and be like you know what if we are patching better this week than we did last week we're moving forward and we'll get there eventually it's a marathon and it's not a Sprint so make sure that you are pacing yourself appropriately thanks everyone for coming out here today really appreciate you if

you have any questions comments or rude noises please let me

know