Metrics that Matter: Confidence & Resilience in Your Security Program

thank you mike saw can you hear me now i think you can hear me now all right sorry about that perfect so we are going to move right along with our next speaker uh we have with us uh mr actually colonel john berger uh retired so i'm gonna go ahead and introduce mr john uh burger and we will go right ahead and go through so john we can bring a few individuals onto the stage we can't bring everyone onto the stage but what i'm gonna ask is if anyone wants to interact with uh john just let us know in the chat um and then chris would be someone who can actually bring you on stage and you

can direct conversation with john uh about what he's gonna talk about today i'm going to go right into it colonel retired john berger was the chief information security officer and vice president of i.t infrastructure prior to joining reliaques he served 27 years in multiple assignments including the sizzo at the united states central command from 2010 to 2012. as this is oh he directed the efforts of the national security agency hunt teams to protect and defend a war farting network of over 1 million devices in the middle east in 2012 he was selected as the chief of cyber warfare where he directed the cyber attacks in afghanistan iraq and the planning for offensive cyber operation against iran

colonel it's really good to have you here and with that being said i'll go ahead and let you take it away and whenever we have someone that needs to come on stage uh chris from dean summits can pull tomorrow all right hey thanks wilfredo hey the real hero in all this was uh wilfredo uh you know he we actually served together i found out just a few minutes ago and uh you know he was an operator uh doing all the high-speed stuff when uh when we were both serving together so thank you for your service uh wilfredo i appreciate you and um listen everybody um uh you know i've been the sister of a lot

quest for a few years um and uh you know when you're in the when you're where you're uniso job you you learn you're always learning along the way and one of the things i've found with the customers that i interact with is i always try to figure out like what are they being asked what are their challenges and how are they answering the questions that they get asked by their bosses by the boards are they using measures for operational performance and measurement are they taking measures to the board and how are we measuring the effectiveness of our programs and what i'd like to share with folks is some of those things and i'll share out

a screen or two here from time to time but what i really have had a lot of success doing is actually talking with folks because quite honestly um what i don't like is to join a session where the where the guy up front is a blowhard and you really never have any questions you don't get anything answered that you want to answer i'm not that person in fact this is a two-way street i'll probably learn a few things uh by listening and talking with you all um and that's just the way it works every minute of every day for me anyway so a little frustrated that that we were kind of doing this you know

one-to-many type of discussion but what i want to start out with is um just to kind of set this stage is i want to have somebody you know raise your hand and chris will bring you in what kind of questions are you being asked by your senior executives or you know by the board um and and what when when might metrics be appropriate to to bring an answer to that and at the end of the day we get asked all kinds of questions what types of questions are being asked i i have a kind of a super set of all the questions i've heard from our customers but i kind of want to get folks to kind

of start telling me what are you getting asked and and and what are you saying at the end of the day and then maybe get a discussion going on hey uh what might you suggest to to to someone to give give somebody that answer um you know we we uh you know so if somebody wants to to to get up on stage and kind of put themselves out there that would be great otherwise you're going to get the colonel being a blowhard right wilfredo well thank you so much and i think i can start this discussion um by asking one of uh the questions that i think a lot of people sort of get asked especially if you work in a sock

or any sort of organization does monitoring what is our time to action what's our time to resolution and how do we improve those or how do we make sure that the quality is good enough when we have a good time to action in time to resolution yeah i think um there's a lot that goes into that but you should measure it right so you know it's not that hard to measure when an alert and we do this even for our customers you know we'll measure you know we have certain slash we have to perform to and when we escalate um that's when the kind of kind of when the when the clock starts and then when

our customers respond that's when the clock ends now here's how that uh here's how that can work right so meantime to resolve is what most of our cisos want not just mean time to respond now if you have an alert that fires and it's a false positive um or let's just say it's anomalous safe and let me define it i'm i'll say if it's not it's something that a bad guy or a good guy could do in this case it turned out to be a good guy using us you know interactive log on with a service account would be something that a good guy could do a bad guy could do or scanning a network's a good guy does

it a bad guy does it normally when you get a false positive or a nominal safe as i just articulated you know you can respond and essentially that's when the clock stops if it's a true positive right or it requires further escalation that's where the time keeps ticking and so at the end of the day i typically want to break those out i'll look at the overall average but i'll typically want to see the time that i get those false positives adjudicated with a tuning and tuning action in and the anomalous safe where i go yep johnny joe it was johnny jones scanning again and um i can tell you from my network um i

measure the mean time to respond and you know i want it to be in hours right and obviously if it's critical i want it to be less than if it's high whether it's medium and on down the line um so i measure mean time to respond i'm a consumer of our own service so when we get escalated alerts i measure it and it's different between when it's a true positive or when it's an almost safe or false positive um now the question you ask is what's good enough um to me um i i believe and you know this um for the most part unless you have um some sort of unabated worm-like uh propagation of of malware

most hacks happen at the speed of humans um and so i just need to be a pretty quick human here and i think well freda you'd probably agree with me like a lot of people go oh you better have that done in 10 minutes don't get me wrong i want to go fast but i want to go right right i want to make sure that i get the right call on what this is before i i i go fast so effectiveness and accuracy in the assessment is much more important than um what i would say overarching you know over emphasis on speed hours is technically where i'm at right and uh i find that uh

when we get really too crazy around you know just getting an answer too fast we're gonna miss something and that's a false negative and that's way worse so i'm uh you know your thoughts on that wilfredo i'd like to hear from you on what your thoughts are on that yeah i think you hit a lot of great key points in terms of having good measurable metrics that actually makes sense that will help improve processes um something that i sort of have learned throughout my career is that there are a billion different types of metrics that you can take um and what might be important to me as the sock director is not necessarily important to the

board member um so how can i um sort of take what you've applied and i think actually craig craig mcguire asked that is uh he's saying you know the bishop is the biggest issue that he has is that people say i want to see metrics on the incidents we're handling i don't know what i want but can you just build me something to show right so we have all this data in the software stopping breaches weekly but then the board they want they just want the information presented to them how can we sort of help identify what is going to be important to leadership if they don't already know yeah no it's a great question now here's

what i'll tell you all boards are different right all leaders are different and you know as we as i've talked with our our my fellow cisos and our customers across 200 plus um i can tell you uh there's no one answer to this question um you're gonna you know you all know everybody on this on this call knows when the washington post or the new york times publishes a threat or publishes something um you can bet you're going to get asked about it so sometimes you know answering what's going on in the threat landscape is something your leadership wants to know right there are other my board happens to be very risk-centric and and i track i track 14 adversarial

uh risk scenarios quarter over quarter or a quarter and those include everything from a possible you know i now i have a great name for it solarwinds i've tried i've been tracking solarwinds like uh like uh exploit on my side because we write code for a long time but now i can call it the solar link now i can call it something because everybody knows what it means everything from ransomware to the solarwinds type of scenario to insiders um with intellectual property so i'm i've been a very you know you're going to get questions around you know here's how you typically get the question from the board when it talks about risks is what are you most worried about and

where are we most vulnerable right that's how you're going to get a question like that and what they're really wanting to know is you know where where are our highest risk the most likely most dangerous and welfare you understand this from the military side commanders always get that from their intel officer they want to know the most likely and the most dangerous well obviously if something is most likely and most dangerous that's the highest risk that's the definition of risk so you know for me when i look at my detection program my detection program is a very essential control set for almost every i can't think of one that it's not almost every single one of those risk

scenarios that i have right i've got i've got detection traps in place for ransomware i've got detection traps in place for our cic.cicd pipeline on how we how we build code and deploy code sign code all that kind of stuff to make sure i don't i'm not the next i'm not the next solarwinds so for me i don't know that i would be applying things like mttr um things like mitre coverage of attack chain unless your board wants it to me at the end of the day the board is going to is going to want to know very high level things like you know what are you worried about um what are our highest vulnerabilities

that's a risk question at the end of the day they might want to ask they might ask questions like how are you doing um how are we doing you know um are we prepared right that's a maturity question um you'll get a lot with a board around comparison right hey i'm in the i'm in the finance industry so where how am i doing relative to to my peers right that seems to be when i find that question being asked when and to be honest with you it's not a bad thing to ask but it seems to be asked when people just don't have a clue on what to ask right it's like i don't know what to ask

you but other than how are we doing relative to our competitors right or our peers in the same industry so you'll get the comparison piece but but again i'd like to hear um a little bit more did and did i answer the question or what else can i get back to me on it yeah no that that answered the question greatly and i know that we have a question from uh someone who's on the stage now rj45 um so i'll go ahead and let the uh open your face what's that hey yeah so go ahead yeah so basically i keep hearing about cyber rain simulators being all the rage like i guess similar to gns3 or cisco

packet tracer have you guys used any of that for actually safely showing like what a real world apt ransomware sort of threat or malware news of the day would actually do to the customers environment we we haven't done any of those simulations in customer environments right now it's not something that we typically offer um what we will do is we'll have smaller simulations that can be contest the detection controls that are in their their sim or their endpoint controls or whatever um so that's that's how i would answer that question um but at the end of the day most of the time i don't i don't know if that's a i don't know if that answered your question

we don't find that our uh that's not usually a question that the board's interested in i'm i'm interested in that but that's uh that's just uh that if that answers questions um let me know if you want to come back to me on that if you're asking yeah go ahead go ahead rj45 yeah but is that more custom built-in house for your team or is there any recommended partners that offer stuff like that for yeah like if you wanna if you're talking about doing any type of red teaming and simulation there's a number of them i like spectre ops it's a pretty high-end group um there's a couple of other high-end groups out there if you want to hit me

afterwards i can always get you a list of those um if you're looking for that kind of what i'll call simulation and or um threat emulation appreciate it what else uh what else are are you being asked anybody have anything wilfredo you keep coming in and out

for sure um we do have another there is a helmet um and i think it's actually we have a question but the best

so i think what they're trying to ask is how can we better incorporate operational security into our commercial infrastructure so that we can reduce the recon uh landscape yeah you broke up a little bit there wilfredo and maybe it's me um can can you repeat the question yeah so uh it's best way to get opsec into commercial enterprises i.e no emails to service providers with host names and ip addresses inserts on vpns with names like vpn for company x um and i'm not sure who asked that question but perhaps we can uh oh so it wasn't it wasn't a question it was just a talking point okay um i see paul's as paul's joined anybody that has a

question if you'd raise your hand i'd like to get you into the into the uh on the stage if you will and uh so um i'm uh i'm actually a room monitor i was going to repeat the question for wilfred until we get somebody else to join in uh i've been uh for i work for an industry now i used to be a retired air force um and i'm getting a lot of questions about ccmc's cyber security capability maturity model and how do we achieve that and i mean it's gone all the way up to cs through trying to figure out how we're doing assessment optimizations for our systems so what's your take on how industry is supposed to prepare for

ccm yeah so um it's the cyber security maturity model and just for the for the audience what that is is um so what it is is a standard that that that hasn't been met but hasn't been in place before so so if i'm an entity and i directly contract with the government i've always been required to to follow i want to say it's this you know something 171 dash whatever i forget the exact um um and if anybody knows i put in the chat um i want to say 800 171 or something it's 171 something that's if i'm a prime so think if i do direct direct business with the government i had a requirement

to meet certain standards as a direct vendor to the government but what's never been in place has been what the third parties to that prime vendor or the subs to that prime vendor um are required to meet certain standards um in order to be a sub or be a third party to that event to that prime vendor and so what cmmc does is it sets a standard for what what used to be what has been a very gray area which is certain controls would flow down as a requirement from the prime to subs but it was never it was never clear it was never a requirement so cmmc is the way that commercial industry and third parties

and subs and so forth will be required to meet a standard for cyber security and there's there's there's five levels but i think they've only operationalized you know cmnc level one there's a three and i want to say there's a five and i think they left two and four open because they they figure there's gonna be a few more um so the way you can get certified and here's the rollout like they're rolling this out um based on prime contracts going into uh the industry um year over year and in 2021 or let's just say fy2021 there's only 15 major prime contracts there are going to be that are going to be required to be cmmc

um you know compliant and so the question i would ask your boss or whatever is there's if you know you have to be cmmc compliant because you're going to bid and or the prime or you are a prime and one of the contracts that you're going after requires cmmc then obviously you have a bonafide requirement what i'm finding now is and even some of our customers they want to be cmmc compliant early because they want to to be able to market this as a differentiator that they're ready to be a sub to another prime so there's a kind of a marketing side to this um so how do you get it um there are a

number of what they call three paos third-party assessment organizations i think it's what it stands for um that will either a do a readiness test or be do the certification for you they also have to get certified and they're just now getting certified so um there's there's been a time lag on that the um and so that's how you get certified as you contact one of them um ours happens to be a line it's it's a it's a basically a risk and compliance company they're everybody's doing it now and you just gotta find somebody that has been trained and certified um as a cmmc authorized uh vendor and that's how you do it um but i would really check on the

timelines before i would jump right in on that the controls are all out there you can go through and you know you can get all the controls set it's it's not hard at all to find out what you have to do so you can do a self-assessment way before you would ever have to uh have somebody come in and certify you over appreciate the response

any other questions wilfredo i can start uh i've got some other things i can go ahead so how are you testing the people the processes and technology that all work together and and how often are you testing those people processes and technologies to make sure they're actually working efficiently yeah so a couple different ways so let's talk about you know i could talk about some of the metrics i use for my it sounds like you're interested in security operations right and these are the metrics that i that we use as a company um you know my log source coverage is probably my one of my most important metrics right so think about and and my vendor which is reliable measures

it for me so think about it this way if if i tell if you tell me you got 10 firewalls and i'm only seeing five of them in the sim that only five are logging then your coverage by definition you know it's 50 so i i use that across all of my sensors um and i need to be essentially 90 like you everybody knows that at any given time certain things can be off um you know you can have issues with logging so i'm i'm very very um we're maniacal about it but there's really no hundred percent here big deal is coverage have to have coverage the other metric i like to um i like to

to use for my detection program is miter coverage essentially of all the techniques um how many do i have covered for now i can have different use cases or call them alert correlations that cover down on multiple techniques and um so i'm looking for the techniques um and there are yeah i might have the i think i might have the the there are when they if you dedupe all the techniques there's about 227 last count that i recall uh techniques and what i'm always trying to figure out is okay am i can i can i see those techniques right do i have detection use cases in on all those techniques um so that tells me a lot about what's

the goodness of my plant is it is it you know of course is it uh do i have the right sensors in place and do i have the right correlations on those sensors to be able to to detect threats then you get into the basic availability stuff on your plan is it up what's up time let's make sure all those log sources are healthy gotta have that right so what's my sim health right that's a normal one nobody you know it's table stakes and then you get into the people people side of this and for me mttr is a big one um i can also tell you that your false positive rate relative to your your true positive and anomalous

safe will tell you a lot and then what's your mean time to tune right because if you're not tuning um you're gonna get you're gonna get bombarded with false positives over time so those are some that that i like to use that work really well for me is at the end of the day looking at your percentages of false positives anomalous safe and i have a pretty high anomalous safe rate because i want to see all the stuff that we have a lot of admins and developers and they do crap the bad guys you would think would do because they have privilege access and i kind of want to see it all um and i and i do

there's not much and it's good there's a deterrent effect there too by the way for insider they know we're watching because they get called up hey did you do this oh yeah it was me okay um did you get a couple out of that one wilfredo absolutely thank you so much and actually we do have some other people on stage so i'm gonna let them go ahead and ask their questions yep go ahead john hey colonel jonah goldsmith sorry for the covet haircut uh my wife likes long hair um former uh centcom contractor thing client and uh socom contractor how are you nice hey um used to work uh rob bill rodan um and uh over at uh d6 and um

uh excuse me so calm anyway um i'm right now at tenable so i'm your pc guy yes oh i know i was gonna say hey how are you um anyway um opsec good god instead of using like help systems and trouble tickets where it's based on let's just say salesforce where you could ask questions i got zillions of clients just basically dial you know putting uh post you know host names and uh ip addresses directly in emails um it happens often that you believe and um also even funnier than that um you have major banks out there uh basically with uh vpn um certs bank of the universe uh vpn system where you know you're basically um just

have to look up on showman for vpn so i mean how do you basically drive upset into the it and even the it security force this is not exactly covered on security plus or um you know your cissp questions so it's like yeah i don't know if there's i don't know that i have any silver bullets other than it's you know it's just a discipline issue in my mind right like um i mean there's no excuse for not encrypting stuff even you know today at all if you need to communicate something i mean oh 365's encryption service is freaking easy all of them are really easy so for me it seems to me that um to me this is a leadership issue more

than anything you're not going to solve it as a technician now you need to get visibility on it and i think you could probably get some visibility with some pretty simple regexes around ip address screens and and um and uh even hosting that might be a little tougher domain names you could probably do too right like so for me um that that's just uh that's a leadership issue no question in my mind um and a policy issue that needs to be enforced i don't have any silver bullets here like you could get into you can you can get into dlp right and and start you know we and we do for things that we know you know

for credit cards for socials for certain things but i would make the dlp self help and i know people will go well why would i do that yeah hey we screen this it's you know you you're not allowed to be you know we screen this and this how we do it for like socials or for credit card numbers we'll kick it back but it'll go to the user and then the user can release it if they want right and and to be honest a lot of times they don't they didn't even think about it and then they'll release it they'll release it if it's not actually the thing because sometimes you know these things are false positives

and the last thing i want is for my guys to have to go through and look at every single one of them um right and so at the end of the day if if somebody deliberately you know if it's a mistake and they go in and they say hey no that's not that's not a social and they release it fine um so that's how we handle it right now um but i don't necessarily see a lot of that other stuff going out it's just we encrypt the customers all the time right i mean it's just a service that we use so i don't know if that answers your question i think you should always be detecting

and and and use the automation to tell people that they're they're non-compliant then they can go back in most of them if they if they actually did something by mistake are going to then go back in and encrypt it so the response says hey your email was flagged for this reason suspected social if it's not in fact the social go ahead and click here to release if it is go back into your sent items encrypt it and send it on that's kind of how we handle it okay it's just a um i guess awareness issue on a lot of folks yeah i mean anybody who you know graduated uh socom or uh centcom or you know who worked

those that put the hostname in there yeah and you know those um guys and girls are now working out my other clients like publix and you know all around the state they don't do silly things like that but um yeah it's just something that the general population needs just um wanted your opinion on that okay hey thank you again yeah my pleasure so some other metrics that that i like to use look like some input on so um patching performance right so vulnerability management how are you measuring the operational effectiveness of that i do it to two ways but i'd love to hear others on this um one is you know you're never gonna be

nobody will ever really understand that probably at the board level what this means unless my board does they're pretty technical but um at the end of the day um you want to normalize the number of vulnerabilities you have in your infrastructure by buy host right or by endpoint so what are my number of vulnerabilities i i happen to to do an exploitable versus non-exploitable um by host right so that's a normalized number uh that i use and it's just math at the end of the day now in terms of standards and patching performance um i use the typical pci patching performance where critical is under 30 um highs under under 60 mediums under 90. we do much

better than that on average but that's the standard that i go by everybody knows there can be some fairly intractable uh you know vulnerabilities in the inventory so something that that i use for operational metrics any anybody else do anything interesting with respect to their uh their their vulnerability statistics that works for them that they would like to share so john i know rj45 is on the stage and he did have a question that got five upvotes so rj45 if you want to ask your question now you can open up your mic so with the work at home with the covid more do you see a lot of traditional on-site security operation is going more remote i think it's going

to stay both having more scenarios um like the dot splits for instance probably can't really support that long term but do you see that becoming more of a thing with other companies that can't as well as measuring more statistics um well fredo is it me am i getting is it my end i seem to be getting and i can't hear very well so i seem to be getting some so john what might work is muting your end when the speaker yeah we're getting output from your side and so rj45 if you want to ask that question one more time uh i think john might be able to hear it a little bit better okay so with the

kovid 19 a lot more companies are adopting remote work at home do you see that becoming more of a thing with security operation centers versus having the analyst drive in to the company to do it for instance or do you think some places like dod space are probably gonna have to keep that long term yeah there's two two questions there one um you know the commercial space um in my mind we're we're uh we're a physical facility security operations center company i believe in that i've always believed in it um i the there's just so much collaboration and um and value and having a physical uh a physical facility in my mind um i do i don't know where

others are gonna go i mean there's a the issue come becomes talent um what i tend to think is there seems to be kind of a honeymoon phase for work from home right now but anybody that's really worked from home for any amount of time me included like more than eight months to a year um if you talked if you were to survey all those folks they'd say they they don't really like to work from home full-time and that it's maybe it's a hybrid i tend to think um i know if i do business with a company and i want a security operation center i want it physical um and but but i also know even before kobe

there were what they called vsox virtual security operations centers um and i tend to think that um you know with the right mix of people can it work yeah i don't think it's as effective so that's that's a personal opinion others may have other opinions i can tell you that the dod is not gonna accept it right like at the end of the day um you know the department of defense is going to want to monitor all their networks in a given security operations center and they're probably going to be in the one that has the highest classification right so they only need one um so they're definitely going to be in a physical physical facility um

but well freda i'll let you answer that one that's how i would answer for the dod they're not gonna let you hang out at home

yeah i think that's pretty accurate um you know especially in dod spaces where you're dealing with higher classifications of you know secret ts systems uh you know that work has to stay in-house just because of all of the physical security capabilities and essentially the the modern day faraday cage if you will of buildings that that prevents sort of that insider threat um capability so i do think for the dod space the capabilities for virtual socks particularly in the more classified spaces it's probably not going to be very realistic and they're going to have a hard time sort of dealing with that but i know that they are looking at ways of being more innovative in these top secret spaces

and there's you know a live a wide range of research out there about can you really have a a top secret space with the technologies that we have today and there's a lot of testing about how can we lock down mobile devices so that we can have mobile devices and skips and things of that sort so i am very interested to see where that research kind of goes oh um i i i think it'll be a long time coming for a lot for the culture of the department to accept some of that back to metrics how many people use if anybody uses use phishing metrics and kind of as a training and awareness metric um for me when i get into training

awareness we all have to kind of do a very standard you know kind of training and awareness program um i try to get it down to as many key points as i can but the thing i want to really drive and train is fishing um for obvious reasons does anybody use fishing as a as a performance metric for the workforce i'm curious because i think it's a good one um at the end of the day awesome i haven't used metrics for performance based learning or training uh but have used metrics for fishing to determine where our security holes are right if the fish is coming from the same sort of infrastructure why does it continue to get in where can

we make uh changes to plug in those holes but i am interested to hear about how you know we can use those metrics to help train users on secured topics yeah obviously phishing there's a number of controls that are in what i'll call the defense in depth chain right so first of all a fish has to get through any type of security gateway if a fish comes from a new domain there's also things that that can prevent that domain from actually even coming into being right there's also there's also obviously the email gateway then you get to the human right so now we talk about whatever the human firewall right and how you how are you training

that right how you it's how do we configure the human firewalls that's kind of why i always talk about it but you know i've always told my bosses we can't rely on that um but we need to make it a control um and then of course you know you have your endpoint controls then for pivoting and all that kind of stuff you've got your east west and your ids your network controls and segmentation so there's obviously a defense in depth series of controls on any fishing scenario what i'm talking about is at the end of the day what what a lot of executives and boards are always interested in it it's probably more of an interest area

than it is like a big big deal is how are we doing in our ability to identify and and remediate fish fishing emails that get through and that now are in the humans uh purview and um just you can gamify it and it works pretty well um and it's pretty cool uh-oh speaking of a fish michael music yeah i don't know i don't know if i can y'all can sir hear me but um oh jeez yeah i got pulled up here because i put i put uh don't forget you know edr and b in that change it's when it comes to phishing at least from the attacker's side like a lot of things need to go right your

domain has to be good like you said colonel like if you spin up a domain with your with the targets um maybe commissioner you know it's gonna get flagged and then your domain has to be categorized right and then you know you have to go through it you gotta you know actually get through the email security gateway and you have to get through the edr then you gotta hope they click you know there's i got pulled up here i think there's that comment but yeah just there's a lot that needs to go right in that chain i'm also curious to know how many people are using some sort of maturity framework like cyber you know the nist csf or cmmc

or you know cis top 20 anybody using that when someone when when you get asked a question of like hey how are we doing or how are we measuring our program um anybody using anything like that i get that from a lot of customers the defacto standard seems to be the nist cyber security framework um you know another metric you can measure yourself against

not hearing anything wilfred do we have any questions i'm going through um i know we have a lot of comments on here more than more than just questions um rj45 did it his employer measures phishing but they're integrating awareness training soon so i think that's a good sort of segue we have this data of users the human the layer 8 firewall if you will so now how do we take those metrics and then implement good awareness training so that we could improve on those metrics

and you're muted john yeah so we're i'm sorry no go ahead go ahead ask the question if if the person that asked it is actually live here i'd rather hear from them go ahead yeah so we're looking to integrate that as well as more insider threats with um not only like how many times are specific employees attacked as well as fished weekly monthly quarterly yearly basis but yeah we're looking to go more beyond the fish with the awareness like how did they do on their annual awareness training or the follow-up ones so that's why we're looking into those metrics yeah i don't know i mean to me honestly i've been spent a lot of time

on on really you know getting after people around how well they score on the kind of the periodic training awareness tests or quiz um i really when it comes down to how much are you going to get out of your user base um unless you have a specific issue to me my focus has always been phishing and the reason it is is just look at any one of the breach reports or any anybody whether it's verizon preacher for mandy right they're all going to say and and and uh proof points or minecasts are all you know of course it's always mid-90s where you know which is the percentage of breaches that that are initiated through fishing so there's no

question it's a it's a vector that we need to worry about and so for that for that reason if i'm going to push my workforce in one area because attention is a scarce resource it's fishing it's fishing and to try to gamify it as as best as i can so that that's uh that would be my recommendation um and being you know being a little bit um less uh less aggressive on some of the other things now i say that not understanding exactly what your business might be and what things what risks employees in your business might be bringing to to uh to the business over yeah and i can offline that one later my

um only other question going back to the security operations center so i remember last year i guess a few of your employees were offering tours but had to cut back on that with the covid is that something your company is going to potentially do in the future to show other companies what like a real security operation center looks like or can you at least offer any recommendations on a company to build one out from the ground up i you know it's a good question i'd have to go back to my dod times because we built a bunch of them in iraq and afghanistan um like are you looking to build like a real you know

it's probably an offline conversation it's not how big you want to make it at the end of the day you know do you want a stadium step type of thing or or is it kind of small and you want to expand or there's there's a lot of there's a lot there to really understand what uh what what the requirement might be yeah and it's small for now but we hope to grow it over time but all offline i'll offline with you later thank you colonel i got one for you um maybe not necessarily this really applies to more managers director um type roles that may be overseeing these types of engagement engagements when it comes to offensive

operations like engagement so pen pass red team campaigns phone management you know not just label management but maybe it applies i mean what what in your experience is the best way to assign targets to those you know engagements such that you're getting the best bang for your buck i mean i know people have risk registers but sometimes red teams aren't using risk registers that's not how they're picking things sometimes it's their discretion it's a it's a director's discretion and and that kind of ends up with the red teamers spin testers not necessarily producing the best bang for their buck um so to speak you know like what's the what americans like best impact you know best impact um

these types of engagements how do you approach prioritization uh mike i i definitely think we should always be approaching it with most likely most dangerous and you know at the end of the day um you know we've we've been you know even for our company we just did an app that that looked at a couple of things um and so here are some of my my you know everybody everybody has theirs but obviously insider and intellectual property and making sure we secure our code base was a huge one we connect to many customers so can somebody get you know can an actor compromise real iquest and get into a customer environment because we support a number of customers uh

high-risk venture ransomware high-risk venture um so in my mind those are the scenarios um and there's probably only like four to six scenarios in every company that i would say are high to very high risk but most companies aren't measuring them um and and looking at the controls and that that that mitigate them so in my mind the best way to apply red team resources is to confirm or deny that the controls that you at least have in place for those higher risk scenarios are there um and to give them the targets that would have the most um impactful outcomes so to me it's a no-brainer but i don't know that a lot of companies know about it

and that's the that's the interesting because you know even with our customers i'll interview them and i actually got i walked through a number of them walking through our kind of a risk workshop to try to really focus in on the things that we should be getting more visibility around so that's my answer is there's no question that you want to be be focused on those things that are of value or or are our highest risk to the business um but you got to study it in order to be able to say that with clarity over yeah it sounds like it sounds like your offensive road map should align with your defensive roadmap absolutely yeah i mean why wouldn't it

right i mean at the end of the day you're trying to mitigate the the worst things and and so why you want to make sure that those things work so absolutely i would agree with that statement you know and we've seen it before i mean the other thing that can happen is um you you take some time to get some mitigations in place and then you run the red team but it should probably follow the defensive roadmap all right and now colonel we're about 10 minutes until the presentation is over do you have any uh parting thoughts that you can bestow upon us yeah i mean i think um i don't know that we got a lot into some of the metrics i

mean maybe we had some other questions and i'm happy to answer any questions for anybody but let me just kind of sum up where we're kind of what i've seen and talking with fellow cisos and and leaders in the industry when it comes time to kind of um you know hey what kind of questions are you being asked here are the types of things we talk about threat like hey what what threats are we most worried about hey are we vulnerable to solar winds or did we have to do anything when wannacry hit right you always have to be prepared to talk about threats you're gonna get asked questions on threads you're gonna get asked questions around

risk and you're gonna get asked them in very very interesting ways you're gonna get asked hey where are we most vulnerable um how effective are our controls you're gonna get asked how resilient are we like if hey if we went down how long is it going to take for us to come back up um what's our readiness to respond are we able to respond to an attack those are all really at the end of the day risk questions um they're just disguised um in in in in other in other ways you're gonna get asked questions around if i had a dollar if you had a dollar to spend where would you spend it are we getting the biggest bang for our

bucks you're gonna get money questions from your bosses and from the board you may get asked compliance questions if you're very if you're an industry that's highly regulated um so compliance can be an area for certain companies maybe it is maybe it's not um maturity is an area that you're going to ask questions around like how well are we doing um how are we making you know how how are we improving our program those are all questions that you're going to get around around maturity and then you're going to get things like hey how do we compare to our industry how do we compare to our competitors you're going to get comparison type questions i found over time you know

those are like the six areas that all questions that have ever been asked fall into those categories the latest threat so threat risk investment compliance maturity comparison almost every question that i've ever heard from a ciso will fall into that category one of those six categories and then when it comes time to key metrics things that i have seen and think are probably valuable to measure um you know things that you want to measure to reduce probability um are things like fishing performance we talked about things like patching performance and patching exposure those reduce the probability that somebody's going to be able to get into your environment um your controls coverage you know how well are your controls coverage is

endpoint working on every one of your machines how do you know are you measuring it what i hold our team accountable for 98 coverage of all endpoints and that includes bitlocker carbon black it includes a you know we use a windows defender firewall um carbon black protect there's we have four or five controls on the endpoint and i want to make sure it's working what good is it if they're not working and then you get into measures of performance that reduce impacts things like um you know uh visibility right so you know how are you measuring the visibility that you have in your detection program you can do that with mitre attack model you can do that with log source coverage

you can do that with false position rates how am i measuring my detection program so that if something happens i can respond and i can reduce that time that the actor gets a head start on me obviously um resilience is a big one to reduce impacts what's your mean time to recover right or percentage of high availability devices is another one that i like to use so those are kind of risk so risk there's a bunch of metrics around risk that i find to be helpful i'm a very uh risk-centric person investment that's going to be another area you can measure which is okay how much am i paying on each one of those 14 risk scenarios i've just

started to do that but it's a very interesting exercise and then maturity in comparison we happen to do a lot of comparison in our service to other other industry you know you can say hey here's where i'm at in my industry where are you compared to your peers um and then you know maturity in this cyber security framework is a good one that a lot of our customers use cmmi is another one and they actually have comparisons online so um just some of the areas some of the metrics that i found work over time um and that that would kind of conclude kind of my thoughts on hey what do we need to measure and what works for

what works for some of those work really well for me i don't use every single one but i've heard them from customers fantastic thank you so much colonel this was a fantastic talk and i really do appreciate all the insight it looks like we have one last comment mean time to eyes glass over is sort of one to be certainly be aware of know your audience when presenting that is from michael phillips that's always a good one [Laughter] but thank you so much for taking the time and i think with that uh this wraps up this uh wonderful talk yeah thank you alfredo thanks everybody