Measure what matters - Mark Overholser

You don't have the gut. I don't have the gut. Yeah, I'll stand sideways here. Um, appreciate that very much. Uh, so I had six years of it before I went into infosc. I've been infosc for about 12 years now officially and this is just downing years after graduating from college. I was an information security team lead with Medline Industries in the Chicagoland area for a couple of years. Uh then I went into the vendor side of things being a systems engineer specialist with Light Cyber and then PaloAlto Networks when they bought Light Cyiber. uh and then I came to Corlite where I've been for almost six years now in a couple of different roles and I

also have the distinction of for the last couple of years uh I have been a threat hunter in the black hat conference network operation center which is a lot of fun uh but it's also very challenging um so I've been to six six black hats worldwide uh since spring of 2023 uh which is a lot it's a lot of time in the air to do all of that just got back from blackhat Asia a couple couple weeks ago. Um, so why measure? We're talking we're here to talk about measurements. Measurement from a security uh perspective and information security perspective. So why measure? First, I say without data, you're actually just guessing, right? If you don't have data about what's going on

with your information security organization, you are just guessing about whether or not you're doing a good job, whether or not you're improving, whether or not you're adding value to the business. You probably are, right? We all know that every business needs an information security uh organization. But you also need to have data about your information security organization so you can prove it to the business, right? So measurement also focuses you, it forces you to agree with the business on what is of value and how to talk about what is of value, which I think is very important because it makes it a lot easier for you to continue to agree going on down the line. And uh as I

already mentioned, having data places you in a position to defend your team's value when it comes time when they you know a lot of people uh information security and it is a cost center right so you need to be able to demonstrate what is the value that you're offering in exchange for that cost so that the business can uh agree to continue to fund your pursuits right so first things first in the how to measure category my first tip is always make sure that you're comparing apples to apples Okay. And the I think the biggest thing there is if you start to measure something and you're measuring it a certain way and then you change the way

that you're measuring it, you can no longer compare what you're currently measuring to what you were measuring before. You had an apple and now you have an orange and don't compare them anymore. Right? So when you agree on the definition of how to measure something, stick to that definition and understand that if you do change that definition, you kind of have to set what you had before aside into its own category and start a new category for the new thing that you're measuring and then start to build that data up. So always make sure you're keeping that in mind, measuring, keep comparing apples to apples. The other thing, I've already sort of alluded to this, be very clear about

definitions. You have to sit down and you have to talk about what is the definition of an event? What is the definition of an incident? How are those different? How do we determine where are the boundaries there? Because you are going to get into situations where you think something is an incident and then the application owner comes into the room and they say this is not an incident. And why why do they say it's not an incident? Because they look bad, right? So if it's not an incident and they aren't responsible for the incident, that's one less thing for them to have to worry about. So you have to be very clear about what those definitions are

and you have to have of course executive or management buyin on what those definitions are so that when those conflicts arise and you're in the room, you can point to the definition and say it meets these criteria. That is why I believe it's a cyber security incident and force the other person to declare their stance and then you can appeal to management and say okay what is it at the end of the day you have the call what is it so be very clear about definitions for example I pulled this definition from the uh NIST framework for improving critical infrastructure cyber security version 1.1 it is a seven-year-old document but it's a good starting point a cyber security event

that has been determined to have an impact on the organization prompting the need for response and recovery. I think it's a good start, but what is an event, right? You have to define that too. And then uh determined to have an impact. Please define impact. Right? There's lots of fights you could pick with a definition like this. It's a good starting point, but you can take it and start to refine it and provide examples and document that and get sign in so that you have a definition of what an incident is so you can determine whether or not you have one at any given time, right? Another thing for measurement I say be smart and I borrowed this from the

project management world. SMARTRT means it's an acronym for specific, measurable, achievable, relevant and timebound. And I like I said I borrowed this from project management world and it is a way to uh assess a goal and understand whether your goal is well defined enough right is it specific? Is it very specific? Is it measurable? Of course, I'm saying things we want to measure in uh information security. Obviously, they should be measurable. That's only mildly tautological. Uh but we need to make sure they're specific and measurable. Achievable. I'm going to scratch out. We're just going to draw a line through that. We're going to ignore that for now. We want to make sure that they're relevant. We'll talk about that

a little bit more later. Make sure that the things that we're measuring are relevant to our message and timebound. This is something I think a lot of people forget is that we we really want to make sure that we're drawing time boundaries around things around predictions around what we're measuring from the past. You know this we had this many uh cyber security events. Was that in the last week? Was it in the last month? Was it in the last quarter? Was it in the last year? What was it? Right? We spent uh this much time on cyber security events. Was that total time? Was that average time? Was that time per analyst per week? Was it like lots of

things? So, make sure that when you're measuring things, you set up your definitions to be very specific. You set your measurements to be very specific. Uh you make sure that they're relevant and you make sure that they're time time bound. Now, let's get into a few mistakes that we can make when we're measuring things. Mistakes are very easy to make. So, don't be upset when you make one. Everyone makes them. But here's a few that you can keep a lookout for. It is very easy to make the mistake of measuring what is convenient instead of what is important. People make this mistake in every aspect of their lives. They go and they find and they say I'm

going to measure this. Right? I think probably the common example the example my daughter would want me to tell everyone is lots of people measure the size of their bank account instead of whether or not they are happy with their life. Right? So make sure you you measure what's important. And you always hear those stories about people, they get to their deathbed and then they think, "Oh, I wish I'd spent more time with family, right?" So, uh, a couple examples here. My appliance gives me this statistic, therefore, it must be important. No, that's that appliance probably gives you that stat statistic to let you know that it's doing something. And it also probably gives you that statistic because it is

available, right? So, the more specific example I have here, there were 7 million blocked connections on the firewall last week. Who gives a crap? What does it mean? What does it mean that there were 7 million? Is that more than the previous week? Like, what is the narrative that you're trying to tell to management by citing that statistic? Is it that the firewall is good? Is it that the firewall is doing a lot of work? Is it the that the firewall is overloaded and you need money to buy a bigger one? What is it? What does that statistic mean? Right? So, don't just go reaching for statistics numbers that are available. Stop and think, does this

statistic support my narrative? That's one ask. The other more important one is, is this statistic actually important? Does it answer a question? Right? Another easy mistake that people frequently make is not realizing that they are measuring a proxy. And I'm not talking about a network proxy. A proxy is a substitute, right? In the in the case of a network proxy, you're you're think you're connecting to server A, you're connecting to the proxy and the proxy is connecting to server A, right? It's a substitute for the thing you're trying to connect to and it's there usually for pro for caching purposes or something. But a proxy is a substitute. It's something you expect tracks the metric you want, but you measure it

because you can't directly measure the thing you want to measure, right? It's fine to use proxies. It's fine to measure a proxy statistic, but you need to know that you're measuring a proxy statistic. You need to be aware of the fact that you're measuring a proxy statistic. Um, so my example here is actually not uh network specific or network security specific. In the US, for example, this one I I don't know if people actually think of this as a proxy very much, but when you go uh to buy a six-pack of beer um or uh a carton of cigarettes or you go to vote, what do they ask you for? ID. They ask you for an ID and that ID

has on it your

Exactly. He has the emotional maturity of a 10-year-old, self- admitted. No, I um Right. Your ID has your date of birth, which can be used to calculate your age. We have substituted that in the United States for emotional and intellectual maturity and education, which are the things that you really need to be able to make the decision. Is it a good idea for me to smoke this pack of cigarettes? Is it a good idea for me to drink that six-ack of beer? or maybe just one of the beers in moderation. Uh or to gamble, right? There are lots of things in the United States that we we have decided to time bound them to are

you old enough because we're using that as a proxy. Generally speaking, as you get older, you become more intellectually mature, you become more emotionally mature, and you gather more experience and education from your life. And therefore, you are better prepared to make those kinds of decisions about higher risk activities because we cannot expect the person at 7-Eleven to sit and have an hourlong conversation with you to judge whether or not you are emotionally mature enough to buy a pack of Marbor brought to you by Marbor. Just kidding. So, that is the idea of a proxy. Always make sure that you are measuring the thing you want to measure as closely as possible and keep an eye

out for proxies. And if you're measuring a proxy, that's fine as long as you have no other choice. And be clear about the fact that you are measuring a proxy and acknowledge that there may be some error involved. Right? I for a long time I was measuring um this was back in the the Windows XP was going end of life. We had to get it off the network, right? because we're worried that it uh its uh support was going to end and then there are going to be vulnerabilities discovered that are not going to get patched and so it becomes a higher risk activity to continue to run it. So, I was tracking how many Windows XP

machines we had left to get rid of, but I was measuring that by going into Active Directory and looking for computer objects that had logged in within the last, I don't know, two weeks, a month, I don't remember exactly, that were tagged as Windows XP based on what was reported to the domain controller when the machine authenticated. Why is that a proxy?

It updates not in real time. And your point was that not every computer running Windows XP was necessarily domain joint, right? It was a proxy. It gave me a good idea of where we were in the project. But just because that number was zero doesn't mean that we didn't have any more Windows XP machines, right? There were probably non-domain joint machines. But I did not have, you know, tanium and I had no guarantee that even if we had something like tanium that absolutely everything that was corporate owned was going to have that agent on it, right? So there was always going to be some amount of error there. Another easy mistake to make is placing incentive to hit benchmarks.

Once you start measuring things, things like meanantime to resolution, you're going to have this idea maybe Maybe if I incentivize my team to work faster to get a lower meanantime to resolution, maybe our stuff will improve. The problem with that is what you need to keep in mind is Goodart's law. When a measure becomes a target, it ceases to be a good measure. What I mean by that is watch out for perverse incentives. Once you place an incentive for someone to hit a measurement at a specific target or level, you give them incentive to do it, but you you may also give them incentive to cut corners to do it, right? Like I'm going to click

acknowledge even though I haven't read the ticket because the sooner I click it, the better that metric is and the better I look and the closer I come to getting that bonus you said I could have if my acknowledgement number comes down, right? But what you really want is for me to read the ticket before I acknowledge it.

Tenable will flag more things in which way? Positively or negatively? Gotcha. So if you turn on Gotcha. That's that's a great example of a of a perverse incentive is that if you turn on encryption, you will possibly get more findings about the encryption you've turned on in your tenable console in your active vulnerability scanner console and management wants the number to go down. So this is a perfect example of this. Perfect example. Management wants the findings to go down. You click that button, they're going to say, "What the hell did you do? Undo it." Right? Nobody's ever gotten that call. All right, he said sarcastically. So keep that in mind. Measure to measure and measure to

understand, but be careful that you don't try to start compensating based on your measurements because then you create you create an economy where people will give you what you want rather than the truth in some cases. Okay. So what to measure? Security of course offers value to the business not just in like dealing with incidents and things like that but also saying okay like my example earlier Windows XP is going end of life we need to get rid of it we need to move to Windows 7. All right so in this case we're saying we're going from Windows N to Windows N plus one. You want to track KPIs over time how many Windows N

installations still remain. And also you can calculate things as long as you're noting this like you're measuring at the same time every day or at the same time every week putting this in a spreadsheet you can calculate things like velocity like how many are we dealing with per day? How many are we dealing with per week? And how is that charting over time? Is our velocity increasing? Have we plateaued? Have we hit our stride? Is our velocity decreasing? And you can ask other questions like is the velocity decreasing because we're getting worse at it or is it is it getting wor you know the velocity decreasing because Jim took vacation or is it uh decreasing

because we're hitting the long tail we're finding it harder and harder we're getting towards the end we're finding it harder and harder to find machines and we're getting to all the exceptions and we have to deal with all the exceptions right or something like transitioning from an old VPN appliance to a new VPN appliance things like daily active users on each and what is the velocity of the move from one to the other things like that. So by measuring those KPIs, you can add value to management by giving them things like pretty charts. We make fun of management all the time for liking pretty charts in nice colors, but charts can convey a lot of information

in a very small amount of time. And so if you track things like this and you turn them into a chart and you give them to management, they will look favorably upon you and they will remember that when it's budget time, right? They will be less hostile with you. Now we get into some like incident response things, right? If you're going to be dealing with things like security events, security incidents, you want to be thinking about metrics from a security in uh incident response perspective. Meantime to detection or MTD is kind of straightforward. And like I said earlier, we want to make sure that our metrics track a question or answer a question. So this answers the question,

do we have good visibility? Right? And is our visibility improving? If you're comparing your MTD from this quarter to last quarter, you can say we put in XYZ product or we took out XYZ old product. Did our did our meanantime to detection improve or did did uh did it get worse? Why? You start to ask those questions. So meanantime to detection is in my case I have declared it as the time from the estimated start of the incident to the awareness of the incident. You don't have to define it that way. You could define it other ways. And uh it'll be a little clearer once we get to a couple timelines. Um but make sure that

whatever you declare to be that you're consistent with it, that you're happy with that and that you keep in mind that you if you change it, your old measurements are invalid to compare to your new measurements. Right? Another potential one here is meanantime to acknowledgement. And this answers a question for instance, do we have enough analysts? Right? Are they able to look at the tickets quickly enough? And it is the mean I have defined it here is the mean from the time from ticket ticket creation to acknowledgement right time from ticket going into the system to somebody clicks a button that says I have looked at this ticket another one to track there meanantime to what mtr there's lots of

definitions out there for that one meantime to response meanantime to recovery meanantime to resolution or remediation uh ultimately I think this answers the question are we dealing with incidents quickly are we getting them done? And for response, I'm saying that's the time from detection to the start of incident response. From the time from the detection, however you define that, to the time when you initiate response, not necessarily complete it. And for remediation, I'm saying the start time from the start of incident response until you are done with incident response. And those tell you two different things, right? Are we making decisions about what to do and how to respond quickly? That's one. And the other is once we've decided what to do,

are we doing it quickly and effectively? Those are two different answers. Uh and then resolution. I am saying here this is different than remediation. Remediation is when you've gotten to I'm done working this ticket. Resolution is time from the start of instant response to all systems being at and I'll say an estimated 90% of pre-inccident capability. Right? You can choose from among those. You can refine them. Again, whatever your definitions are, they're for your organization and uh and nobody else can tell you that they're wrong as long as they're working for you. Now, timelines are a little hard to read on this one, but the point is we have a timeline here of an incident.

User clicked a link in a fishing email. Then a couple minutes later, they entered their credentials in a fishing site. Then, uh this is a few days later, there was a login to a virtual desktop exposed to the internet using those fish credentials. Then couple minutes after that, the thread actor tried to RDP to a domain controller from the virtual desktop and they failed. So then they gave up for the night or for the day. Couple days later they log back in and they realize that virtual desktop has access to an ERP system something like SAP for example that has invoice data in it and those credentials that they fished also let them into SAP. So they

log in, they start downloading invoice system and extracting it and copying it over the RDP session out. Then several days later they figure out that they can RDP to a domain controller via a different mechanism, different username and password perhaps. Then they spend a few minutes about 10 minutes uh setting up a group policy object to deploy ransomware across the organization. If nothing was detected here, you detect here. Very bad for the organization. And here's our example of time to detection. We're saying the incident started at the time that there was actual impact when there was actually a login. Right? If there was never any login, this is my argument and you may disagree with me.

If there was never any login to that RDP uh to that uh domain uh not domain control to that virtual desktop, then there was never any impact. Right? If the credentials were fished but they were never used, I argue there was never any impact to the organization and therefore according to our original definition there is no incident. So the incident began when there was a login when there was actual use of the stolen credentials. You could say that the incident maybe started at the time of the invoice data being copied. Maybe that's an actual impact. Again, we have to get back to what is our definition of impact to the organization, right? That might be something we'd have to sit down

and flesh out. Taking a look at another incident. This one's a little more realistic in that here we have question marks. We don't know what happened there. Detection happens when there's an alert for the user's computer scanning the network. We see acknowledgement. This happens over the weekend. Security analyst doesn't look at it until Monday. That's where we define acknowledgement. Response happens five minutes later. They decide to quarantine. Then we have resolution down here. The uh the PC is replaced and the credentials are reset. Right? So now we can get into pointing out those intervals. Time to acknowledgement, time to detection, time to resolution, remediation, those sorts of things. And one last thing I want to get

into here is the idea of measuring risk, which is kind of a hard one. I struggled with it for a long time when I was building security organization. Um, and the idea is it's answering questions. Are risk levels acceptable for the business and are we improving the risk profile commensurate commensurates with what we're spending to do so? That is an important question for a business. That's a very business question, right? I'm getting a benefit. How much am I paying for it? Am I paying more than I'm getting? Right? I exploit expected value from statistics as a simple risk calculation. Risk is impact times likelihood. But this requires you to estimate likelihood and to estimate impact which might be hard.

My first stab at it, I used probability. I I created a risk register. Yes, it was in a spreadsheet. I did not use a database. Um spreadsheets are databases. Fight me. Has the item. I rank a probability one through 10. Then I rank an impact one through 10. I multiply those. Then I get a value one through 100. This allows me to very easily sort. So it's useful. I wouldn't take a picture of that one. I would take a picture of the next one. But you can take a picture of this one. So you can compare to the other one and say this is why this one's better. Um I'm sure it did. Yeah. But

there's a better way to measure this. Right? This gives me I can sort easily. I can say this is the highest most important thing. This is the lowest most important thing. But it doesn't allow me to ask the question when I improve one of these things by putting in a measure is the money worth it which is why we have to think in dollars this is the way I think now probability as a percent likelihood in the next 90 days time bound right I say forget or lose the password you use to encrypt the private keys half a percent likelihood in the next 90 days but the impact is you lose all the in the wallet. It's

$500,000. So, the expected cost is $2500. So, if it costs $50 to buy a password manager for the year, and it costs I don't know what it costs to get a safety deposit box, $500 for the year, I don't know. But even if it's $500 for a safety deposit box and $50 for a password manager, $550 is definitely worth spending to save yourself 2500, right?

Yes, exactly. That is exactly what I'm doing. I'm calculating this impact value is how much does the business lose if this thing happens times the likelihood that I think it's going to happen in the next 90 days. I multiply those and this is the this is the amount that I think it and then I can rank them, right? Right? I can sort and say this is clearly the one that I need to focus on the most because it's the one that's going to cost us the most based on these calculations. And I don't have a column here for that, but I can have another column where I say the mitigation that I'm proposing is X and it's going to

cost Y over Z time frame. And then we can very clearly understand are we spending $10 to save five or are we spending $1 to save 10. Right? I am making them up. Which takes us, it's a perfect leadin to the next slide, which is what if I don't know? What if I don't know what the likelihood of that is? What if I don't know how much the impact is going to be? What if I don't know when the incident began? What if I don't know what the user did before their machine started doing all that crap on the network? Guess. That is counterintuitive to the information security community. I say to the extent you can know, no. But when

you can't know, yes. and be clear about the fact that you're guessing.

Well, yes, you can use a quant a qualitative value as opposed to a quantitative value. But I think to do this math, you have to have a quantitative value at some point. So that's my this is my last slide. Take your best guess. And even more important than taking your best guess is re-evaluate. Look back at your past guesses and be honest with yourself about how right you were and how wrong you were. Right? And you will improve over time, but you will only do so through experience. So try and fail and try again. Thank you for coming. [Applause]