Metrics That Don't Suck: A New Way To Measure Security Effectiveness

it's been a good day you guys been having a good day good be talkative people have been tarted quiet in some of these talks so I'm hoping for a bit more interaction here and I know I've made a few mistakes so you know that should be good for getting an interaction with the audience one of things I've learned today well too two lessons that I saw repeated through all the talks number one is always have a cat picture in your slides I fail that I have a bunny picture but no no cat picture and some of the folks watching on the internet Dan you know who you are you should have caught that one and when reviewing these

slides the other one is oh dear I have an optimistic message that makes me an idiot doesn't it how could I bring an audience like this an optimistic message but hey you know we'll see how we get on we'll see what kind of rocks you want to throw at my point about why there's some good news here because I want to talk about security metrics that don't suck now since we publish an abstract I've actually you know besides always generates lots of firsts for me I've got more feedback from talks done at b-sides well well after the show than any other show and I've had feedback on abstracts that we submitted here before the show

even started before the talk even came up a lot of people have gotten in touch with me about this idea that security metrics that don't suck so hopefully you folks are here because you two are tired of some of the problems in security metrics if it's several reference to that in talks already today on security metrics being a hard problem so just to start with the basics why our security metrics a hard problem well okay I think a lot of folks you're familiar but just so we set common ground rules right but the big problem in security of course is if you want to measure security you're measuring the absence of something happening and that's fundamentally

difficult what are you gonna do are you gonna publish to your executive board how many days it is since you were last on the cover of the Wall Street Journal this isn't a very good metric this isn't going to help you very much with justifying the spend you want to do right and you're going to be very hard to differentiate one technology from another based on how many days that you're not on the cover of The Wall Street Journal so measuring security is fundamentally difficult okay not expecting much controversy on that but unfortunately we as an industry figure this out quite a few years ago and we've been talking to ourselves about security metrics so quite a while now right and

the problem is a lot of people have tempted to come up with metrics and I think they found it quite challenging that is we start talking about what do we learn from secure Metrix wonder and the first lesson that I think we were that I want to emphasize is don't measure how fast your treadmills turning over all right what we find is those big pressure to measure security and what did people do well they started looking around saying okay I need to I need to be like those quality guys over in manufacturing or there's operations guys who have their availability metrics I need to do that for my security world okay well so what what can we do what can we count um the

first thing you find is anything that leaves a record anything any process you run how often do you run your change control process how often do you update your antivirus signatures how often do you do things and we'll measure that and an awful lot of schematics I'll give some examples in a second just end up being measures of busyness rather than business right and the problem with that is that's just a treadmill metric so so watch out for treadmill metrics they are everywhere the problem with them of course is the reason you wanted metrics you know if anybody saw the presentation three ago in here with the decision making loop for which you need metrics

if you're going to use a metric that measures your busyness how do you show next quarter or next year you're being successful well let's see you could go to the executives ensure that your met your friend was slowing down no that should be great right if we actually had less security work today wouldn't that be fantastic except that means you're telling the executive team that you're being lazy not good the alternative is the metric keeps going up oh look we're doing better and better because I've treadmill is faster and faster and faster this is also the road to hell right you don't want to do that so you have to watch out for these kinds of

things why well because you've got to think about the way that metrics involve behavior now I'm going to mess up the rhythm of my talk here because I have to just stand here in let you read this for one second maybe some of you remember the this particular deal but cartoon from a while ago but I'll just give 10 seconds to catch up on that I mean for once the pointy head boss is trying to do something sensible what he wants is some quality code and he imposes a metric program to figure out whether he can get high quality code and while he's saying I'm gonna write me a new minivan this afternoon you know always remember

those words if you're thinking about a securitymetrics program think about I'm gonna code me a new minivan this afternoon right the this to me is you know it's not just a good Dilbert joke it's a really profound lesson about metrics I'm a statistician by background and we think of these things and it's really really important to realize that people react to metrics anybody read freakanomics anybody good yeah a book entirely about the fact that in the real world people's behavior shift when you measure them there are plenty of psychological studies on how exactly this works and so we have to think if you have a bad metric you'll get bad behavior and that will get you a bad outcome and that's

what we're here to stop right all of us so we have to be very very careful in our security metrics programs not to encourage Wally to code himself a new mini man so as irv we have the problem called the hamster wheel of pain i think mister Jaquith and some of these folks are actually over at mini metric on rather than being hurt besides blue hiss on them but i know some folks have come come back over here but here's a recent study this was worked on to go and survey who is doing metrics and who's using which metrics so difficulty have metrics programs they've attempt to do anybody willing to admit I'm not gonna

call on you for details but who is actually trying to do metrics already yeah and you recognize in your metrics on on this list but you notice how an awful lot of them are really just measuring busyness right they're just measuring hey little incidents here little incidents there I mean so most of these are criticized as being bad treadmill metrics these fractions are the the fraction of companies using these so these are high numbers right but my favorite is where is it any intrusion successes really I mean seriously either they don't mean what we mean by intrusion or something really weird is going on or they getting lots and lots of zeros right well and and of course

that that's one of the easiest metrics to get how many attempts were there right that's that's way up there about intrusion attempts right now so you get vast numbers people are studying that and that's a classic hamster wheel of pain metric right how often do i alarms going off of course that's the one put your alarm vendor told you use right because it's really really juicy right anybody not seen when you plug in an intrusion system or like a billion attacks prevented oh come on I'm I'm really gonna go to my CFO and say hey if I value those breaches at a million bar and we just prevented three million of them come on right

hey anybody's here to talks ago with the costs of breach Gillis's talk right if you did that kind of math on the alarm rates coming off your IDs the value of your company has been consumed a thousand times over something is wrong with this this is a security metric that sucks like you have to measure something else because we all know stuff happens right so what do we want well I claim like I say it could could be debatable here but I claim we want to be like the guys over in operations everybody in operations particularly network operations has an intense focus on a single number its availability right its measured in 90 you have 99

percent uptime ninety nine point nine ninety-nine point nine nine and everybody in that organization worries about that number and nothing else and so what they have is a closed loop my CEO at red seals a nuclear physicist and he recognizes this as control theory like you're using nuclear power station you got to think think about a negative feedback loop you have to think about what process and then what control in the process and availability is a really really good measure those guys they have it easy over in network operations they can go to the CFO and say we are at two nines and if you spend this much money we'll get to three nines if you want

spend that much money you can get to four nines all right what a sweet thing to be able to do if we could get even close to that kind of clarity in our communication with the rest of the business we'd be far better off but I'm gonna go back one does anybody think these metrics make that point to the business I certainly do and I've gone through this list I don't see a single one that would help you if you think about what those guys and operations can do now we know the name of what we want we know it's not availability we know we have this problem that you know the phone never rings when there's an

increase in risk in the organization right the phone rings for availability problems right so again those guys in operations they have it easy we unfortunately have to measure risk we have to measure that things are not happening we hope a lot of a grand scale at least many many small things are but the really big ones the Black Swan events right people read Black Swan then the seem nickels till they're bright the whole thing about you know it's like as great as a Christmas turkey until that last day right brilliant insight about this sort of stuff so so we have to measure the thing that's insidiously going wrong and we know its name we know it's called

risk but it's quite hard to measure I wouldn't I wouldn't have been invited to speak here if we all thought this was an easily solved problem but let me try and show you some ways to make progress on that on the risk question now this is about measurements I know what I'm gonna keep keep to philosophy for a second I will eventually get to some specifics about actual calculations to do but still talking about why do all this right good measurements can tell stories and which story you want to tell depends on who's involved in the conversation so obvious members are folks like all of us right the team members in security teams and we've got our friend this is oh we

hope that's your friend then there's the conversation outwards to the CFO and then I've got a fourth participant I blacked it out here and any guesses who the fourth participant might be shareholders is good not the answer I was expecting and a good one but not the one I'm looking for no customers another good guess no it's a tricky one we'll come back I was expecting more people to say CEO but now I really don't think there's a meaningful conversation about these kinds of scores with a CEO even the operations guys can't get the CEO worked up about availability numbers right everybody thinks oh if I just go up the pyramid all my problems will be solved

it's just not true right press interesting angle actually I could do any talk on that one but no that was not my angle good one I wish life was so much easier to talk to vendors about risk me a happy guy because that's what I do all day okay so we understand that our conversations within the security between the team members in the CISO right we know what we want risk measures for we need to prioritize I've heard that several times in other people's talks who are not even talking about what I was here to talk about right we know we are overwhelmed right I have I have never met a security team that said

you know I'm really short of data anybody here want to put their hand up and say yes yes I'm short of data in security I know there are things we'd love to know but with most organizations I know a drowning in data and so we need internally within security we need decent measurements so we can prioritize we can understand whether to focus well two of my particular obsessions are should we worry about networks should we worry about endpoints all right there other about should I worry about full disk encryption should I worry about exfiltration right you need prioritization you need ways to measure this so this is one things we want and of course because we need a top ten list

right we're all used to put the products we surround ourselves with which I've heard criticized very well today generate phone books what do you do with a phone book of data how do you turn that in something meaningful we'll need measures and of course we also should be asking ourselves and I think anybody coming to a conference like this does ask themselves are we effective right that's another thing you can do if you design your metrics right you can actually use them for your own introspection your own team's introspection about yes but is this metric actually show being effective or not now of course there are outbound conversations too like as you talk out to the rest of the business this is so

out talking to the CFO where of course the first line is not are we being effective it's not introspection it's like how effective we're being right very very important to have measurements that you can use outbound that prove that thing we were worrying about a second ago you wanna be able to show reductions in risk and ideally show why all the money is being spent I've heard several comments on that during the day as well because of course you're always going back to ask for more money anybody who's not just doing b-sides we're also going over to RSA all those damn things cost a lot of money and we need a way to justify to back to the CFO that makes

sense this is going out on the internet so maybe I should be slightly careful about what I say here but I'm really really fond of that graphic on the right thing we do recognize this this is the Department of State's measurement approach project called I post that there are some other names for it as well and I love this because of what it says about the last two slides right the time the state under FISMA with continuous monitoring all this pressure they are the thought leaders in the government on this particular point of measure measure measure they do a great job they really do a lot of credit to them they've done a fantastic job

publishing letter grades for individual embassies because they have no control of the individual embassies they have to essentially figure out because each embassy it's basically some fiefdom they have people to measure how it's doing and so they do they actually do an excellent job of some basic parts of measurement and they can measure and give out letter grades to motivate the various embassy owners the ambassador's to spend more and pay more attention to security and they have indeed had a risk reduction this is the chart they like to use to prove it and I do let me be clear they have indeed had a risk but if you can read there up here it actually says 89 percent reduction in

red and 90% in blue for the domestic and foreign sites well you wouldn't think the domestic and foreign sites were very correlated with each other would you shouldn't they be like independent operations I was just making the point they're all independent any statistically savvy graph readers in here if you look at these two graphs the lumps and bumps you see that spike right there and that one right there and this one right here this one right here and the way the bumps through here look really awful look like that what the heck is going on what's happening is that changing the grade levels so a large amount of this is actually shifts in the question now again the deposit

has done great work on this they were doing this genuinely that they were working with the other season trying to get buy-in right so another thing about the psychology of metrics I'm going way off my intended script here but on a good one slide 9 so I can afford this they actually thought very hard about how to get the embassies to buy in because if you've tried this if you've actually tried to measure the business and then report to them the first thing they do is they Brussels they don't hear from you guys about what a crappy job they're doing because they all know they're not paying that much attention right they all know they're not as

focused on security as you are so just going in and explaining to them they get bad grades isn't terribly interesting so the private state thought hard about this problem what they did was they they got the embassies to participate in assessing the grade through going to be used before they started publishing them and then as they published them they tuned up the score because some people so it's not fair that I got a D grade for this thing over here that was really unfair and so they would tweak the scores a bit and they were doing an excellent job of improving the metric as it went along but if I look at those scores I can see that more than I can

see the reduction in risk but that's me being picky the fact is they did an excellent job with a metrics program they did an excellent job working with the community to get a real change in behavior and they made some grand claims out where it's about the degree of risk reduction that I find a little stretched but you know if I get you know shuffled into a black car on the way out of here that's because the Dom the state heard always live over the internet they're gonna come get me now but I really think they've done an excellent job with that caveat so that whole thing about asking the CFO for more money well we know what the CFO once CFO once

an ROI here we are at Eli speak they say I spend I want to save Bella why you ever been able to do that you'll be able to show a CFO that sort of thing what do we do we tell them the same old FUD stories look what happened to Sony do you want that to happen to you anybody not tired of this yet it's a stupid game we have to play it with the CFO they try and ask us for our financial justifications we fail to produce them we produce our flood stories instead ho-hum they know it's gonna happen we know it's gonna happen it's tedious right we need to do better so let's get creative let's look around

who else could we call on I heard several ideas earlier shareholders customers etc my claim is um have you thought about your insurance agent now other insurance professionals in the room good good good the the clipart was a little bit rude not an actual an agent for anyone on the internet real ones are much nicer but insurance agents they're interesting people and you know what you get right down to it they know more about risk than we do we know an awful lot more about security the S word but they understand the R word a whole lot better right we like to claim that our domain is so incredibly complicated but you know they've seen it all before security you

know security professionals could learn a lot from engaging with insurance agents and all details a little bit more although I was talking about this earlier today and I was reminded of a point of a book by Doug Hubbard who writes to within this community about risk but he's also written a book not aimed at security at all but aimed out to a general audience about you can measure anything hey it's called how to measure anything it's very good book give it a read in there he's talking about how to measure intangible things in business like customer satisfaction and things like that's right things that are very very hard to measure you know all the stuff that doesn't have an ROI

right and we should care about this because we have this problem right and being not agreed that we have an ROI problem security I mean Gillis's talk is fantastic on just how much money can blow up but that's just the Sony fuzz story how often does it happen will this spend cause that reduction in risk we generally can't do that right so we need to do better and this this is a guy who can help us well let me get back to Hubbard's book for a second how to measure anything he's trying to explain to the audience reduce book who are supposed to be general people in business about how people resist the idea that something could be measured

particularly domain experts need the main experts in the room yeah wanted to they're very resistant to admitting that their stuff can be measured and his canonical example of this one of the leading things in the book is an IT security professional who wanted to tell him at one of his weekend courses where he teaches people had to do confidence in to automation a photo to get the details of that he tries to educate this guy on look you do know more about this and and I can't afford to repeat the whole thesis of the book but it's good and this guy said no securities fight you complicate you don't understand all the stuff we have to worry about in IT

security it's so complicated there's so many moving parts the evolving threat landscape yada yada yada yeah like I said how about ask him where do you work oh I'm in the IT shop at a large insurance firm dude all those people that you are doing that security for all those insurance officials what do you think they do well they there's two main risk for things that are really hard I mean heck they insure people really seriously you want to tell me your computers are more complicated than those people when we write insurance policies and we don't have all the facts and they change their behavior as we write the policy and as we adjust the

claims and seriously you really want to tell us an insurance professional that oh no my worlds too complicated so I think we should take these people seriously and they are increasingly taking us seriously right data breach insurance is more commonly available I'm I'm tracking this out adding insurance energy is it anybody comfortable with the data breach insurance they have today anybody know whether they've got any yeah a couple of folks so good I mean I am seeing it come out there and from the figures I can see from the insurance side they're trying to figure out which policies they can sells there they're some of the early evidence is they can make some good money here they can see a

real risk opportunity here because they what they want is a risk transfer market they want to find people who want to give away risk will pay to do so you guys match that description at all right and there's a very good reason to transfer risk to these people right I can't go all the way off into history of insurance and ships and Scottish Widows and so on but there's some very important lessons from all that world for us it's about transfer it's about pooling risk it's about when you don't know what's going to be if you don't know your ship's gonna sink insurance is a really good way for all of a ship owners to get together and pool that so

that we can all withstand the horrible things that happen to us but there's another factor there's a second bullet got here a heading tires slides on this that I had to cut for time there in the appendix but the insurance agents represent companies that don't have the transparency problem we all have I've heard a couple of references back to Gil's historian where somebody asked how can we get more transparency how can we understand more about the breaches that are happened to other people hard problem insurance companies don't have that problem insurance companies write policies and they get the best data possible there you get to say for which incidents did they have to pay out data they pay out

money because the data loss so when did they have to pay out money to whom did they have to pay out money and what were the behaviors of the people to whom they had to pay out that money oh this is interesting because they see all these breaches not just yours if you compare it to car insurance right wheel drive car most folks here drive cars your your regular person's lifetime car accident expectations about one in very raw numbers so you can expect about one significant car accident driver I've had mine I hope I don't have another one but that means I have very little information on whether you know mid window rear brake lights impact security

or impact crash risk or not right I can't get that data make sure it's company can they can study what's going on they can look at all the various actions they can understand hey the cars that had anti-lock brakes were a little bit safer about this much because they can see all the incidence and you can't do that I can't do that I visited an awful lot of companies and I can see a lot through red seal I can see a lot of their posture I can understand a lot about who runs a good operation and who runs a bad operation that's part of what I do for a living but I still don't get the

incident data because the companies won't tell me and they won't tell you guys either and I've heard many people complaining today about the fact that we can't get this data the insurance companies won't give it to you either but at least they got it so we can work with the insurance companies they can learn what works and now if you haven't read ahead already then all the studies say people always read your slide before you even get there the argument with the CFO is the CFO might finally have that light bulb moment about what this is about good security is that which reduces my insurance premium you might not have to go straight to a one to one right

too picky about that but at least imagine being able to talk to the CFO with a straight face and say I will save you money let me save you money out of here and I may need to spend a little more over there we know that's gonna be a little bit tricky but at least to be able to get to that kind of conversation with the CFO would be an excellent improvement for us as an industry so my suggestion my positive messages is surround the CFO work with your insurance agent it's not asking questions about data breach insurance it doesn't cover everything it doesn't cover everything we're worried about at this conference but it's good stuff and

there are very good reasons to think that this might help you justify what else you want to do because particularly imagine if you could actually negotiate the other thing because it's a nascent industry there's a lot of haggling right this is not like buying car insurance you know if you're in San all that's all very entertaining but premiums are more or less well established people know what anti-lock brakes are worth be insurance professionals they know the risk reduction from antioch brakes but they don't really know this stuff yet so if you could demonstrate that you have superior hygiene to the next guy you can actually negotiate a meaningful discount and this is a very healthy conversation

be having this is a good good conversation with the CFO and you know what if we did this just right if we if we can actually work with those insurance professionals and get a sense of these technologies working those don't I've heard several claims today about that technology over there sucks that technology over there doesn't work we actually have ways to quantify that here right not just PCI says I must do it but no my insurance agent says no I must do it now that's different right that's a significant difference in meaning between those two anyway too much philosophy some people starting to nod off because I'm not talking about metrics enough so time to start talking

about metrics anybody actually old enough to remember this yeah yeah okay good good good we just want to make sure nobody fall asleep so let's talk about ways to measure meaningful posture so just so we get the pieces out on the table and get every clearer about what we're gonna talk about every it's got some assets you need to protect right everybody's got some it could be personally identifying information could be intellectual property somebody's really mission critical some that people will die if you mishandle it or a bad thing happens to it and those folks tend to be more focused on these kinds of problems but that doesn't mean they have massively superior environments I find I

study these for a living and yeah I see some pretty wacky mission critical you know life critical environments out there we know there are vulnerabilities bad guys exploiting so we scare now again I've heard some good stuff today on the weaknesses of vulnerability mapping but at least tracking the known knowns is an accepted practice it's not everything you know here at a conference like this because we know that's not everything but it's a damn good data source at least of the known knowns that we should be taking care of and we have countermeasures like firewalls at the most basic level and moving up to fancier fancier levels so you have to think about how these things

combine and I don't have a cat picture bed you have a bunny picture so 1/2 1/2 points I hope so the trick is to attack the trick is to attack the environment we've heard conversation earlier today on on pen testers well you want to know your defensive posture if you could measure your defensive posture that's half of the risk equation solved it's not the whole thing you'd also need incidence data that thing I'm saying insurers can get and if we could come to the table with posture awareness because we understand the technical gear a whole lot better than maybe they aren't really not a firewall is it's not their job they understand risk and they understand

and you know actuarial tables and if we can show them our defensive posture and they can measure the incidence data then we could actually pull off an interesting dance here so our job is established defensive posture means we got to find the weak points so we want to attack just like the rabbit bunny and we want to measure ease of compromise and by the way by the way there are some stands along the way to help and standards of good thing so we'll try to use those so what do I mean by attack well if you can model environment what you're worried about is the bad guy getting to the bug everybody ask me what's the black widow for I haven't had

a better way to show a bug okay so the bad guys going to try and reach the bag and if you can do so that's a problem okay I know I'm doing baby talk for you guys but bear with me for a second I'll get something a bit more quantified you put in some countermeasure something to block it to make a low-risk situation the bad guy being not stupid that's the point we'll find a way to pivot around and so now you've got this very complicated chess game where if they can find any path in you've got a problem you have to find every path to be able to stop them they only need one an

unfair game because you need so it's a tough chess game right it's harder on you than it is on the attacker so you need some way to measure all this stuff all right let me do an example quickly this is a real corporate organization there's an internet cloud up over there on the left in red they've got a DMZ out here because they're a good organization it's fairly large a few so each one of these little gray things is a separate sub that each one of the denser blue dots is a router so that's a fairly large DMZ right that's a quite big environment that's actually almost a company size thing on its own so this is a pretty

major presence and then this is actually the rest of the corporate site back here so a fairly large corporation this is actually one of their sites but what they want to do is attack this they want to do a sample a tempura chain to understand their environment okay so how do you do that well you can do some calculations to say what axis is there and then which vulnerabilities could you reach down these paths so you're looking at the firewalls the countermeasures the other things that can block attacks and you look at okay where can you get from the internet and you find an answer like this and this is good this means they built the DMZ for a

reason right we know why we build DMZ so we want to push all right goofy stuff out to the edge there these guys did a good job of that so yes they had a working DMZ many times I'll come into a company and all the red goes that way all right but not here this is actually fairly well done but in that set or any of these important what would an attacker do well one of these is much more serious than all the rest it's a little hard to see here but one of those attack points where you come in from the internet you hit a machine over here there's now a second attack vector from

there going all the way over here and from there you can fan out and do more else whatever you want all right so that's a chain simulation that's the methodology I want to talk about now of course I'm doing this at high scale with some software but you can use this in smaller environments this is not vendor pitch I'm trying to talk about a methodology here of how you could do this and I'm gonna give away some of how we do this at least mechanically in the calculations so the summary result then after you do an attack is you start this accurate picture we draw you start on the outside you've got this attack surface from there we can compromise

this much your organization and the good news if you've got a picture like this is that much the organization is safe many organizations to start out we don't have a very big green beach ball but some do some do better than us and the objective over times to get the stuff in it okay that is a conversation about operational security about how you could prioritize within the security team I haven't said anything about metrics yet okay so I just wanted that as ground rules and now I want to talk about ok now measuring this happy I can extract a measurement from doing all this kind of work and this is a fairly complicated slide but I

did promise some some beef on how to do some calculations you know my company Red Seal we've taken over 40 million dollars worth of funding this is 40 million dollars worth of math on this slide and it's really simple to do right it's a little hard to scale so I do actually have a company for a reason but if you wanted to use a methodology like this if you wanted to think about calculations you can steal my ideas I've published they're not uniquely mine but you can steal my company's ideas right so publishing here a mechanical way to measure the significance of an attack now I have to be very careful I know there are some folks in here yes you can

have a copy these slides I see people shooting pictures of it so I want to be very very careful as I know some people in here as soon as they have that word up there I posted some the other day about you know today's definition of risk using the word risk in a slide in front of an audience like this but I want to tread very carefully here and there is a jump between the first line in the second line so I'm gonna emphasize that for the folks who track the details you're not worried about this philosophy so much bear with me for a second so all statisticians will agree with this top line right the risk is

always you know it's the expected value calculation it's the probability the bad thing happening times the damage you take if that bad thing happens right that's straightforward everybody agrees with that but then how exactly you measure that is where all the art comes in right this there is no debate about the formula for risk it's some small semantic finery of some variations but the trick is how do you proxy these values if you haven't got them right so the actual dollar value of that web server over there not not a sheet metal and silicon but value to the business we all know we can't do that right so we can't directly get value we have to use

proxies for value likewise I already I already conceded probability of an incident okay if you know you have this vulnerability of you what is the probability of a bad guy coming tomorrow and taking you down with that anybody think they know that anybody think for a vulnerability you can actually assess the probability of exploit and if some people do study the sort of stuff so I might find some folks nobody's quite willing to extend that claim okay so we have to use a proxy again so the proxy I'm going to suggest is we refer to this as exposure this is where I'm starting to give away company secrets so what we need to do is we need a proxy for value

okay we use a 1 to 100 scale because we find nobody can do dollar value nobody can say that machine is worth a million dollars there are people who can say downtime of this sort on that machine is worth so many transaction volume so many billion dollars but that still don't get you into reputation risk all the stuff we you know that we all as an industry think about so we put in a proxy sometimes referred to as a jobsworth yeah 100 scale you set machines to 100 if they're worth more than your job is all right that's lots of jobs worth that's a unit value of 100 on the scale so we allocate values to hosts based on the application

values that we see there on the relative scale with a default that the databases tend to be the juicy stuff good basic rule to start with you can tune from there to your heart's content and anybody here can do it and then we use this CBS s standard the common vulnerability scoring system right everybody is familiar with CBS through escape through the NVD and the common vulnerability scoring system it's a framework not all the values in every single vulnerability attune to the umpteenth degree but you do have the right to change it yourself you don't have to agree even have a slot in there we can tune it yourself called the environmental metric so but just keep

life simple and then do a simple example of how to use this so if you allocate value to hosts and you got a CBS s score and let's at least take this as a proxy of how bad the vulnerability is right now I clearly can't get a time base components what I'm saying the insurance guys could come help us all with so I don't have that time-based component but I can use this as a measure you know that's a doozy a CBS s 9 on your internet facing environment would suck okay so let's call that an exposure of 0.9 and that means I get a risk proxy score here of 68 I go down to the next

machine and he's good he's a he's a jobsworth right there's a value 100 jobs worth but he's only got a CBS s6 now in his case what we're gonna use is the point six from the CBS s I'm going to multiply it by the strength of the ease of attack to get in here so if this guy is easily exposed then this guy is correspondingly easy to expose so it's a chain of attack so if he's very very simple math it's just simply saying start with an attack intensity of 1 and then multiply by normalized CBS s value so you get an exposure of 0.9 here you multiply that point 9 for the first guy

by the point 6 on the second guy and this guy's point 9 by 0.65 point 3 and then you can multiply each one of these by the value of the asset and you get these scores like 3 here 54 here and 68 there tends to be higher risk towards the outside but you'll actually find that the internal assets they're reducing the crown jewels will hike way up when you do this because you know crown jewels in middle of the castle if those can be stolen that's bad news that's what that score picks up there's another calculation on here that I didn't mention that is blue thing downstream risk all that's happening there you can see it's just

adding up everybody down in the graph right the idea idea there is just a quick quick point if you measure risk and you always figure out okay here's my castle here's the crown jewels in middle of castle oh look the crown jewels are at risk that tells you have a problem doesn't tell you what to do we've generally found that in most environments at downstream risk measures are better we figure out what to do downstream we'll notice things like a forgotten ftp server or not not thinking of any hotlines in particular an HR web server where some SQL injection could allow somebody to inject something a forgotten resource like that would have high downstream risks because it's not

such a valuable asset but you can use it as a path to get into other things right so that's what the blue calculation picks up all right so I promised some mathematics that's a method that's an example of a way to do this kind of calculation and as I'm giving away you know how we do this it's a little difficult to scale up to 100,000 houses so we do come along with software yes I do make software I make software to do stuff like that and take in vulnerability data about your network your scanner all that stuff you're all those phone books that you're buried in I'll crank it through this model using the nvd as a data source we package

design I think all the TRL I'm not going to talk about this inferred stuff over here we create that through a calculation and we come up with a risk map looks kind of like that it's it shows red things where you've got problems and then you can pivot it and say okay okay I know that the crown jewels can get broken into but now what do I do you can pivot it and say well where do I have height downstream roots what are the most important defects in the castle wall will allow the bad guys in instead of focusing on the crown jewels all the time okay so regardless of whether you like my software for doing this or not

the point of the methodology here is this approach can be rolled up to give you meaningful scores and we are finding this is very effective for people this is a real shot of how the product packages this up it's unfortunately not real data I do have tiny samples of some real data but of course it's very hard to me to disclose some of that for some of the reasons you know very well but what we're trying to measure here is that overall risk score now of course I know some people in the audience know more than enough about risk to know that what I strictly have there is a proxy for risk because I had to proxy the value

score and I had to proxy the probability of exploit yes I confess that is a overall proxy risk score if you want to be picky throw things at me later if you even objected that characterization but at least it's trying to measure how easily a bad guy could get in that is a measure you can at least talk within a security team about but you can also talk outside about ah we spent all this money and a drop like that happen of course if you spend all that money and that happen that's not so good all right we can then break that down into we can separate it out into measurements right if you're tracking the mathematics of what it did

it's actually relatively decomposable score you can decompose it into okay well where do I have issues on the attack surface side there in the network and then when do I have those in the host side I mentioned that distinction early all right that usually reflects back to silos of work inside organisations so you have to know you know which phone number to call when you know you've got a problem right so you have field to break that down and you can never do anything about any of this without talking compliance that's why it's a compliance but at the end that's your PCI status or any other standards you want to use for whether or not you're following network security

standards PCI section one if you're really deep into that stuff you can also use these trend lines I think I already mentioned this about whether or not the investments are working and to focus on where you need to improve right so all of that in one page and again my point is to try and show you a methodology I don't care if you do it my way what I do care about is that you attempt to measure for an objective that get hits towards the right outcomes that is concluding I claim defensive posture can be measured you can vary it to other messes of the ones I've used but it is critical to start measuring this to not

be stuck on the hamster wheel of pain where we're just measuring that we're being busy we need to measure things that can drive to better outcomes and if we can measure posture that will improve our posture instead of just making the hamster wheel run faster and faster and faster I can claim real instances where it does help the CFO's light bulb to get to go on and ultimate objective we can sleep better these are my grand claims arguments it's it's too warm in here I see people nodding yes

sure the question is but can you use this operationally not just don't let me put words in your mouth but but but you know we often call this proactive right proactive security intelligence is the blurb for this then the point of course you're doing that ahead of the attack you're behaving more like a fire marshal than a fire alarm or the idea of noticing here you've got a whole bunch of burnable material next to the lift shaft there don't do that as opposed to burglar alarms the Gulf when you know firearms it off when the buildings on fire so if you do have a firearm going off what uses this other stuff is that a

fair summary of the question yes so operationally what we find is today this is often a swivel chair right there are quite a few socks you can go into where you can see something like what I do the proactive stuff on one screen and then so the screen over you've got the events stuff going on the trick is to do the proactive stuff you have to map the environment really really well right Sun Tzu's all about you know know the terrain and know yourself before you worry about the enemy right so the space that I'm in that kind of vendors who do this we're all focused on that on the sunsoo point of know the terrain and of

course is that is knowing the terrain useful when you know their cavalry F charging through over here well yeah it is kind of useful but I'll be very straightforward today it's very much a school chair kind of thing so people do pickup live events and say I could a is attacking beep well is that gonna work is there a countermeasure in the way if they get to be where can they get downstream from B these are good questions to ask you you're mapping and proactive vendor rather than your alarm dinner cuz the alarm vendor is just trying to deal with the firehose of junk coming at them right so the kinds of measurements I don't think the risk

metrics have proven so effective for operational response but the preconditions for them have it's one of the funny things there to be able to get these risk metrics a Kaffir actually standing on a reasonably clean operational model and the mapped fundamentally the map of the terrain is the thing that helps in fact if I can even run with that foot foot for one second I have a suspicion but this is absolutely unproven right most this stuff up here is solid demonstrable stuff today but I have a suspicion if we really get straight with those insurance guys about what they want to see I'm not sure they ultimately need a measure of how easily somebody could break in and

attack what they could really use is knowing whether you know because if you're the kind of organization who can do these measurements that I'm talking about if you have the data quality almost regardless of what the measurements say you're on top of your game you've measured the environment Eve worked with the guys over network operations you actually could map your environment you've actually got valen scanning and you can prove that it's got good coverage of the environment right what one of the routine findings we have when we only help people do this is as you saw if I go all the way back to my inputs right you need to gather countermeasure stuff network stuff

firewall stuff like that and vulnerability data well those are two different data silence they've been operating completely in isolation in most organizations until people like us show up okay you take these two data silos you pour the two of them together what do you find they don't map together at all but yeah I do this hand gesture a lot right there's a whole bunch of scan data that doesn't fall in the network and it's a whole bunch of network that cost an awful lot of money that seems to have no hosts in it what's that about right so to even ask the questions I'm talking about I've stepped through a couple of earlier levels of maturity about just

even mapping the environment the military types do you call it you know situational awareness force accounting terrain mapping exactly those principles play out very well here if you can do those principles well then during incident response you will be a far better incident responder ultimately it's not the metrics that do that it's the preconditions of those metrics if you can get that data and if you can understand it then you can respond to incidents better yes yeah yeah if you block the light then I can see

well how do I find out how effective these things are Oh

wait yes so this is why I do think it's good to go back and look at what the insurance companies are doing and what they're not doing because you're right they do not yet do everything but what they are doing is useful right the data breach insurance sometimes called data privacy insurance it is certainly not covering everything we worry about but I don't think it's gonna get better unless we engage with those guys they're not worried about your EULA when you broke the shrink-wrap then worried holistically about your business right they already insure your business against workman's comp issues but some guy falling down on the job and breaking their leg right and where were the

EULA's you know whose liabilities insurance cover Altima he doesn't care they care about what kind of business you're in and what's the probability of somebody suffering a serious injury in that in that business and they have ways of measuring that they have ways of figuring out your premiums based on whether or not there's a whole lot of heavy lifting in your company right so the insurance companies if you what if you want to told me if my message here makes any sense at all if you want to talk to those insurance companies they're gonna come at your holistic Li they're not gonna worry about the minutiae of what exactly you've agreed to between you and the various other

vendors they're just gonna say there are costs here for breach occurs notification costs paying for all those you know credit protection services all that kind of stuff we can insure you against those costs but to to buy this insurance it's already true to buy this insurance you have to pass a very very basic qualification you can't just go out on the street and buy this stuff you have to at least demonstrate that you can fog a mirror in security terms right because you know they're not crazy you guys then they're not in the charity business right we're all clear about that right so they're not they're not coming on to solve your problem they're coming on to

make a profit so you will have to demonstrate that you can fog a mirror but they do not care about the finer points they only care at a high level about whether you're in a high-risk business and are you able to do meaningful security things so I hope I'm not misinterpreting your question but but if we plow into our bodies and a liability or that other software vendors liability or no that's a lawyer question that's not an insurance question feel free to throw things you still disagree yes

hmm

yeah that's an excellent question that is a good catch I think this slide even has the worst word on yes it has the worst possible word and I thought it did sorry this is a freshly built slide and you you are dead right when I go back and edit this sorry Bob if Bob is watching that word right there yeah that's where a lot of people start but you're dead right that's that's not all there is to this right I just find in most organizations if we try and model their environment if I ask the security professionals what are you worried about you know if I give them a tool to mark threat sources they go crazy they go

like a three-year-old with a rubber stamp everything is a threat source and the problem that you don't get any quality of metric metric out of that you have to have some degree of differentiation so you need to be able to improve resolution so my practical methodology for people is start with something simple and basic that everybody including the CFO can agree let's start with just your internet feed that does not mean that's the bill an end of the question you are dead right that is a fault on this slide because it implies this is purely about a prior 2d perimeter ization word world what we call it a perimeter eyes world you know so yes you can use this

methodology and substitute your own you know monster onto the bed into that Red Cloud and you can use the same calculation I'm pointing out how the calculations done in hopes it's useful to you but of course it depends critic the results you get do depend critically on how you set that going back to where I was being rude about the Department of State right I do love their trip the trend lines just because there's so many stories in them but of course at that point I made that you know this drop here appears to be a changing the rules not a change in reality that happens to of course if you took a model like mine

and you were only worried about internet then you say okay now I want to do Wireless I still worry that Wireless is too easy to compromise so I'm going to go stamp stamp stamp stamp stamp and go say all my wireless egress points scare me you can absolutely do that you'll get the reverse once you get one of these things right you can absolutely take whatever paranoid stance you want I'm just saying that my experience as I go security team to security team is it's a very bad idea to ask a security team to list the things they're worried about because it's everything right so you have to start somewhere so I hope I'm not being at all evasive on that it is

complicated question I do recommend narrow focus to start with even if it makes it sound like you're not being paranoid enough yes

that very good question yeah what happens if you run this engine backwards and try and figure out what it could tell you if instead of imposing from the from the get-go which of the value biases try and figure out from say the stance in the graph which the the juicy points are and I'll tell you what yes and it doesn't tend to find mission critical assets mission critical assets actually tend to have relatively limited attack service what it finds more than anything else is your antivirus servers or your network operation center the true nerve centers right the things that can connect anywhere so that a von in that spot is so much more devastating right you know you need every place in

your organization to be able to reach your antivirus server right well okay but most people end up writing this both ways to say the antivirus server can also go anywhere which means a vulnerability there is game over right and so I yes yes other templates of graph analytics like that I did not find business value of assets what I found was people not thinking hard it hard enough about zone defense which by the way is another point earlier about what if the zones gonna break down PCI has some great principles in it again I don't know if that's a controversial opinion around here but I do think PCI section one has a very good zone

definition of you've got to keep the place where you've got my money right there's credit-card companies you've got my money in your network it needs to be in a box that needs to be DMZ separating that box in the outside world very very simple but that is a good design of defense in depth I do think shocking though this may seem some movements in regulation are helping us improve security because they're forcing us to finally spend the money for this defense in depth that we've been talking about for decades in this game but that will be a start of a whole new talk so I should cut that one off okay

yeah yeah no exactly right that's some of the segmentation thinking and the virtualization groups they've got going on and doing some good work on that on that right now so yes segmentation is key I'll be doing a whole talk at ions on that and it's well beyond my topic here that was only supposed to be giving away how to do metrics maybe one more a two-minute warning sure so everything I've said is vaguely right is the talk to so yes yes and that is a very sensible inference on the features you claim is you know anytime you see two graphs as tightly correlated as that pair that it looks like double counting I think I think

it's a wise comment on the graph I have both stared at this graph the report behind it I don't know something about the people who made the data and I actually don't think that's true in this case I think that's a smart observation but there's of course there's the data and then there's a standard used to measure the later I really do think in this one it's the the fact that they were shifting the standards plus they're not that tightly quarantine you account for the way the scale is built here some of the falls here don't quite map out down here quite as clearly so that that's a very good suspicion about the data but the way embassies work they

really are massively isolated from each other and the work that went on here to make sure if we agreed the yardstick was fair actually was a good way to address that problem they'd actually done pretty good data hygiene to say the stuff from the embassy in Bahrain is only the stuff in Bahrain right so they actually had reasonably good controls that but but that's a very good paranoia looking at the picture yes

and they'll be highly highly Cora yeah no I do fundamentally it made me suspicion the first place was exactly doing up that visual mental map that if you scale them together they're far too correlated

they haven't said what they're doing at the moment you can barely read it here but that scale then there is actually this is all data through 2009 and if you read the reports these things come out here you can see that's only when they started right so any metrics program you know this isn't my advice that's a book by Doug Dexter as a nice comment about this right you know baseline is gonna be grim right if you measure anything you're not gonna publish the results of the first three months because you'll be learning how to do it right it's gonna be a mess like this it's gonna be falling all over the place going up

going down looking like the grass I have from from my product that sort of stuff goes on and you need a way to say okay kachunk it's now this you know we've now baseline we've gotten to a stable set of standards the the thing I'm harping about here is you know they pop they publish date here from before they'd really stabilize the standards so that they don't revise most fast anymore there and you know honestly I don't think they're cheating I just think there's a defect that the paranoid can perceive in this data I really do think they've actually improved the security the State Department by measuring and frankly the name and shame that's been

quite effective for them

yep

that is exactly right yes and the CV assess standard that I've been talking about that that particularly has a knob in there you can tune that's you're supposed to use for exactly this purpose I've yet to find a single Enterprise doing so but there are any folks in government who do precisely that that's exactly what you're picking up on they do indeed we harvest them from the NVD we republish the matter I user basically picked that up without effort but most people are so overwhelmed that they're not actually in the business of tweeting and this is the CBS s environmental scored themselves so they just want the automatic stuff but you're dead right the Department state does have people

who are doing that and that will cause tweaks the number so of course to Microsoft patched Tuesdays all right anything we do this you will get correlation of Chrome oh dear it was Tuesday right that absolutely happens I think that's my time out thank you so much [Applause]