Walt Williams Security BSides Boston 2013 - Realistic Affordable & Quantitative Risk Management

and i actually have a book coming out with crc press sometime probably next year on security for service oriented architectures so um i'm definitely pimping the vote moving on uh a quick thank you to so many individuals who have been responsible for my success within the profession i want to do a brief shout out to dr mike lloyd without whom this presentation did not exist as well as the folks at fair the open group um everybody in the side society for information security risk catalysts which is an organization that i highly recommend you consider joining and of course the good folks without whom i would not be a security professional today and the information um

i always forget what issa stands for to be honest with you it's a membership-based organization we meet six times a year we promote security education for security professionals very very worthwhile organization it's something i've been part of the past 15 years um has helped shape my grade i highly encourage if you're not a member to continue forward so let's start out with what is information security risk and how can we measure information security risk is a subset of business risk it is um what could happen to business data what could happen to business information and obviously as you manage it how do you prevent those things from happening the trouble is that there are many many

different definitions of exactly what information security risk is and i want to give you a brief overview of that before we get into exactly how you measure it so let's start out with the classic definition this is the definition that comes with out of this uh that this is the probability of an event happening multiplied by the impact of that same event and the trouble with this philosophy of highly risk that would allow you to say that the risk of a low impact high frequency event spam is exactly the same risk as a high impact low frequency event a bond going on i severely doubt that anybody in this room would classify spam as having the

same risk your organization as a bomb going on and certainly you're not going to put the same level of controls behind that so the problem with this particular definition is it doesn't help you make good sound business decisions so we're going to take it we're going to throw it out along with so much else that comes from this so what is risk there risk is the potential that a given threat will exploit a vulnerability or group on a group of assets and pose harm to an organization that definition comes from iso 27 obama five it implies a metric and that metric is harm so to understand and measure risk you have to understand and measure threat

vulnerability assets and impact your measure so i'm going to start out with how do you assess risk are there a variety of different methodologies of going about doing this all of which have some pros and cons and i'm not here to pimp any particular risk assessment methodology i am using one in particular but just because that's what i'm using doesn't mean that you can't use the analytical techniques i'm going to talk about in part of any of these others i uh octave is very customizable you can use this methodology inside of octave you can do the same with iso 275 risk it is a great analysis tool especially if you're using cobit stay the hell away from this because they're

going to force you to use that classic definition of risk which we just drowned and i would recommend that you stay away from para as well because it only looks at risk of being attacked and that's not exactly a comprehensive look at information security risk the risk management cookbook is a very very nice analytical methodology for doing analysis of risk that came from a partnership of the open group and the folks who did the factor analysis of information security risk fair they have published this this is freely available it's a nice synthesis of iso 27.5 which you now don't have to pay for and the methodology behind the factory analysis of integration and the best thing is

that the open group modified fare to make it usable that's what i'm going to build off of it gives you both a process and a means to measure risk so in order to identify risk define your scope identify your assets identify the threats identified controls vulnerabilities consequences let's go into it a little bit more in detail so with assets your cfo is going to have one definition of an asset your chief executive officer is going to have another definition of an asset your um your id management doesn't have another definition of an asset your um your accountant is going to have different definitions and they are all right there's no reason to use one set

of definitions versus intensity of definition capture everything that they all have to say about the assets to organization the most important asset likely is your data after all you are in the information security profession information is likely your most important asset to you but it's not necessarily the case to others in the organization everybody is right here aggregate and when you're starting to put together what value is volume is more than just money sometimes the value of something is based upon your reputation or fines that will be incurred or the cost of the incident response of the loss of productivity when you're sitting down what the ass the value of an asset is taking into

account all of these different uh factors if you're a public company what's the impact your investors if you're a private company it still may be what's the impact your investors after all private companies have investors to the coal venture capitalists many of these values if you don't have precise numbers can be estimated if if you don't have precise numbers using a monte carlo situation we'll get more into that later on this gets you a value towards harm if you go no further than identifying what the value of these assets are you've got a long way thread many different kinds of threats have the same impact on your organization so what you want to do here is protect

against the impact not protect against the threat if you plan around the impact your protecting your organization at the same time against a tsunami in a bomb and an earthquake a tornado and a hurricane if you're prevented against the threats then you're going to have to worry about oh my god how do we protect against the hurricane versus oh my god how do i act against a bomb protect against the impact not the threat now when you start to look at what are my threats so that i know how to protect against them it pays to work with categories that other people have already thought out so the methodology i'm presenting to you is based upon the baseline threat

category which comes out of the financial industry so this is going to be a slant within that you'll see categories of internal fraud versus categories of external fraud which may or may not have relevance to your organization as we dive down into them but it's a very very useful set of predetermined categories that you can build off of it's a starting place and if you're curious to know what the details of are these threat categories here they are the new color i know by the way they're in the spreadsheet controls you have to know and understand what controls you have in place more importantly you have to know why you have those controls in place and yes you

should make some determination of how effective those controls are so how do you get this knowledge you ask you order to test and there'll be other presentations that get into well how do you go about auditing some of these things how do you go about testing some of these things testing penetration testing auditing your auditor is your best friend control categories again when you're doing analysis it helps to divide these things into different categories i like the iso 27.00 two category ones other people like the copic categories which i like most about the iso 272 i'm lazy somebody's already taken iso 27002 and knocked them against those bits so i've got this nice matrix of thread

versus control and we're going to leverage that as we go forward vulnerabilities you'll find so much literature in the security space about vulnerabilities it's really just a method through which a threat can act on an asset or you can think of it in terms of a gap in control now you can only protect what you know which is why you don't protect assets against vulnerabilities you protect assets against threats impact harm this is what you really care about now not all impacts compromise the entire value of an asset it's possible to have an impact against control that does not impact confidentiality if somebody steals the encrypted database it's no longer in your control they've got a

copy of the encrypted database they can launch out-of-band attacks against that database but for the moment at least because it's encrypted it is still confidential until they break the encryption you've just lost control different levels of impact have well different consequences the value of the protection however should always be less than the value of the asset so if you're doing aes 256 level encryption against um a listing of lunch menus you've probably gone too far so your ceo says x your cfo says why we've already gone through this they're all correct moving towards analysis so where multiple assets are impacted aggregate results establish a scale and the scale should be proportional to the impact it should be proportional to

frequency capability etcetera the idea is you're establishing a model models reflect reality models are never reality models can always be adjusted so what i'm presenting with you is the shell of that model that you're getting into as you get closer to analysis you measure the impact by talking to people getting an idea of what the asset values are frequency take a look at your rising reports take a look at your phone demand reports take a look at your microsoft reports get a sense of how frequency events like this occur take a look at the capability of the threading of the strengths of controls you have look at the degree of vulnerability and some of these measurements can inform other

things for instance a lawsuit frequency is really just a factor of the vulnerability in the threat of end frequency when you start doing aggregation you want to aggregate different elements as well and sometimes it pays to take a look at things at the level of threat event frequency sometimes it pays to do a deeper dive and take a look uh sorry backwards sometimes it takes take a look at the level of those event frequencies sometimes you want to do a deeper dive and look at vulnerabilities it's really a matter of the context in which you need to do your analysis and sometimes you need to estimate estimation is not a bad thing the trick is that if you understand the

relationship between the two categories estimation come become more form this is the chart that's out of that an open group cookbook by mentioning it lays out the relationship between the different categories this is free available download users so it shows how primary loss factors are derived from a combination of assets and threat it shows how in their methodology secondary laws factors come from organizational versus environmental and how they are tied into a lost magnitude to get at that impact level of harm it shows how in their methodology you look at vulnerability as threat versus control or in other words threat of the vulnerability is really the gap in the controls etc that's the decision can be done at any

point on the trade where you don't have hard data and as you move forward with the methodology you will put in place better data this thing gets more precise over time so vulnerability is again with capability to get past your control it's nothing more than that the cbs has two numbers provide a point base of comparison but only if they are placed in the context of your controls so the numbers that you get from your nessa scan or from your um from your rapid 7 scan or from your other vendor scan they're a great starting place but the cbss numbers are meant to be placed into the context of your environment and so what you're supposed to do after

you get those numbers from your scan is put them inside of this calculator in the context of the environment to find out that that 10 is really for you and two because of where it sits in your network and the other controls in place that too is the number you care about not the 10 that next pose game probability i love this probability is the most misunderstood part of statistical analysis let me give you an example everyone in this room has a 100 probability of dying yes i'm known as the killjoy parties your chance of dying right now however is significantly less than one percent that's an event frequency this is why probability is not terribly

useful what you want to care about is what is the chance of dying right now not the chance of dying which is 100 so event frequency is the chance of something happening right now and that's what you want to take a look at and by the way if you didn't understand why the method of measuring risk was broken before keep in mind that nist measures risk off of probability not off of event frequency this is why looking at risk from the classic point of view of the probability of an event times impactive event is a broken way of doing analysis frequency frequency can be derived from historical data but past performance is no guarantee of future events see sony

the day before the compromise so taking a look at historical data to try to get a sense of what's going to happen is not exactly the best way to go about it this is why the people who do well at the stock market don't look at historical data they do some event frequency analysis now we can do that a little bit based upon how easy this is to contact the vulnerability what is the probability that somebody would go about exploring this how attractive are we as a target and you can estimate these things using a beta per distribution and things get better as you calibrate i'll talk a little bit about calibration in a moment

but i will expose the mechanics of how beta perk works and to talk about it as an analytical model versus other analytical models and what do i mean by calibrating so let's start with calibration is a measure of your confidence in the numbers you provide and sometimes the more knowledgeable you are you get a full sense of confidence and let me give you an example on what day was the declaration of independence voted on by congress can anybody tell me who hasn't seen this presentation before and is done googling phase jim july 2nd july 2nd anybody else in the room i'm here in stony silence july 2nd just happens to be 100 correct thank you jim

it was ratified on july 4th which is the date that pretty much everybody knows but the moment i threw the question in the air everybody who thought they knew july 4th questioned their own knowledge well what you should have done and said is that okay it could not have been later than july 4th so i should probably pick a date between june 1st and july 4th and i have now calibrated my answer that's what calibration is is choosing a range of possibilities and saying is between here and here and more precision than that may not be more important for what i'm trying to understand experts overestimate their level of confidence until they've learned how to

calibrate and by the way if you haven't run out and purchase how to measure anything by doug hubbard no i don't get proceeds on the book go out buy the book read the book it's more important than anything it teaches you how to calibrate there's some great calibration exercises in it and he's got some great papers on doing calibration that are supplements the books that are freely downloadable on this website how to measure anything an excellent excellent book so now with the beta part beta pert provides a reliable way to estimate probability it basically allows you to calibrate a mean differently than your classic average takes a look at an optimistic estimate added to g times the most likely

estimate added to the pessimistic estimate divided by that same g plus two to get to the estimate of likelihood and in the classic part distribution g is a constant g is four now a lovely lovely analyst by the name of david rose proposed a replace gene with a value indicating confidence let me get a more realistic estimate of frequency bidding for this formula like right down here which is the formula of the diameter that i use in my spreadsheet now confidence is obviously a variable around four how close to four you get how close to the classic first distribution you are the further away from before you are the less confidence you have in your

estimates and you can use that as you collect your data in order to express your own uncertainty about your estimates providing a more reliable analysis in the process now there are some people who think that part distributions need random number inputs uh namely microsoft they give you these great formulas for pert built into excel that depend upon random number influence other people disagree with that one of the reasons why i went with iso 27 a long time is the idea that the measurement should be consistent over time and that does not depend upon random numbers other analysts think that no you need to introduce a randomness into this make your own decisions and go this is what a beta part distribution

looks like and the target you're trying to hit is that memorize that that is your mode that is the number that you want to hit an alternative model is to take a look at the power law distribution parallel distribution you calculate the slope in the intercept using a random generator the size of the event in the event frequency and this has been a great model for showing that the frequency the magnitude of disasters and is often used to calculate hurricanes and earthquakes and the impact of those magnitudes so this is a great alternative model especially if you're doing business impact analysis sorry to me regarding business continuity risk so again you don't have to grow with a per

distribution model you can go with the power level distribution model it will provide you with a different analysis go with the one that's more appropriate for your contacts now i mentioned excel excel is a wonderful tool and there are some powerful distribution formulas built into itself there's a purchase formula built into excel but it depends upon randomness so what a bunch of folks did is they created a lovely free add-on to excel called open part that gives you better formulas that do not rely upon randomness and also the wonderful part of commercial tools developed from those software from individuals earlier and risk amp which provides their own part distribution for much less than uh what david rose does and of course is

always excel itself which is a commercial tool now calculating risk so you've got your acid kind of which provide from interviews you have an impact to the organization you've got your thread catalog bits or other thread catalogs you've got your controls catalog iso 2700x or kobit or something else you've got you've done your vulnerability analysis you've got your cvs test scores if you're smart you put them in the environmental context using that freely available calculator on the nist site and you get in the sense this takes time it does but you do some frequency estimation calibration and finally you've got your impact so you're now ready to do some miscalculation and i'm going to throw a curveball into

this before we do so and that is what's called a monte carlo simulation monte carlo simulation is a methodology of estimating reality it was used uh within the manhattan project and it works wonderfully if you feed into it the results of your beta part distributions because it depends upon needing a uniform distribution with a large variety of inputs it performs a deterministic computation and allows you to aggregate the results it is the perfect tool to estimate impact and it's freely available and results look somewhat like this and again you care about the numbers up in here you don't want to worry about the outliers you want to carry about your modes those chances where it's most likely to

happen frequency analysis not probability now we'll worry about after the analysis after the analysis let's get into the goal so this is a tool that i built and the reason why i built it is because to be honest with you i'm the security manager of a small enterprise i don't have twenty thousand dollars to spend on somebody else's tool i don't have twenty thousand dollars to spend on a consultant who's gonna come in and tell me oh in order to measure your risk what you do is you take the cbss numbers of all your vulnerabilities from point of address to your data point you add them together and that's your risk metric i've just wasted twenty thousand dollars

and by the way there are consultants who do exactly that so this is that fifth spreadsheet that i mentioned which is freely available off the bit site what i've done is i've expanded on it what i've added to it here is first of all i've used the filtering of excel to allow me to filter on different um asset categories now let me just expand upon that and here's my basal loss categories let me expand upon that so you can see everything that's in here expand on my thread events and now everything else is already expanded and now you can see as i go through this that i have as i page down just a few

items about that there's over 600 line items in the spreadsheet which is why having the filtering really helps you out because if all you care about is doing an analysis of let's say bomb threats i'd isolate that down and take a look at all the controls i have in place regarding bomb threats or something slightly more realistic let's say that i want to take a look at all the threat events i have regarding venus failure i can isolate that one down and just focus it on dns failure and i do this on a parasite basis let's take a look at something some of the other things we have there now that's based upon your third events but what

about the different vulnerability categories and you can expand on this but let's say that you only want to care about expanding myself here a little bit so you can read these things you want to care about confidential discussions taking place in unsecured areas because i'm running a hospital i can isolate out that particular vulnerability and do an analysis based upon that or i can do the entire thing as important are you accepting questions i have accepting questions let's try to keep all the questions for the end okay um and again there are different security controls that just happen to map to those vulnerabilities how i am trying to prevent these things ideally as i what i've done is i've added to

that fine work that bits put in okay great and i've added in the analysis so i have a thread capability i have a control strength so let me expand out my thread capability and walk through one of these things so what is the minimum skill needed in order to perform internal fraud uh impact um it's quite from the vulnerability regarding um times log ons what's the minimum skill set needed for that what's the most likely skill set needed for that what's the maximum still setting it for that and a scale of 100. i put down those numbers i put in my confidence on those numbers that i've just estimated today and i get an analysis using the correct

distribution formula that i've talked about which i've planned to put in so you can see the analysis being done that gives me my estimate on the likelihood of this happening in my organization once i've captured that i then separately analyze the strengths of the controls that i've put in place against those in order to get into what is my actual vulnerability as i compare the strength of my control to how easy it is to exploit the vulnerability now i'm pulling this out of an embedded spreadsheet that does that comparison so let's take a look at that particular spreadsheet so that's the vulnerability spreadsheet the vulnerability worksheet i'll show you what i'm doing here so i compare control strength versus

vulnerability strings to get into an aggregate number and i've built both of those vulnerability strengths and control strengths burst upon a perk distribution of my confidence level uh comparing the middle and the max voltage going back here and so i do this with vulnerability i do this with threat event frequency with also better frequency and then i start putting in the actual impact impacts i've pulled the impacts from the profitable magazine table but what i'm doing here is i'm putting in again the minimum mode maximum my confidence level of the actual dollar amounts of an impact from our organization based upon cost to respond cost complaints and fines that may incur etc now if i'm smart

i take in those basic estimates which remember our aggregate numbers i get my from my cfo from my ceo from my accountants from other experts within the organization i aggregate those numbers and then i do a lot to call them simulation on them and you're probably saying that looks like a lot of work let me show you what open part gets you it's an add-on it plugs very very nicely and what we're going to do is we're going to do a benefit simulation and let's say we're going to put in a minimum estimate on one million dollars yes one million dollars and the most likely estimate is 2.5 volume you typed in 100k i can't read without my reading glasses

i can't see any of you so i've got the minimum of 100k i've got a likelihood or more likely 2.5 billion and let's put in 10 whatever that number is and here we go a nice little monte carlo simulation that tells me where i need to pay attention to it in terms of the impact of my organization numbers that i can capture out of here bring back into my analysis in order to get access to risk so i bring those numbers backwards all the way back to my key risk measurement tool and i'll put those numbers in for each one of those categories productivity response replacement defines competitive advantage the final get at for this particular

threat events i get acting on this vulnerability against that control said in my environment in the context of these impacts this is the likely impact of my organization and then which ones do i care about i care about the big ones i start to manage those firms very very elegant way of deciding what to care about first if i just filter on as an example everything that is a million dollars or more turning off over these things i can turn off seeing all the vulnerabilities that have a negligible impact among organizations i don't worry about those right now i worry about the ones that are going to have an impact to 10 million of a

million dollars alone and if my enterprise is big enough where you can afford a risk loss of over a million dollars i filter higher but now we start with the greatest impacts that i use that to prioritize the management of risk getting back to finally okay i've done the analysis what do i do i determine i communicate i bring this up to upper management i let them make a business decision what we're going to do to remediate those risks so we shore up the controls on vulnerabilities where we have a million dollars exposure are warm and all of a sudden i have no trouble getting budget in order to address the concerns and they had input to these things and i

can show them how their numbers impacted the results of this analysis and none of this is rocket science this is all standard business analysis this is stuff they'll get does anybody have any questions yes um early in your in your talk you made a really fascinating statement protect against the impact not the threat and i was wondering if this was that right exactly okay this is exactly that that's why i asked the questions at the end yes so a bunch of us last year been trying to push adversary-centric analysis into this because it's not simply what's most important to you it's what's most important to them back to the excellent point and there's a very high affinity between an

adversary class and certain asset types and other methods so you can look at the verizon report and see very tight clustering that's less than ten percent of the article possible for scripting etc yeah the trouble with the ryzen report and you want to keep this in mind when looking at some of these reports is they have interesting little biases i agree i agree that's not the point this is true there are tight clusters of who wants what and how they go about it and many of our models like this don't back to that so you may have a million dollar asset that isn't targeted by anybody right are you starting to work adversary priorities into these

absolutely in fact that is why you take a look at um i agree with you about bias yeah that's why what you want to do is when you take a look at this stuff see if i have time to break it down with you um probability of action so you nest it in there i nested them now so if i if i break that down that's where i look into what about what is the likelihood of my adversaries taking action against this is this the target that they're after yeah i think one of the mistakes is people protecting pci data but not public websites and anonymous care about pci data they care about public websites

so i'm telling my executives respond very well to adversary categories they do and uh the terror model by the way of risk analysis is built off of exactly that kind of perspective and so it's not a it's not a bad approach to take um the impact your organization is going to be necessarily low relative to other um threat events so it's just one kind of analysis that you're going to want to do for certainly what anonymous did the sony with their um with their attacks was very colorful made the news but in terms of actual dollar values on the balance sheet it had a negligible impact compared to what other attacks have as an example the attack on hannaford

and what that did to that bottom line so yes pci data may not be the what's anonymous is going after but the impact of the organization if you get compromised is significantly higher and you want to be managing your risks based upon reputational data or based on actual harm to the organization the answer to the question is yes the big issue there is prioritization and by capturing that language inside of this part of the analysis you've got your hacker story captured and unfortunately i only have an hour so i don't have a chance to go into every aspect of it anybody else's questions slide deck available online the high deck is available um it will be available part of the

security b-side stuff i can make it available to anybody who does it right away uh the spreadsheet is available uh this is all built off of other people's work just being brought together into one place so it's a free tool you want it you got it anybody else can we get yours yes you can as long as you come and work for me obviously now this is how done on my company is confidential you'll have to sign an mba or from workforce but yes you can get it no i just want like the formulas free built in no i will give you the spreadsheet you want the spreadsheet is yours thank you anybody else thank you for coming