GT - The Power Law of Information - Michael Roytman

oh

and okay uh we're going to going get started here uh my name is Rob bird I'm going to be hosting your ground truth track today um like to welcome everybody to the first machine learning and math track at any Conference held ever well at least in security so thank you all for your interest and excitement about this um first up is Mike Reitman with risk iio and he's going to do a presentation here on the power law of information so with that Mike we'll let you take it away cool I feel really sporty on a track track all right so many empirical quantities they cluster around a typical value the dice rolls in these casinos the number of

reporto in the wall of sheep the pressure of the sea level and the temperature on a sunny day at black hat all of these things have an average and their distributions sometimes shift away a negligible amount from the typical value sometimes maybe an order of magnitude off but for the most part they stay within the representative samples so it's a really useful statement to say that it's really [ __ ] hot in Vegas in August every time blackhead happens because it never really deviates from that even in the largest deviations which are exceptionally rare it can be a factor or two away it can be 70° it can be 110 but the distribution is really well

characterized by quoting its mean and standard deviation however that's not true for everything and that's what this talk is going to be about the times that matter to you as information security practitioners when the mean and the standard deviation just can't capture what's going

on so my name is Alex hudon and I model risk for a small too big to fail Bank last year like every other day I woke up and I built a risk model and since we're a bank we track the prices of a lot of things so for one of those widgets I built a price distribution model this one is a normal Devi de distribution and I assumed that the standard deviation was about 3% which is a typical number for Price fluctuation in a financial Market and my boss used this to make some decisions and we were quite happy we made Millions from the everyday price fluctuations however today we are [ __ ] because today's Black Monday October

19th 1987 and the S&P drops by 21% my boss freaks out the firm is in financial ruin and my kids starve so how did this happen under my model the probability of a 21% fluctuation is 10 to the -16th which is zero so what happened distribution of price actually has a really fat tail in fact the mistake I made was using a normal distribution so take a look at what happens if we use a simple power law distribution instead so this is a students uh distribution with three degrees of freedom as the power law and now the chance of a 21% fluctuation is actually 0.008% which is something that my risk model would have certainly included and certainly would

have changed our behavior on the markets the good news is that most Financial firms actually practice this and they use these models because they've had those massive black Monday failures many times over but in information security we're just not there yet we haven't had those failures guide us the right way so often Russell Thomas likes to point out people mistake what they did not predict for a Black Swan however what makes a Black Swan event is not the event itself it's rather how that event fits into the object Observer system what that means really specifically for us is how your vulnerability management practice or your risk management practice observes those risk squand and makes those black swans and makes

assumptions about the risk that they pose so in fact the paradigm shift to using a power law distribution instead of a normal distribution to describe many of the variables that we use in information security explains away plenty of the things that we consider black swans may make making the object Observer system more receptive to those rare high impact events the really simple way to think about this is something like the target breach which if you look at the distribution of breaches beforehand and model it with a normal distribution you probably wouldn't have thought that it would have had that high of an impact if you model it as a power Lot distribution using the past data it's safe to assume that that

was an event that had some tangible probability of happening so in fact nothing is linear in information security and I have some evidence of this which I'll present and I don't have evidence for some of it as well this is an ongoing process but I want to make the the claim that just about every variable we use is power law distributed and I'll explain why I think that's true this talk is about power laws that occur in infosec and I've done some research um I just hope to make everybody think twice about using the normal distribution in any model again so what are Power laws par laws are distributions that describe scale-free phenomenon what this means in

layman's terms is that the same mechanism Works across any range of orders of magnitude or scales in fact power laws are a necessary and sufficient condition for this kind of scale-free phenomenon what that really means is that if attackers behave the same way at you know the scale where they have 10,000 breaches a day versus one breach where they perpetrate 10,000 breaches a day versus one breach a day the mechanism under which they do those attacks and the mechanisms under which they interact with Defenders should be the same and we'll see how that's true later the importance and the uqu of scale free Behavior was first pointed out by Mandel BR who coined the term

fractals and that's what we see in nature that's what you see in all these geometric shapes that no matter how far you zoom in or zoom out look exactly the same in fractals the behavior can be the same across length time price or any other relevant variable so think of that when you think about the distributions I'm about to show you a quantity is said to follow power law if it's drawn from a probability distribution that looks kind of like that um Alpha is a constant parameter of the distribution it's known as the scaling parameter and typical scaling parameters are in the 2 to three range but there are exceptions and infosec is one of those exceptions I

think so lots of things follow power laws um the oldest and cleanest statistical regularity in our understanding of international relations or more specifically combat is Richardson's Law which was from 1948 and his law states the severity of warfare is power law distributed Richardson was actually a doctor and a soldier and a statistician he collected data about the severity of C casties over through in Wars over time and found that that distribution closely mirror a power law distribution and since then as more data has come in as we've gotten more data collected more data we see that this holds true and this behavior is not unique to Wars but I wanted to talk about the war analogy first because

information security is a lot like that there's an attacker and a Defender and there's some kind of cost to that interaction and so if it occurs in human combat there's no reason why when we scale it up when we make it electronic and move faster and have more variable it wouldn't hold to the same distribution principles but this Behavior also occurs in traffic jams and earthquakes in coastlines asteroid impacts language wealth firm size salaries Guild sizes and World of Warcraft you name it um these power laws are considered Fingerprints of a complex system so in Academia what they mean by complex is this shifting concept nobody really knows but essentially the things that we can gleam out of their

definition of complex is that these systems produce patterned outputs but these outputs have no standard for lack of a better term mean in the gausian sense uh more often than not uh Power L distribution only applies to the tail and not the actual thing so let's say uh for firms that are bigger than 20 people those follow a power law and so in infos when we think about say the distribution of breaches or the distribution of attacks by cve type that should also be true you know the ones that have only one attack on a particular CBE type probably don't fall into that distribution parameter so on the slide behind me and this is one of the few

slides in this talk you actually have to pay attention to um there are examples of power law distributions uh take a look at the top left there's cities which follow power law distributions there's earthquakes and there's citation patterns I think all of those are kind of good models for talking about breaches or attacks and information security firstly cities or the size of cities will certainly affect how attackers and Defenders interact because those have infrastructure that is being attacked or defended earthquakes are a good model because they're actually the loss functions for a natural phenomenon but our loss function is closely eily similar to that actually when I show you the distribution of breaches over time

or impact of breaches that distribution looks a lot like that very middle one and on the bottom right you have citations um and so while that's not reflective itself there's also email and all sorts of things that relate to information itself and since what is being stolen or what is being breached for us is information the fact that the distribution of the sizes of or the frequency of this information is power La distributed is also a hint towards us not using normal distributions so tails of these distributions are vitally important a power law is an instance of what people call a fat tailed or longtailed distribution and this is just a loose term for something that as the order of

magnitude or as the impact of the thing keeps getting bigger the probability stays afloat uh so there's never a zero or almost never a zero probability some event happening in a power law distribution that is to say if what you think is the average amount of a tax on a particular type of cve or against your business on a day is 10,000 um in a normal distribution the probability that it's 2 million is probably zero um in a power LW distribution that is a non-zero probability and it will always be nonzero no matter what number you pick and that's really important for us in modeling risk and measuring those fat taals is what we're going to talk about

uh measuring a fat tail is quite difficult the data I'm going to show you I'm not particularly great at measuring it I use some regression models and uh least error mean least Le least error estimators or minimizers um but that's not a scientifically sound way to do it um it gives you a proof that a power law might exist but I think the fact that the tail is fat is sufficient in guiding the kind of decision processes that we make um just the fact that it's not normal is enough for our discussion um the question of proving whether something is power law is often reduced to just the question of how fat is the tail and the fatter it is both the more

confident you can be that it's a power L distribution and the more these uh strategic insights that I want to talk about uh are pull true I guess so here's a comparison from Russell Thomas on his blog about uh on normal distribution a log normal and a Paro which is an instance of a Power log and when you look at them you know at their initial scales there doesn't seem to be much of a difference in the Tails but if you dig a little bit deeper you can see that in a in the truncated parator distribution the probability is kind of asymptotically never hit zero no matter which order of magnitude you go out to there's some probability that

that event will happen whereas in the normal and even in the log normal at some point it hits zero and that's the essential that's the important takeaway that you know you should expect the unexpected and there's always some probability of a largement happen but more interestingly if you look at the mean of these distributions as you get more data and that's what this graph is showing as you get more data you can be more certain about the mean of a normal or a lock normal distribution but you can't say the same about a parallel distribution the mean follows a random walk process you have no idea if you get more data whether the mean will stay the same or

shift and if you think about it intuitively in information security that's been true over the past couple of years we had this conception of what an average breach might look like or we had a conception of what an average mechanism for a breach might look like and as new methods keep coming out or as new breaches occur that mean shifts so wildly and moves up or sometimes down if we're talking about methods so much that that past distribution wasn't really useful for a strategic decision-making and this is where a lot of businesses fail because they rely on those old assumptions of what was happening rather than taking into account the new so why does this matter it's because uh when

the tales are small we can say meaningful things about the and also the variance of the distribution so an interesting aspect of power laws in general and a mathematical aspect is that that exponent Alpha has a really natural interpretation as well it's the cut off above which moment generating functions don't exist and so what that really means is if it's less than two the variance probably doesn't exist for that distribution and if it's less than one then the mean does not exist for that distribution um so for this reason we can say things like there's no such thing as an average flood right you've never heard somebody say the average flood is this much water instead they

say this is a 10year flood or this is a 100-year flood and my contention is that we should be thinking about losses in infosec the same way there's no such thing as an average loss amount there's no such thing as an average dollar cost per record loss there's instead a 10year breach or a hundred-year breach so um back in our own industry why would information security exhibit such power LW behavior and where does it actually occur so first when two distributions are combined the fattest taale always wins which is to say if something is a factor of something that's normally distributed and power law distributed that new thing will be power law distributed um if you think

about it intuitively that then the things that make up the attacker Defender interaction and information security if their power law distributed so should the results and cities the size of the internet even the size of terrorist attacks are all power law distributed second information security is not only the combination of those many factors but it also makes the the the system more complex because we interact with it right we as Defenders change the way in which attackers interact with the system we go to these conferences and we share knowledge about it so that has an exogenous effect on information security as well we would expect it not only to inherit those variables but also create this kind of

complex behavior that people talk about power LW being a fingerprint of meaning that the tactics shift and the size of the landscape shifts constantly in such ways that distributions that would character ize it at a point in time don't really work so let's talk about a couple of the takeaways that we have and this is where I'm going to dive into the data that I have the first law that I can significant that I can prove with a lot of confidence is that breach frequency by cve type is power law distributed so this comes from the open threat exchange and this is data for Riso and these are successful exploits over the past year and I've grouped them by their cve type

why is that useful it's because it tells you how your remediation practice should work or how you should think about the metrics in that remediation practice so the chance that a particular CV has a high breach volume is substantially higher than we previously thought is what this tells us it's just like in the Alex Hutton example that the sap drops by 21% you would underestimate it significantly if you were to say a new cve came out and you know based on the prior distribution of cves chances are the risk that it poses to my organization Falls somewhere in this range that's just not true and the assessments have to be done in as they come out and in terms of other data

namely about what's happening in real time um I have a different talk that I gave last year here which is about why CVSs scores don't particularly work and this is a good way to conceptualize that power LW distributions are scale invariant and time invariant so as the amount of breaches per cve type or the importance of a defending against a particular cve changes of over time a CVSs score is just a snapshot of what people thought that was at the time it's probably in fact it is drawn from a normal distribution if you look at the way that the scores were conceived originally whereas the way we should be thinking about these now is there's a

non-zero chance and likely a high chance that a new CV type might see a million breaches in the next month or two million breaches in the next month something that a normal distribution of CV scores just couldn't conceptualize um for those technically minded the X cut off for this is 15 so that means uh if there's less than 15 breaches per day on a particular cve it doesn't follow a power law distribution and that's probably because they're just super targeted attacks or things that are super hard to exfiltrate they're not things that have exploit code or have met spits written against them um the alpha is 1.5 and the confidence in this test is uh 0.9 which is the kagor of

smov DV value which is the way that you test for whether something is a power LW not visually but visually usually this one is pretty much a power law and that value Falls within the C off of saying yeah we can accept the hypothesis that this follow as a power law distribution regardless of whether it does or not though the tail is really fat and we should start thinking about cves that way so what does this mean for you it means that one vulnerability will have an extremely high probability of causing a breach or a group of vulnerabilities and since this breach data comes from how attackers are actually behaving it means that having a handle on threat

intelligence globally allows you to identify which vulnerabilities are most likely to cause those breaches this also means that that intelligence needs to be fairly real time or you're lagging behind and you'll never expect the high probability CVS it means that that you have to shift your strategy away from trying to fix everything or trying to assume that there's some kind of average risk that a vulnerability poses to your environment and instead uh focus on identifying and reming vulnerabilities which are the most likely to cause a breach and framing your risk management practice or framing your evaluation of vulnerabilities in terms of which one is most likely to cause a breach is nonlinear thinking driven by these power

law distributions and that's really what I want to impart with the difference between what I think a lot of us assume and what is actually happening the second law is about actual breach sizes Uh Kevin Thompson has a talk tomorrow at 2m and this talk talks about there he is talks about the vars Community database it's a research project aimed at Gathering all the news articles about information security and trying to get some data about how big those breaches are and and he in his talk also supposes that this is a parot distribution uh if you just look at this analytically it looks to be to to follow a parw distribution um and what does

this mean this is by records count by record count in a um in a breach that they found in the news so there's other proof of this that has been around for a while but people don't really look at it Mard and saor from eth Zurich in 2009 looked at data loss DB and they looked at ID theft frequency so there idea of a record was not whatever data it was either social security number or a credit card something that caused actual Financial impact and they looked at all the incidents where they had those metrics and they found that this follows a power law extremely well and they have really nice proofs that you can look up what's interesting to note

here is that the alpha is stably 7 and what that means is that there's no mean whatsoever there's no way to characterize the average amount of financial damage or ID theft that can be caused by an a a breach there's also no way to characterize a standard deviation this is one of those power laws that's you know terrifyingly hard to predict anything about the future what's also interesting is that if you look at the impact of Terror attacks globally over time that also follows an extremely stable distribution of 0.008 and it's not surprising that the two are closely linked because both involve attacker Defender interactions where you have to have some intelligence about the opponent and you might not and you might

be messing up how you actually do it another interesting uh finding from their paper is that if you separate by industry and they don't have that many they just have like industry government education and medical nothing changes um and that finding is really interesting it also extends to the size of the firms actually but I don't have the graph for that that finding is really interesting because it means that as the sizes of firms go up or maybe as the types go up or down I don't know whether government is above or below medicine but as those things shift uh so do the ways that attackers interact with them meaning that if you're a really large bank and

you have a really great security practice um the end result or the residual risk exposure that you're faced with is likely the same as a small organization with a much shittier security practice just because you're a bigger Target of opportunity for attackers so that's why this kind of scale-free phenomenon exists because the attackers are interacting with your firm size or with the type of firm that you have much in the same way that you are so you know your budget is bigger they're targeting you more that's not you know their motivation for attacking anymore but those things just seem to correlate in such a way that regardless of what industry or size you're in this

seems to be the fact the case um so what do those things mean about the impact or record sizes of breaches um those are just some ways of modeling impact um lots of people have other ones but it's safe to say that if these are somehow have a really fat tail uh then the actual losses that are you will find by incorporating more factors will also have effect F tail for the previous reasons of combining two distributions uh you have to keep that fat tail so the takeaway here is that impact is concentrated in the fat taals of the distributions as well so both the probability of the attack path and of the impact is concentrated in the fat

tail of distribution and that means that our strategies need to be tailored to preventing one big breach this also means that there's no average breach and estimates of potential losses need to be catering to those scenarios like the Black Monday example that we missed in the opening example rather than catering to something average or the average day in security um this is also a really useful way to talk to you know talk to the sea level about what's happening in information security you might have an average snapshot of how much risk or what the blast breach cost you and that might be driving your budgets but the real argument is that that's not descriptive of what might happen

tomorrow and that's not descriptive because we have a lot of data about how that's shifted over time and how those distributions look and the last takeaway is that the breach frequency by day also follows a power law distribution uh eily similar to the cve distribution so what this means is that the amount of breaches or successful exploits that happen at any given day also the the highest frequency things are concentrated in the fat tail of this distribution and the combination of these three things I suppose means that both the vector by which you'll be ATT attacked the impact of that attack and the frequency of how those attack happen attacks happen have no average value and

are concentrated in the fat tales of the distributions that means that your strategies need to shift accordingly and I have some of my own ideas as to how those strategies should shift but Russell Thomas writes an excellent blog post about it and I'm going to go through the points of his that I think are the the smartest about what fat taals imply for analysts and decision makers so his first point is that um the methods of statistical data analysis that are frequentists or based on historical data need to be augmented by other things I think this talk or the data in this talk is one of those other things that you could augment your frequentist analysis with you could say

historically we've seen this but Reitman gave a talk last year at bsides and it indicates that that historical picture is not sufficient to capture our simulations or to capture what will happen in the future um the next point is that you could also run simulations to augmented you could run lab experiments or just use subjective probability estimates of which this is probably one the second recommendation he has is to resist using colloquial terms like average typical worst case and I think I've tried to drive that point home throughout that talk that those things just add confusion and misunderstanding and most importantly they add an underestimation of the amount of risk that you're dealing with um communicate and decide using

quantiles and not summary statistics so the mean and the standard deviation don't mean much in most information security settings and so saying something like you know the the probability of a 10year breach is this or the probability of a one of you know a 100 million doll breach is this is a lot more useful than saying the average one has some fixed dollar amount assigned to it I then put some effort into estimating what the fatness of that tail is um and so there's a really nice r package called Power R law that you can just throw in a bunch of uh a bunch of the data like I did and it'll give you good outputs that you can then think

about whether it's an actual power LW or not based on your understanding of the data but more importantly than this just try to see if you have a distribution of something that you're dealing with or you have some historical data plot it out on a log log scale try to see if it actually has a thin or a fat tail it's intuitively really easy to tell and if it's closer to the fat tail then you're probably making some assumptions using that data that shouldn't hold and so let's talk about the four things I've written up there they're like the only things that I've written on the slides whatsoever investing uh time to fix 100% of your

vulnerabilities is just a terrible idea you need to model your risk differently because the way that you end up getting the way that you end up dealing with residual risk indicates that you're modeling it poorly it indicates that those large probability breach events keep happening and keep growing in size and the impacts keep growing in size and the attack paths keep changing in such a way that there's always more attack there's constantly large numbers of automated attacks happening on different cdes um when big when the big loss event happens only one or a few vulnerabilities will be exploited and to stay ahead of them what you need to do is you need to fix a portfolio

vulnerabilities that try to Encompass the things that you think are in that fat tail of distribution that way you're driving down risk the most so to think about this strategically you know that the highest the things that are most useful to attackers are the things that frequently are getting them the biggest impact and the most success are somewhere in the in the fat tail distribution and you need to find ways to characterize what resides in that fat tail so interestingly enough uh the historical quote unquote data analysis that I did in drafting these distributions tells you which of those CVS have the highest breach counts right that comes from a bunch of thread intelligence that we

have at risio that tells us that looking at the landscape the things that are in the fat tail of distribution are this um there are other factors that you might use to figure this out so let's say you analyze your network traffic and you find that there are some things that are happening a large amount of time that are targeting something specific that might give you a clue as to what things are being most successful or similarly what things have been most successful um so the the last thing that I want to leave you with is that I've shown three facets that I think are important and they're important to vulnerability management they're important to risk

management that is probability or attack path attack impact and attack type um and those things while I might have some proof that they're power law distributed are not the only things that are meaning that likely the reason that I'm finding these things and likely the reason that I can give you statistical analysis that proves or doesn't prove that it's a power law is that the system is complex in such a way that most things in it are power law distributed so never use an average metric think about how it's distributed and realize that something like average vulnerability closed or average score reduction or the average tax perct days probably don't mean anything because the system interacts in

a statistically complex way where those types of Statistics just don't apply that's

it okay so we're going to move on to questions I have T-shirts for questions and he has t-shirts for questions so if you have a question or just want a t-shirt ask a question and I'll give you this microphone so you can you

heard so oh wow so use one of those to wipe my hands sure yeah go ahead okay so so my question is uh you talk about talking to the sea level and one of the things that you constantly hear back from the sea level is uh you know they want the average because they want to know how much to spend how do you convert that into a spent right you I we can go back and go yeah you've got to pay attention to this and this is the real metric we should be using but from what it sounds like I need infinite spend and they're not going to like that yeah I don't think it's infinite spend

right these things approach some kind of astical zero or don't really ever hit that zero but I think you could say that realistically we're not concerned with the1 billion do loss because our company doesn't cost that much um the most like you say the most we could ever lose is this much and if we model our losses as a power Lot distribution leading up to that just cut it off there find the area under that curve that's the amount and I think if you look at that that amount will be orders of magnitude bigger than what you're currently dealing with and I think the easier way to convert it is just to say um look I modeled this one

particular point and here like the example I gave at the beginning so here's the $100 million breach on the distribution I'm using to tell you the average here's what it is if I can't really convey that average to you okay tell me how you want me to act right so basically how much money you want to lose yeah exactly that and I think the the argument that this talk makes is that you're going to lose a lot more money than you think continuously throughout the fat tail thank you

Kim so this is a lot of pretty advanced math for a lot of response organizations um whether they're you know funded for this or not is is another question so if you were to boil down the here's the stuff you need to worry about most into you know the key takeaways what should have you have you distilled it down to that level for the teams that can't do this level of analysis on their data um like what would your what would your go-to recommendations be for somebody who's like crap I'm doing it wrong but I don't know how to fix that yeah I think no so I haven't distilled it right that's something I'm struggling with as well

but I think the what I can tell you is that the these things are mostly publicly available data right and you don't need to rerun the analysis you can just say um you know Kevin and Michael analyze the open source databases and they find that the record distributions look like this um how you distill the high level math is a different question right how you convey the difference in those distributions but I think in terms of which factors or how you run that analysis I don't know if you really need to I think you can just say um I suspect it's way worse because it looks like this I just know you know most of the

people who stayed through this full talk like we're data nerds we get this but then when you take it back and you're trying to translate it to the decision makers and explain to them you know it's not about average number of vulnerabilities closed it's about closing the right vulnerabilities it's it's being able to communicate that in an actionable way right so yeah I I agree with you and I'm completely open to ideas of how to distill the how to take away the math right how to simplify it I think that's not just a question for me that's a question for staticians everywhere um to me it's just like the the examples that have actually happened

is the best way to do it um without talking about the distributions at all right to you I've proven that that is the case but to the the sea level I think what matters is like look Black Friday happened once look Target wasn't accounted for in these previous distributions it was a great talk thank

you I think actually the flood analogy and and things that uh non uh uh Security Professionals understand by an by analogy is is the real answer to her question I was don't even deal with the math basically the takeaway is a uh something else in their lives they understand and they've had experience with that uh basically are dealing with what they would consider to be rare events that really aren't that rare yeah and actually as you said the the earthquake one is really good too because it has a scale built into it right so you could say on our security oror scale we're looking at this that's we're only going to focus on these and okay

so I give you two questions you can answer whichever one you want um I'll try to answer both maybe uh so do you think that there's a relationship between the power law distribution and paths through a graph that's the first question the other is um what do you think the relationship is between the um existence of power law distributions in information security problems and uh free will or rational actors kind of those economic Concepts those are both really good questions so to the first one I don't know enough graph Theory right I think that question actually has an answer that we could Google uh based on what is that like fake nodes

okay oh um wait I have a list of references here and one thing that will answer your well like if you want to do this or something there's more too but feel free to email me um the last one the Dan gear talk he actually talks about the node network of the internet and how it propagates itself and so one of the arguments that he makes for why internet for why malware or for why attacks travel so well and follow a power law distribution is because of that node connect those node connections um so the things that make the internet really good are also the things that make the internet really good at being bad

essentially is the argument there um in terms of are these power laws economic yeah I think so but I also don't know if there's a real answer to whether economic power laws are economic or factors of city-state power laws or you know naturalistic power laws and like where you can build a city and how big can it be hi so I'm somewhere in between U statistician or data scientist and an instant uh responder so um my question is pretty much along the lines of within infos SEC given you talked about all the different Industries and different use cases how they still generally follow a power law but even within infos given that there's different spheres different

like within the kill chain there's different things to look at over there have you um kind of spliced it into like into that level granularity and and seeing like that type of analysis and breaking down your data or is it just pretty much hears us overall sum of what we're looking at um correct me like stop me any time if I misunderstanding the question but I think when we talk about breach impact that's the terminal distribution that's the one that's the combination of all the previous ones and all the previous steps of like how did your incident response team do how did your V management team do how did Dev interact with this environment um the end game of

that is the breach power lck um and if that one follows a really fat tail distribution of 007 uh there are other factors that are contributing to fattening that tail which means there's some step in that kill chain that's causing it to be such a fat tail uh what that is is an awesome thing for us to find out in the next 10 years okay that's pretty much the question like within the killchain have you is does that take into account like is it mostly um right so I don't take that into account right I work vulnerability management that's the data I have I can tell you that our power laws seem to be about at a 1.5 coefficient they don't

have standard deviations they might have a mean or might not have a mean uh the terminal ones are 007 so there's something else in the kill chain that's probably worse than vulnerability management okay cool thank you uh so when you were uh oh by the way great talk I to my hat to you if I was wearing one you as well I saw your slides oh thank you thank you so um when you look at the Varys Community database and the the slope there you're looking at all every industry that we have and every size of an organization that we have and stuff so if somebody was trying to apply this yeah that one right there if somebody

was was trying to apply this in their own organization is it fair to use that and that and that slope or do they really need to kind of localize more and this is kind of going back to uh her question you know where she was talking about how can we take the math out of it and I'm wondering actually can we really take the math out of it or do you really need to do this analysis within your own organization and on breaches affecting organizations similar to you so I'd like your opinion on that

okay right right right um so I'm going to give you two answers the first one is one I disagree with but so there's a theoretical argument that I make at some point here and it's like look at these other people have found that the power laws are consistent throughout industry so you shouldn't really worry about it right we found this Factor across Industries and also there's another argument that says that you can decompose them and they'll stay the same and then you know combine those two logical things into a syllogism and you're good to go I don't really agree with that in practice cuz like I would prefer to have data about those things um so that's the kind of

shitty answer of like yeah it applies to everything but the better answer is that what's awesome about power laws is that they're scale and variant and so the same mechanism as at work means that if we find that mechanism is at work on the global scale it is true that it is also at work on your scale hopefully if we prove them correctly right we could be completely wrong about the power LW it's actually something else and then for you it doesn't work that way okay uh I actually have a question myself um so you mentioned that we shouldn't use averages and of course that's true of essentially using that centrality estimator for anything that's

not a normal distribution right so you shouldn't be using averages for anything that doesn't have a bell curve shape anyway um but doesn't the field of robust estimation like using medians uh you mentioned using percentiles or quantiles which are of course derivative of the median um doesn't that generally provide a tool that can be applied kind of generically I mean obviously medians are the safest centrality estimator out there that's why for example the uh US Census has changed to using medians for all their centrality estimation um and there's some pairwise as a f on question there's some pairwise distance-based uh there's something called The Crew rousis estimator for anyone that's touched that that's based on basically

you know just taking the pair wise distances between all your points that can give you a notion of dispersion over any data set it's a robust estimator of dispersion um so it works on on single tailed it works on fat tail distributions I was curious if you've touched on any of those kind of median estimation problems um I think I just conceptualized what you meant by the the point one I don't really see it's inherent value right now but that's just because it's the first time I've heard of it but quants yeah I think the the easiest like strategic thing you can take from this is use quantiles instead of averages and you'll be better off

right away um to say like you know this is in the 90th percentile of things that pose risk to us or things that we should be fixing is what I'm trying to advocate for here right I think that's the solution and I think what I see with a lot of our customers and what I see with a lot of literature that are read in infosec is people are still using averages or they're using metrics that presuppose that the average is a good outcome which is not true like I'd like to see somebody talk about the 90th percentile of vulnerability close rates as their metric for operations but it just doesn't happen I mean I I recently

reviewed a paper where somebody was actually using the average and the variance for a zian distribution which is just absurd right so I understand the problem but it is definitely broad broad spread and it kind of makes you wonder if security got into the practice of being careful with its um use of stuff like medians we would probably save ourselves a lot of pain you know the day trying to communicate messages because I mean CEOs understand medians right they understand those kind of basic stats quantiles percent I think there's also a good distance estimator between what you said and what was said here it's that you know we need to figure out the method that replaces the average that is

also the most simple and most accessible okay are there any other questions sorry to hog the mic there okay well let's thank Mike for oh wait we have one in the

back

okay no I have not and intuitively to me the losses that I'm talking about are not organizational losses that happen after the fact they're actual loss of data or records right that's why I'm never talking about dollar amount well I said it in my talk but it was like a reflex of instinct to talk about dollar amounts I think the translation from a breach what the breach actually did to your organization to the dollar amount is a question that is better save for the keynote I know nothing about that or for that

dude

that's but at the

it's

EX reduce thatu that

[Music] risk now I have another theory it's that if you probably look at the stuff that happened financially after the fact like maybe the numbers are reduced and maybe like who got impacted more shifts around but that probably also still follows a power L distribution because the ways that firms deal internally with that actual loss is subject to the same problems of distribution analysis hey great well let's give uh Mike a big thank you and we'll go [Applause] next okay next up we've got Alex pinto and Kyle well you guys got about 5 minutes so if you want to cycle in and out