Lessons Learned While Modeling Rare Catastrophic Cyber Loss Events

hi there everyone okay uh we are from RMS uh next slide um my name is Russell Thomas and I've been at RMS about uh three years give or take I come to the organization with the background in data science at a Regional Bank uh PhD work in computational social science and before that uh high-tech Enterprise r d marketing manufacturing Bachelors of Science in electrical engineering and management Chris my name is Chris Voss uh I've been at RMS for about seven years um bit sorry closer look at my head a bit of RMS about seven years um been working on our cyber rest models since since the outsets uh I have a background in mathematical modeling

particularly in the context of natural catastrophes so I have a a masters in risk and environmental hazards and a Bachelors in physical geography both from universities in the UK as you can probably tell from my accent um so as Russell mentioned we do cat modeling and for many of you in the audience when we say those words this might be what comes to mind uh however unfortunately today we won't be spending the next 45 minutes checking out different designs of you know denim jackets for cats and that sort of stuff instead what we'll be talking about is catastrophe modeling which is a field within mathematical modeling focusing on quantifying the risk associated with rare severe phenomena things like

hurricanes earthquakes floods pandemics and of course cyber attacks what we have on this lovely slide here that completes Russell it's just an example of some uh realistic but uh synthetic hurricane um tracks from our North Atlantic hurricane model so passovers Russell so when we say rare catastrophic events uh one of the features of this talk is going to be some Seinfeld memes I think we would all agree this particular episode where Frank Costanza fell backwards on a little few Sealy Jerry statue and it ended up in the proctologist's office as a very rare very catastrophic event so as we mentioned you know Russell and I focus on Cyber risk modeling but uh this is just this is just one of a whole

Suite of different catastrophe models that RMS build our firm builds everything from North Atlantic hurricane models to European flood models Asian earthquake models and you know terrorism pandemic and of course cyber risk models so our primary clients sit within the insurance sphere uh this little slide here just kind of gives an overview of the insurance value chain starting from Individual companies who if they want to buy insurance will speak with insurance brokers they'll get a policy of an insurance company but those insurance companies typically don't want to those those companies um don't want to hold all the risk on their balance balance sheet so typically they interact with reinsurers to shift some of that off and you can see the

flow of risk from left to right here now RMS is a company provides risk quantification tools and services throughout this value chain uh in the context of cyber risk we're primarily on the sort of right hand side insurers reinsurance Brokers and reinsurers themselves so what do we actually mean by risk well very simply risk is often defined as being the product of likelihood and impact now in our context impact actually means the direct Financial losses experienced by companies that unfortunately are on the the bad end or a cyber instance now on the right hand side here you can see some of the types of impacts that are considered by our modeling you can see everything from

Lost Revenue that occurs during an incident so of course as we know when a bad ransomware incident occurs often that means a company can't operate at 100 so we'll be quantifying that sort of thing quantifying forensics cost incident response costs some kinds of fines notification costs and Ransom payments and that sort of thing however we don't quantify is things like post-incident upgrades lost to Share value and this sort of thing and that's primarily because these are the sorts of losses that are not covered by insurance contracts in the Cyber Insurance sphere so as a result we really focus on incidents that are above a particular severity threshold we're really only interested in stuff that causes realized Financial

pain to companies not so much about other kind of incidents that perhaps network security folks are concerned about intrusions that they want to follow up Etc but we're really interested in if it results in a financial loss we're interested if it doesn't not so much let's please and as part of this we model a diverse range of different types of cyber incidents everything from data breaches to ransomware attacks wipers Cloud outages and this sort of thing and each of those different what we would call sub Perils of cyber have different likelihoods and different Associated Financial losses and of course depending on what type of company we're talking about you know a small company versus a large company the

likelihoods might be different to any industry we're talking about the likelihoods also might be different okay a quick show of hands anybody here work in the insurance industry cyber risk okay interesting uh anybody here do risk modeling as a business as opposed to okay so what I want to contrast here in this next series of slides is how these different perspectives of risk vary and overlap but are significantly different that's really critical to understand that how we approach modeling and how it may be different especially from the Enterprise so if you're a risk manager in an Enterprise essentially all risks all bad things that hap can happen your business are important so if you get hit with a really bad

event causes big Financial losses or big reputation damage you've got to go in front of your board or something that could be a catastrophe as you define it but from a population standpoint if you're the only organization get hit by that the population or the people that look at populations like governments Regulators they may not see this as an extraordinary event it's bad for one but not necessarily for the population but if you start seeing events hitting many organizations essentially or roughly in the same time especially the same type of attack and attack severity now we've got a population level uh catastrophe now it's critical to understand that insurance companies View and manage cyber risk in the context of a portfolio

that's how they decide what the premiums are going to be and what the rules for coverage are going to be how much Capital to allocate and how to even acquire reinsurance and report to regulators so portfolios have boundaries who's in and who's out portfolios have rules of coverage in terms of conditions and even every customer is going to buy different levels of coverage so while a single insurer may look at the population and sort of take that Advantage perspective they're always looking at a subset and a key part of the RMS product is to help Insurance customers go from the population or macro view down to their particular portfolio view and say what does this mean for the types of

customers we cover so what RMS does in our model is in in this version six that we've just introduced we model a synthetic population of all firms above a certain size threshold and further we uh separate this synthetic population by the industrial sector as well as the geographic uh jurisdiction and critical to our modeling is what's the footprint of given attacks and we're concerned about how many threat actors there are and there are campaigns and what are their footprints and different campaigns can have different Footprints and that can affect who's uh who's affected and uh how many so some may be horizontal some may be vertical and in the worst case scenarios they may cover a very very large portion

of the population and this is really a prime concern to our customers and to our models cool thanks Russell so to sort of formalize this I think it's helpful to sort of reiterate a couple of things so the first of which is that we can think about risk in two categories we can think about what's called attritional risk and essentially these are incidents like Russell mentioned at the very beginning which are sort of independent which can still be substantial in scale but are not associated with many many companies being hit so an example of that would be something like the 2017 Equifax data breach which was brutal for Equifax but it's not like it hit

thousands and thousands of companies simultaneously then we have tail risk which in our parlance is really focusing on low probability High severity events that hit many many companies simultaneously so these are things like Wanna Cry and not pettier might be examples of Terror risk and of course we can all imagine much more terrifying examples than any of those you know that might potentially occur so we wanted to sort of touch on how this influences insurance premium because that might be something that you know a touch point that you folks have you know directly with the insurance industry and so insurance premium really is covering the the average loss that or the mean loss that your company might

experience in a given year and this includes both attritional and tail risk so on the right hand side I've got a very super simplified example of an imaginary company and let's say an insurance company comes up with a ten thousand dollar technical premium Now a technical premium essentially is defined as being the amount they're charging directly to cover the losses that you might the claims you might bring it doesn't include profit or other sorts of uh other sorts of costs so next one please also so again these numbers are made up but as hopefully it will help us follow through so the attritional component you can see here is about seven thousand five hundred dollars and the way that

this might be computed again this is a simplification is that you might take the mean loss that the company might experience conditional on an incident so given that they've experienced an incident on average what's the dollar cost and then what you do is you combine that with information about the likelihood of that happening or the probability of that happening in this case five percent for easy numbers and then you get what's a probability weighted loss which in this case is seven thousand five hundred dollars but that only covers the attritional component those sort of independent events what you also need to consider is the tail component which is often called the catastrophe load and this is then

considering okay what about all those events which might hit loads of companies simultaneously in this case here our example is saying that on average if our company gets caught up in one of those events that the loss is going to be 250 000 on average but there's only about a one percent chance in a given year that that this company gets caught up in this and when we do the probability weighted loss we get 2 500 sum them and we get 10 000. um nice place so one thing that we thought is kind of useful to to mention is that you know Russell spoke about the different scales of risk and what what's catastrophic in in the eyes of an

Enterprise versus the eyes of an insurance company or the eyes of a population and it's worth mentioning that you know for an individual company you could imagine that perhaps you know a brutal double extortion event might be the worst case scenario right where all of their commercially confidential information gets stolen you know personal information they have get stolen and leaked and at the same time their operations grind to a hole because everything's encrypted horrendous however due to the scaling properties of double extortion and those sorts of attacks might not be the drivers of population level catastrophe risk it might be something like a wiper that is just rolled out through a worm or something like that

instead and ultimately depending on the angle at which you're approaching risk management you might be concerned about certain types of incidents over others so before we really get into the meat of the presentation we thought it'd be helpful to really spell out what catastrophe models in particular are cyberis model does and what it doesn't do what it does is assesses the likelihood of different loss outcomes we're not trying to make predictions of exactly what will happen now to use a simple maybe appropriate analogy in Las Vegas if we imagine a you know a dice what our model is saying is that the dice has six sides and each side has one over six probability of it of it being rolled

what we're not saying is that the next roll is going to be a two um if we did know that then we wouldn't be here and we'd be in the casino instead um laughs what else what else does it do well what it really tries to aim it aim to do is to capture the key drivers of risk what we're not trying to do is reflect all of the complexity of the real world all of the you know huge depth of technical complexity that you're all super aware of and you know all the complexity of decision making of threat actors ultimately we're trying to identify what is really driving risk um and ultimately this comes back to the

fact that all mathematical models are simplifications of the world they are helpful decision making tools but they're not supposed to you know reflect all of the ugly details of the real world and finally what our models do is they complement expert judgment as a decision-making tool they're not supposed to replace humans because as I mentioned before these models are simplifications so we need expert judgment on top of it you know some people might think that the output is too high too low etc etc so um now that we've sort of laid the groundwork of that we'll move on to our first lesson which is the benefits of causal risk modeling so a causal model is a model that

represents the causal or mechanistic relationships in a system on the flip side statistical models are models that reflect the mathematical or statistical relationship between different variables next slide please so here we have a toy example of a statistical model that you might use in the context of cyber risk so this follows a very very popular framework the frequency severity framework so on the left hand side here we have a probability distribution showing the likelihood of a company experiencing an incident here you can see there's like a 75 chance that it doesn't experience an incident just over 20 chance that it experiences one incident you know and a small percentage that hits two three or four incidents and then on the right

hand side you have the severity side which basically says given that a company has experienced an incident what are the range of potential dollar loss outcomes and they're Associated likelihood so on the right hand side that's a probability density function so essentially where the the curve is highest you're sort of most likely to see an outcome but we can see that all the way down to very large numbers there are there's a chance that the loss plays out this way um one thing that those one thing that these kinds of models uh are good for is when you have a lot of data but what they're not very good for is trying to quantify very very extreme outcomes that

you haven't observed if I want to ask the question what's the likelihood that next year 25 of companies get nailed with a wiper this sort of model can't help me because we haven't observed anything like that in the past we need to use different techniques so yeah so just to underline that last point I want to share a brief story conversation I had uh 2008 or 9 with a famous security consultant keynote speaker and he is that 15 minutes did you just wait 15 minutes thank you scared me so anyway he was arguing against the possibility of ever quantifying low probability High magnitude loss his argument was if you take infinitesimal probabilities against incredibly large numbers your margin of

error you can end up with any results so he was uh just trying to dissuade me and other people from going down that path so coming back to our Seinfeld meme how would you any of you estimate the likelihood probability the risk associated with this particular loss event well I would challenge you to take a standard statistical model of frequency and severity and apply it to this it's very hard to get off the ground and have any credible information so what we at RMS do in our version 6 model is we model a synthetic world that has all of the key elements firms software vulnerabilities campaigns threat actors and we connect them in a mechanistic or causal chain

so in this case if we wanted to include this we'd have to have a threat actor of the type Kramer and Kramer would have to have the capability of building weird things like little statues and his attack pattern would be leaving that statue on the ground where somebody might fall on it and then the causal mechanism is how is somebody like Frank the potential victim with vulnerability here likely to fall and if he falls what's the likelihood that he's going to fall in a particular way that he's going to have to visit a proctologist so to go a little bit deeper into this synthetic world as Russell described we here are really trying to call out as I

mentioned earlier on the key drivers of risk in the Cyber kind of risk ecosystem so here what you'll see is components everything from Individual threat actors that we spawn that have various different characteristics size skill motivation Etc things like software different kinds of software that exists with different market share and what kinds of companies use those pieces of software vulnerabilities of course which are crucial to understand and that the rate at which those vulnerabilities spawn and their characteristics and of course you know what their exploitation kind of characteristics look like but then also things like the different ways in which threat actors can gain initial access into uh into corporates right things like social engineering have extremely

different scaling characteristics to worms for example and modeling the specific ways in which those play out is super important so those of you kind of who work in a field a little bit closer to us might be wondering how on Earth do you operationalize all of that well so this is a bit of a simplification but ultimately the way that this works is for fifty thousand uh synthetic years what we do is we simulate we initialize a world in which you have software with different market share characteristics different vulnerabilities that have been spawned with with various different characteristics and then of course putting into place different ways in which threat actors can get into into

companies and different bad things they can do once they're inside and then those threat actors essentially are able to evaluate their different options each of each year and Russell will talk a little bit later on about how we go about that and then these thread actors will choose what nefarious thing they decide to do and we essentially rinse and repeat this process exploring lots of different states of the world because none of us can you know write down on a piece of paper now exactly what's going to happen from a vulnerabilities perspective you know over the coming year so we need to explore a broad range of potential outcomes and once you repeat this process essentially what

happens is you get synthetic attacks occurring some of which are much larger in scale than others [Music] yeah just uh those of you who might be familiar with epidemiological models of infectious disease especially across a network or a geography similar structure those models take the bacteria point of view or the virus point of view and the hosts the susceptible hosts are sort of provide the backdrop for the virus to move around so in this case the threat actors take the primary active role in decision making and determine what campaigns happen and firms at Enterprise by their policies and practices more or less determine the environment in which they operate cool so as you can imagine you know

building this sort of model as opposed to a statistical model is a huge amount of work both from a Time effort and data requirements perspective so why on Earth do we you know pass out to all of that well as we mentioned earlier on it's our firm belief that you cannot robustly quantify extreme cyber risk through simple statistical models and through this kind of model that we've described extreme events are emergent from this particular model and the reason why that's the case is you will get years inside of this 50 000 year simulation where you might have an extremely large skilled and aggressive threat threat group who are operating in a year in which there are a slew of you know very very

high severity broad reaching vulnerabilities and it's essentially the superposition of many bad things occurring at once that ultimately can result in catastrophic activity um there's also a bunch of other different uh benefits I think really importantly is that assumptions in this sort of model can be directly interrogated by domain experts if a domain expert wants to ask how do you consider patching and you know how do you consider that some companies are slower patching than others if you ask that question to the statistical model that we mentioned at the beginning that question doesn't make any sense there's no notion of patching inside of that model whatsoever whilst in the context of a causal model such as our own you

have particular components that are directly addressing that sort of phenomenon real life and you can make sure that you are capturing the process in a robust manner both from a quantitative and a qualitative perspective um uh okay let's just move on next yeah um so yeah we're kind of taking a little longer than we'd hope but uh um so good news on empirical data so there's a lot of good news I think on the empirical data front I think it's things like the Frameworks that miter have produced things like the attack kpec and defend Frameworks for us have helped absolutely enormously in terms of providing a common language to describe different phenomena but also as a way to

be able to homogenize uh data from different sources you know if you've got Source a and Source B both tracking thread actor activity and they're describing you know persistent techniques persistence technique lateral movement techniques all these sorts of things using the same language for us that makes things a million times easier so this is really great next I think incident data is becoming much more readily reported in the past things like data breaches were you know pretty commonly reported due to regulatory reasons but I think more and more companies are becoming less ashamed of of announcing that they've experienced a ransomware incident or other kinds of incidents like that and for us more data results in better models and just

quickly regulatory influence Securities and Exchange Commission privacy reporting requirements are all and obviously the European requirements are all positive here and from our standpoint our customer standpoint we look forward to more of this uh being more comprehensive cool on top of that you know there's a whole series of uh publicly available and extremely well structured data on vulnerabilities so to some extent also on exploits I think you know there's still some uh perhaps some work to be done there but again that's incredibly useful from our perspective I think something else we really want to call out is the substantial body of really high quality statistically robust work that uh sort of is out in the

literature big big shout out to the folks at scientia but also Folks at Verizon and various other different reports I think this sort of information you know really is positive for the the the discipline of cyber risk quantification in general just historical context some of you maybe have been around in the industry long enough to remember cost of a data breach reports by an organization beginning with the letter P so we have uh significantly Advanced beyond that and then finally you know there's some really great work that's going on in the broader Community um you know where there are very talented dedicated professionals putting their you know data science statistics and I guess cyber security knowledge

together to put together products such as the epss which is the exploit prediction scoring system this is super interesting both from our perspective at the macro level but I think also for you know practitioners as providing you know valuable insights to the probability of individual exploits being of individual of individual vulnerabilities being exploited to help with things like prioritization a big improvement over their uh cbss severity system so all of this Foundation of data was really a prerequisite for us to even consider this sort of model certain areas we definitely feel you know are a little bit more tractable with current data than others I think we're very comfortable with where we are with things like software vulnerability

spawning the cost of individual attacks as well as the mechanisms around initial access vectors and the flip side there's some not so good news so in general in this Arena we're always going to be suffering with relatively short time series compared to natural phenomenon like earthquakes and floods and so on so even if you can collect 20 or 30 years of data really only the last 10 years and sometimes five years is applicable because then you have an environment that's more or less consistent with where you're operating today so we are constantly having to go through data analysis and data evaluation process with any empirical data source as to how far back in time we go and how we make

use of that and of course as we look forward the dynamic nature of what we're dealing with is crucial I'll just mention an area of my own work in modeling high-level Financial theft from Banks which was a very active area for certain nation states uh three to five years ago but those nation states have shifted to cryptocurrency thefts uh so one of the things we have to evaluate in a practical sense is if we're going to be putting out a model that our customers can be using for the next year how how much do we anticipate things that may be in the process of changing uh will the uh prevalence of ransomware continue for the next 12 to 18 to 24

months is that the same as it has now crucially there's still struggling for information around incidents especially breaking down the loss event into categories that we can use what's publicized is not always uh the most important or the most useful and the biggest gap of all is information about threat actors and um I'll just give a shout out had some conversations over the last couple of days and put out over Twitter proposing a project collaborative project around modeling threat actor capability Evolution and change so if anybody is interested in getting involved with that think of it as sort of like an attack framework but focused on threat actors not from a threat Intel point of view

that indicators of compromise and ttps but their organizational capabilities and their value chains so come up and see me afterward so this leaves us with the need to um evaluate uh excuse me fill in the gaps with data with expert knowledge so we mentioned here threat actors and one area I worked on was how do threat actors make decisions around campaigns given the options available I just got the 10 minute notice so I'm going to pick up the pace a little bit I hope that's okay so we're going to walk through a simplified example from the end result backwards so the end result we have in our choice model is Choice probabilities here six campaigns are available to this

threat actor and what we end up here with Choice probability and a random draw is made based on those probabilities to the campaign Choice the choice probabilities here are simply the normalized outcome scores so how do we get the outcome scores and how do we come up with six campaigns well in this particular case it's three initial access excuse me three initial access vectors cross product with two particular attack types execution techniques so we've got ransomware the data exfiltration and three different ways of getting into the population and given their weighted attractiveness that determines the outcome score for each now weighted attractiveness is sort of where the magic happens here it we need to discern the relative attractiveness

of the firms to these different types of an attack so a firm that's highly attractive to a ransomware person is not necessarily as attractive to the same to a different attack or even the same attacker if they're trying to exfiltrate a bunch of confidential health information

a mission so we'll move on to the next thing so this next lesson is I guess a bit more of a tutorial for for those of you who are kind of interested in you know applying statistic simple statistical methods to your risk quantification work so we're going to talk about probability distribution so probability distribution is essentially a mathematical function listing a range of possible outcomes for a particular process and the corresponding likelihoods on the right hand side here we have a normal or a gaussian distribution bell-shaped curve for adult male height in the United States Now using observations we can actually come up with some reasonable parameters for what this mathematical function might look like

and the benefits of fitting these functions are quite numerous actually the first of which is enables you to estimate the likelihood of different outcomes if you wanted to know what's the chance that someone is more than six foot three tall you can ask that to this mathematical function you can ask also what's the likelihood that they're between five foot and five foot five tall you can also ask that sort of thing it also enables you to determine the likelihood of unobserved outcomes and enables you to generate synthetic data through sampling in this case here Russell just showed you know if I ask this function give me five numbers it will draw in a way that's proportionate

to the probability density now so how would you actually go about doing this in this imaginary example we're saying you know if I go around my local town and measure the height of 100 different men I might get a histogram like you can see on the top right sorry it's a little bit small and if I was thinking okay what sort of probability distribution do I want to fit to this there's lots of different considerations that I might want to make but one of them should be what is my prior knowledge about this particular process you can see in the histogram it's bimodal meaning it has two humps however having lived as a human for a

number of years I know that actually in general you know there's a sort of a smooth spectrum of heights and there's not a small there's not a valley of height in uh in the male population so we're going to choose a unimodal distribution next so in order to actually fit this there's a lot of different tools that can really make your life easy we use various different languages in our team I quite like Julia and what you can see here is you can fit that red line on the right hand side with just a single line of code it's not too terrifying you know there's of course a rabbit hole that you can go down from the statistics

perspective but just to start getting familiar with this sort of stuff there are a lot of tools out there that can really help now there's a question of how good a job did I do and there are a whole bunch of diagnostic plots that you can do which are pretty straightforward to follow but uh I'm running out of time so I'll pass the button over to Russell again so a huge lesson that we learned through our experience in modeling the iterative process of looking at the data generating the model looking at the output is our definition of tail risk what drives that curve whether it's in the critical area or not so critical area is threat actor capabilities

so I want to give you a window into into this process and how we arrive there and one critical factor in this tail risk is what is the addressable population for any attack so the addressable population for the not Petty attack was huge because it was a wormable uh Windows vulnerability and therefore it could spread very widely so what factors determine that addressable population so obviously attack a strategy exploit capability products they're going after and their coverage in the population uh and I'm going to skip to the bottom which are interdependencies between vulnerabilities if they exploit multiple vulnerabilities a given attack May exploit two three five maybe even more vulnerabilities conditional on the victim that they run

into if those relationships are or logical relationships that expands their Market gives them options but if those are and relationships those are points of failure because if any vulnerability any exploit in that sequence of ands fails the whole sequence fails so we believe that attackers when they're defining their attack strategy either by conscious planning or by experience essentially assemble their attack strategy to try to come up with an addressable population which fits their goals and their capabilities so what one lens on this is the initial access Vector of worms so here we've modeled just the behavior of worms uh and What proportion of the population that they would be able to access and you can see in and this is an

example of a sub analysis within our broader simulation you can see the vast majority of worms are down in the 20 range so what's interesting is this range above 50 percent this is the worst case of the worst case and this slide here it's a bit of an eye chart but this is a diagnostic output from our simulation where we look at in detail the characteristics of these simulated attacks in the far part of the tail so we went in and estimated well how prevalent are multiple uh vulnerabilities cross OS vulnerabilities uh are they wormable or not and we created our own simulated world of these and this helps us evaluate are these assumptions realistic does

this correspond to what anybody is has said from an expert opinion and we can go back and refine that so I'm not going to read all of these points but this iterative process about well how bad can things get what does it take for an attacker to really attack a very high portion of the population led us to not only produce the quantitative output but the reasoning behind why this is relatively rare more rare than than some Cassandra's might be claiming cool so um almost wrapped up um we thought you know we've been speaking about Doom and Gloom so we thought we'd share what some of the output of our modeling is in terms of how bad can bad be so we can ask the

question of our model you know what are the characteristics of the most rare and broadly hitting events so here we're talking about stuff that we think it has a less than a one percent chance of happening next year you could also say that's less than a one in a hundred chance of happening next year so our model suggests that about 9.4 or greater firms are likely to experience an incident from a common cause this is sort of in the realm of our worst case and we think this is most likely to be driven by worms there's also a chance of supply chain attacks on very very large vendors and we think the uh the payload is very likely to be

wipers partly because of the the threat Act of motivation you know on the on the state-sponsored side but also because ransomware attacks typically have on the sort of business side for the threat actors they need to process a large number of of transactions whilst a wiper it's possible for you to just drop it and walk away um and over back onto Russell last point so I've been in this business for quite a while 2007 I presented to uh angel investor group incubator with a bright idea of a research agenda investment opportunity for quantifying information security risk and making it financially material to executives and I got blank stares and polite nods and then I invited myself back

again I said I want to give another presentation because I want to get something going and they said uh I think our calendar is full here um but I kept going and I know people in this room and people in the audience watching on YouTube have been on this journey for a similar time periods 2022 is a different time uh we wanted to have a whole bunch of corporate logos and organization logos of all the people participating in this world now some of the people here in this room uh I kind of feel like we're in a golden age of risk quantification and information security there are so many more participants now there are so many more good platforms

and initiatives academic conferences and resources so if anybody is interested in getting involved with this for a start be happy to talk with you uh if anybody's in it been in it for a while and you want to up your game or get more involved be happy to talk last but not least this event in the ground Truth uh track is evidence of how far this community has come in the past 15 or so years so I uh told my UK colleague Chris that the American audience was very active in asking questions and engaging in conversation so now is your opportunity to do this as we get to our microphone setup

hello Russ be close hello Russ hi I I presume that that uh slide at the beginning with your modeling risk has a little closer as a result of impact times probability was just a straw man argument that you threw out with the rest of your sides which didn't back up that approach or am I completely misreading what you guys have done here because it doesn't thankfully it doesn't seem like you've gone through that dark Road into multiplying apples to times oranges and you might want to educate everybody on exactly uh the how you are modeling risk there uh thank you for that question uh so if everybody understood it he he's saying we introduced one of the early slides risk

is likelihood times impact or severity and that seemed in contrast to the rest of things we're doing so in the early 2000s there was a lot of um I will say expert commentary driving people to do quantitative risk modeling using that simplistic formula alone and trying to get you to think about every single asset and every single vulnerability and applying that formula to every single one uh I was once at a conference where a high-level person at a bank told a war story of taking his team through such an exercise and after two weeks gave up in exhaustion so they had carte blanche to work on this until they got it done for some Board review and they gave up

the trouble with that is it makes sense at a conceptual level so at a very broad definitional level the definitions are consistent but how you get there is crucial and it's not the same you know our whole presentation about causal modeling and getting to the probability of certain events happening in their conditional severity is we think we're arguing is a viable supportable path to getting to that uh risk definition so closer closer so um there are lots of domains that are modeling risk uh as you mentioned earlier like finance and floods and et cetera et cetera um are there similar uh approaches to causal modeling in all of those domains and is there a common methodology

between them yeah I can try and talk to that a bit um so I think there are certainly overlaps um in that the process of trying to create a broad range or a broad catalog of scenarios of bad things that could happen and Associated likelihoods I think is common across the board we described you know our methodology here for something like a hurricane model what they do is they have a model that spawns low pressure systems inside of the North Atlantic and kind of track tracks how they you know intensify for earthquakes you know they have um models on you know for different fault probabilities and these sorts of things so there's I think at a very high level

you know this notion of having a catalog of bad things that can happen and then you combine that with the economy or an insurance portfolio to see uh how it comes out however of course you know in the context of cyber what is very different is where you've got the human decision-making angle right we often talk about how you know you could build the most perfect synthetic world of vulnerabilities and you know software Etc but your assumptions about how threat actors make decisions ultimately is going to be what's driving you know the likelihood of really bad things happening so I think there are certainly areas of commonality but having a man-made risk I think adds an additional

level of uncertainty that is perhaps not in place for those natural catastrophe models anything else yeah uh terrorism risk modeling which is something that RMS does is the closest analog uh and they use similar approaches not identical but similar

hi there I work for a very small precious startup and I would I'm in QA I'm the closest thing they have to security how do I get my very small very new to the startup World very literally young engineering team thinking about these sort of War games these possible things that could happen how do we react to them without scaring the pants off of them that is a courageous huge incredibly important question and Challenge and anybody in this room who claims they know the answer to that or there's a foolproof path I think you're kidding yourself so everybody who Encounters this I mean this is one of the most challenging things in the whole security world and

Quant risk so I'm I'm going to give a try to answer but so it's such a big question it deserves more time I invite you to join the Society of information risk analysts because I assume that you are interested in a quantitative risk approach to this question as opposed to hand waving or framework bashing or please no jargon hurling approaches okay Society of information risk analysts is a community of people pretty Grassroots and there's lots of people and resources who can you can connect to that will help break that down and give you tools to make progress on it awesome thank you uh thank you for the presentation so um you were mostly it looks like looking

at um sort of a large population from the perspective that insurers and things like that uh I'm wondering if you've uh had any experience with how well this kind of causal modeling uh would or wouldn't work for a company trying to predict uh catastrophic events happening specifically to them uh for instance to decide whether or not to invest in some sort of barrier to like or how much how much they should be paying for insurance things like that very good question very forward-looking because that's the next logical well I won't say the next it is a logical progression for firms like ourselves and models like this uh I will say the transition between population perspective and a firm specific or even

a value chain which is sort of an in-between thing is tricky from a technical an Evidence how much granularity you notice in here we abstract attacks into two two phases initial access and everything else firms care a lot about that everything else that's the whole elevation of privilege and horizontal movement and partitioning your networks and so on I would be very interested in talking to anybody who's interested in that path uh official announcement RMS has been purchased by Moody's Analytics Moody's Analytics is a famous publicly held company modeling risk across a broad portfolio and I know that they're interested in this topic I will definitely come and talk to you about that afterwards thank you

hello thanks for the talk it was really interesting um you talked a lot about external threat actors and I was just wondering how do you quantify Insider attacks um the likelihood and impact given that the initial access is kind of taken care of so the question is about Insider attacks as threat actors uh our current model does not include Insider attacks uh maybe Chris can comment it's my understanding that Insider attacks are not normally covered as part of normal cyber insurance uh I mean that may that may well varied from company to company but so what we described just throughout this presentation is our framework for trying to quantify catastrophic cyber risk right so it's our assessment that

although inside a threat is definitely a big thing the likelihood of you know armies of people simultaneously deciding to you know take up the mantle against their corporate oppressors uh is extremely unlikely and that you know the source of attacks that we're trying to model here around big supply chain attacks worms from malicious external threat groups are likely to be the drivers of things that's not to say that we don't consider internal uh uh Insider threat implicitly I mentioned earlier on you know we have the this kind of notion of a statistical model where you have the likel of certain things happening we actually model the attritional risk uh through that sort of method so we have quite a lot of data to

see that you know these sorts of attacks do happen so inside of our model we sort of are implicitly capturing this but we're not you know Express you know we're not calling it out uh so so in principle we could add insiders as a new attack category and I don't think really anything in our framework would fundamentally change we'd have to add ttps that are appropriate to insiders and so forth and but as Chris said it probably wouldn't affect our overall model output now I will say uh my own personal opinion to getting back to this gentleman's question about Enterprise level I think potentially one of the most catastrophic types of attacks for an Enterprise would be executive level

Insider attack Executives who have enough technical Savvy to go back and change your financial systems right so think of Worldcom type scandals and so forth uh if I were an Enterprise risk manager I would have executive or privileged access that causes major financial fraud in your threat model hi Steph I'm from STX I have a question so we have create models in the past like for hurricanes stock strong wind and everything with lots of experience this is all natural science you can do controlled experiments and so now with cyber we are more in social science the environment changes permanently now we know quoting Nasim Talib the bigger the risk or the bigger the event the less we have a clue

where or how do you draw the boundary between stuff we can model and stuff we fundamentally cannot model uh so I'll give us both a chance to answer this Chris can think about it while I'm babbling on uh you've raised a crucial point so I mentioned at the start that my PhD work was in computational social science social science is a very broad loose not tightly integrated set of theories and methods compared to physics for example or chemistry but there is a lot of things to draw on so I mentioned in the model of threat actor decision making that draws upon consumer Choice firm investment so-called maximization of subjective expected utility and so forth so we can

apply these especially in narrow contexts Define a context of certain human behavior for example peer influence to what extent do threat actors think through their own Roi versus they just follow the crowd or they've they follow on the habits of what somebody else does or they acquire a tool set from the dark web and start trying it until it doesn't work and they do something else so there's a fairly good body of knowledge and methods and Theory to inform that now the trick is as you said before and this is a matter of expert professional Judgment at what level do we apply this how granular do we get is it better just to treat this

as a you know random dice throwing because we don't know a lot of details and there's so much uncertainty or can we treat it as you know a rational decision-making process where the agents have a lot of information and they might even make strategic choices in a game theory sense I'm going to trick my opponent and I'm going to fake this and do that so uh it's my answer Chris yeah yeah I guess my I'll just give some maybe a slightly broader comments which I think that ultimately however good your model is there needs to be an acknowledgment I think of the uncertainty in this field is likely to always be greater than in the context

of a hurricane for example for many reasons as we mentioned you know you've got the whole human decision-making side of things the threat landscape changing far quicker than you know than climate change is changing hurricanes for example and then you've also got I think the observability challenge you know with a hurricane you know you can use satellite imagery and you know ground anemometers to measure the wind speed to make sure that your model is producing realistic output with cyber it's extremely difficult for us to at scale you know scan the internet to see which companies actually had a ransomware strained successfully deployed on some of their servers right so I think it doesn't mean that this that

uncertainty does not mean in my view that this Pursuit is not worth doing because I think certainly from the insurance perspective as soon as you insure a company you need to start thinking about this sort of thing and either your sort of finger in the air or you're trying to use techniques like we're trying to use but um yeah I think I would endorse what Russell was saying but I think also yeah there needs to be an acknowledgment of a greater uncertainty in this sphere thank you uh hopefully just a quick one which was one of your points was around the lack of incident information one thing was around the lack of incident information you mentioned that internet information

is hard to come by I was just wondering since you work so closely with insurers um you know their clients of yours they they've sort of received a lot of this modeling wouldn't they have a lot of incident information from where they're paid out and have you been able to leverage that so the question is do we get information from insurance from their claims information and others that we make use of so it depends um I think for insurance companies this is my personal take not that of RMS uh is that you know insurance companies you know they view their claims data as being a form of intellectual property especially if you're a insurance company that ensures

many many companies that data on the frequency and the types of events and all these sorts of things is really really valuable to you certain insurance companies will view that actually they'd rather not share that information with vendors such as ourselves because they don't want their competitors to benefit from that intelligence whilst other kinds of companies um maybe take a longer view that they want to actually having robust quantitative methods is necessary for a cyber insurance for a healthy cyber insurance industry more broadly so uh yeah we get we do have some claims data from some of our clients but we'd always like more but we understand that actually it's a business decision making a business decision as

well as uh other kinds of things there is a informal process where some of that information gets exchanged implicitly it involves them running the model and looking at their own data you know wait a minute we don't what you're saying this and we're we're saying that so whenever we get bones of contention irritation we're too high compared to what they think are too low that sort of informs us indirectly about not the details but the overall summary of where they are in the risk curve I think as well actually it's also worth noting that some insurance companies do release claims statistics reports which are also pretty you know interesting where they you know they're not

releasing the names of the companies that were hit or anything but they're saying that you know for companies with less than five billion dollars of Revenue you know two percent of them in our portfolio took a claim or something like that so I think yeah data is being shared but not as openly as maybe some of us would like yeah thank you

all right I think that's a wrap thanks everybody [Applause]