Building Effective Vulnerability Risk Scoring Systems - Harri Renney

hi everyone thanks for coming to my talk yeah I'm Harry um I actually did my undergrad and PhD studies here at you so it's really nice to be back here uh but since then I've started working at um K uh we do a lot of like Consulting around cyber security we're also developing some uh you know like cyber Security Solutions particularly around vulnerabilities recently and um yeah we also release a lot of like open service and open resources which this is kind of like this talk kind of stems from so let's look at how how can we build like what's the approach to building like a risk scoring system um and how can we make you know like Empower people to do

that and follow this kind of approach that we've used so we start off of what why prioritize why do we need to if there's a vulnerability why can't we just go and remediate them all right let's get rid of all the vulnerabilities well once upon on a time in the early 2000s when there weren't so many vulnerabilities being published you might have been laughing right you might have been lucky to get a vulnerability on your laptop and you just go and remediate it and your work's done right but this is actually a graph showing per year how many new vulnerabilities are being published so these are all accumulating right well over 200,000 vulnerabilities are um registered as a like an official

vulnerability and you can see they're getting progressively higher and higher especially in the recent years so you know we're moving into you can't just do things manually anymore you can't just look at this little list and go and remediate them there needs to be that you know there a data driven world now a lot of these fields become data driven there's so much data you need technology and automated Solutions in this case prioritization to help you go about remediating what you need to in your organization or personally right um so a lot of these cyber security companies like they gather a lot of like uh you know evidence from from their clients and stuff like and so mend the

io for example they've got this number 133% so I think they got done that kind of review over a thousand companies and the you know on average a company can only remediate 13% of the vulnerabilities they have right and I've looked at there's um couple of other like sources for that and it's around 10% like there's a lot of vulnerabilities you need to prioritize which ones are the riskiest so you can do the 13 top perent you know on average 13% of the top riskiest ones right that are going to actually you know likely affect your organization and yeah what are your options so there's loads of amazing great proprietary paid for Solutions uh OB you got like mandant zerox all these

like big companies providing really robust powerful uh you know cyber security solution but in particular what we're looking at is vulnerability like management and Remediation tools but as you can imagine you got pay for a lot of these it could be quite expensive if you're like a plus billion Revenue dollar company you can afford that that's great if you're a small to medium Enterprise or like a community-driven project that's an issue and then what I would say is an even bigger issue is that a lot of these are pretty much black boxes it's like you got to trust them that they're going to do a good job they're a big company that everyone says works well and that's it right

like they don't i' I've purposely put like orus I know orus do publish like a white paper and have gone through their vulnerability score and thing but there's no explicit explicit definition of how they're doing it in there you won't go you won't be able to go and reproduce What they've done right so but these these companies they're all depending on on a lot of there's a lot of openly available standards and data out there now that there they're drawing from and anyone can draw from it's openly available it's all out there so for these small to medium Enterprises or community-driven projects want to build these Solutions you can tap into that and we want to you

know in this talk I want to show you how you can like bring that all together into one kind of concise calculation that's transparent that you can explain to anyone and share let's not you know stuck behind like a FIY black box or anything right is uh this is a really cool kind of graph of the growing you know number of vulnerabilities but also all these like Open Standards that have been being released with you know new iterations and versions uh I highly recommend you check out this guy on LinkedIn he came up with this this graph he's got loads of cool vulnerability visualizations um but the key takeaway I think here is um while Vil is

increasing the the open data available out there is maturing becoming uh yeah there's more information there's more data out there that you can utilize to to Really prioritize your risk score now and I think just point out a cve is basically a standard they created as early as 1999 and that's basically an ID tag they assigned to a vulnerability so ID tag so you can communicate coordinate with between people vulnerability is right and then you can assign all these kind of scores and data to it and information references to where they're used Etc so let's if we want to talk about defining a formula or risk calculation let's go back to what is risk uh so the

N Define it as essentially impact times likelihood so if this vulnerability was to be exploited what's your consequential impact um going to be experienced um and then likelihood so if you have a really severe vulnerability this never going to be exploited through a low likelihood then it's not very it's not going to be considered very risky right but and then vice versa so it's kind of this combination of these these two components like impact how bad is it going to be and then How likely is it actually going to even happen so we want to bring those kind of Concepts into our calculation so one of the Open Standards for severity scoring is called the the

cbss common vulnerability scoring system um this essentially like the vendors or like the expert or the vulnerability owner when this when the CBE this vulnerability gets like published it's kind of a um uh mostly a subjective scoring system so they come in and there for example if we look at this Bas metric group they come in and look at these impact metrics like confidentiality impact Integrity impact and they'll come in and kind of score it with like an ordinal system like num low high this kind of thing and it's it's CVSs has been uh maturing a lot it's gone through a lot of like T problems people have a lot of issues of how things are getting scored

earlier on B on I think literally last week version 4 came out and it's got a lot more than it used to in the sense that they're actually bringing in like threat metrics and steps so I would say going Beyond purely a severity metric but because of that modular way they kind not modular but like the way they break down this SC system you can always get into that impact kind of category there we'll see how how well CVSs matures into like a real risk score um because it's one thing to find in that standard but it's always dependent on these kind of manual subjective inputs a lot of the time so for example they're

not even adopting that into the the databases until next year apparently I heard so we'll see how that actually goes so the the principle still stands we probably still want to make our own risk calculations but out until that actually is a a great solution but yeah important bit is impact metrics are in there and we just talked that impacts a really important part of risk calculations right another really interesting one that I'm always super excited about is the epss so exploit prediction scoring system so this one's about likelihood so How likely is a vulnerability to be exploited really cool um oh so this Studies by the way roughly only 5% of vulnerabilities are known to be

exploited so that's quite useful like a lot of the time you can disregard a lot of vulnerabilities if they're basically have not going to be exploited so really focus on trying to find that 5% cuz remember you can remediate 30% of vulnerability so you can overlap that across that 5% that going to exploit it that's great so that's a really good score and it's actually based on AI model so so using gradient boosted tree model um they've gathered loads of training data so I think they approached a few different companies like Alien Vault and they they they have like honey pots on the internet so they can see what fun abilities are exploited and they provide that to the people who

behind this research they have this labeled data set they could see in like a 30-day window what vulnerabilities had been exploited and then they feed into their machine learning model um those are I think there's like dozens and dozens of parameters it's like you know the cbss score has fed into it but also like um you know what vendor is affected how many references are connected to that that that vulnerability uh um and loads of other things so they feed all these different parameters into the machine learning model train it on the known exploited Ren abilities from these vendors um and it's become really really powerful especially so this is a rock graph I'm not going to

explain exactly what that means but top right is better it means get more true positives but less false uh false positives so you can see that especially for epss version 3 which came out I think this year or early last year really good Improvement on that basically accuracy you can call it like the accuracy of that model capturing true positives and not bringing back a l of the false negatives either but epss is all about exploitability is that likelihood component right so let's revisit our abstract definition of risk and we can start to think about well a risk score simply could just be these two right CVSs and these score I should say epss but yeah so we can do a calculation

like this okay but this is still abstract let's actually go to uh the real calculation in a moment but I just say there's other sources of likelihood right we got a machine learning model predic in and yes it's accurate right but why don't we go and look at what are the known exploits out there as well and bring that into our risk calculation so I think last year this repository of known exploited vun build came out Kev uh metas spit if some exploits in metas if you let go basically scrape that out of their repository you can be pretty sure that that's exploitable vulnerability uh Google's Project zero day full of zero days um there's also

exploit DB basically a database of people uh uploading like proof of concept code for exploiting vulnerabilities so if you go and like you know scrape all these sources you can actually find V ability has an exploit no exploit you're not even using a machine learning model anymore this is a pretty like sure way of knowing something's being exploited in the wild uh you also got GitHub you can scrap GitHub for proof of concept code so although that's not um you know guarantee that's being exploited you know the potential is there that someone's put a proof of concept code out for that uh social media really interesting one so you know X Twitter Reddit they're pretty good sources of like communities

of people talking about vulnerabilities um they have apis you can gather basically search on CVS every day and bring out all the the traffic from there so obviously a second an official source so uh pick up you know it can pick up nonsense so you want to be able to process this and filter this it's not just go and get CVS for me you got a bit of work to do there on the programming side but really useful for picking up any early warning for like zero days so me so there's this with the cve they in the Western World they get published to the nvd the national vulnerability database but sometimes there's a lag

latency before they pick up on that and go and publish it so social media really useful for picking up on that and then you can kind of get a an idea of like how impactful or you know if it's being exploited by considering like what what's a kind of making an estimate what what kind of audience size have you got there so if uh you know some uh if the NBD like on Twitter like talking about this uh CBE and they have you know 10 million followers that's a lot of audience like people are going to be seeing and being exposed to that CV you can like you can rate that higher right in your risk

calculation so that's a lot of talk about data sources we can tap into let's look at building a risk formula okay so I haven't find any of these on here but I'm working on a paper where there's a full definition for this but let's just let's just call this it's a weighted uh calculation right we're going to take all these data points in like CVSs epss number of exploits social media audience let's bring them in let's give them a weight so all the weights should add up to one so when you multiply each of these and add them together you want to have that normalized rage you want your score to come out between 0 one doesn't

have to be 0 to one I prefer that could be 0 to 10 you would just have your weight side up to 10 right so let's look at this in a nice way on the presentation let's just imagine we find a cve with these components with this you know CVS score with this EPS of this one loan exploit and say 150,000 you know people exposed on social media to to this CD okay so let's just pop them through like a weighted calculation with these weights and we've just got 15,000 2221 okay it's useless because we're not getting the right normalized range for all our all our metrics right all our data sources it's just heavily favored

Twitter like it's nothing else matters okay that wasn't that didn't work so what's the what what do we need to do as a solution to get around that problem is basically just mapping um mapping these ranges to a sensor 0 to one range to go through the calculation so they are just functions but there's a few like in animation and like game development using function is essentially mapping from uh like usually the time domain to like a physical like placement or something so when you throw a ball it's like track in that so you pick one of these and you'd be able to mimic physics without doing a full physics simulation this kind of thing um

I like so I found this you know on this website they have a bunch of different easing functions and you can kind of find something that's appropriate to map from your from one of your data sources to this so let's just go for an example use this way so an example I would use here social media audience so with social media audience you don't want to have if there's like 10 20 people talking about a CV or being you know exposed to CV from like a tiny Twitter account you don't want much impact as you start to get maybe to T of thousands or hundreds of thousands you want that because they're on the Y AIS this

show on here because I um actually mapped out with numbers that would help so yeah when you're in the thousands not too too much impact but then when you come up to like the hundreds of thousands you start getting some actual contribution to your risk formula coming out so 100,000 there only 0.175 and then on this on this on this curve I built 225 is actually like you know 0.66 when you get to a million then it's like okay you're up here you get diminishing returns anything over a million is basically almost one right so awesome we got way to map massive audience sizes and there's no upper limit on it who knows how many people on

Twitter could be exposed to CV this handles it perfectly well I say perfectly but it's a good solution I think for that me across and obviously you get kind of uh parameters to control the shape of your curve so if you want more impact a lower number of audience you can you can control that the shape of the curve uh just another example known exploits so I would go for this kind of exponential curve kind of like this find like that and that's if you've got a known exploit you probably want to just bump something up straight up to a really high uh risk value right so as soon as you have one you're you're just increasing the

risk comp contribution from know exploits into your formula a lot and then yeah okay if you got like two you're basically on the full impact and then you're again diminishing the turns because once you have no exploits that's that's it in the back right and again you can you got parameter like to control the shape of your curve if you want different values to come out of that okay sweet so build you can now pass through your values of different ranges different metrics or whatever you want to call them and you can bring them through these easing functions they're just functions and get out a normalized range so now everything when you pop it through your

weighted risk formula you got a value between Z and one so now everything only cve data you put through your calculation you're going to get between zero and one it's all normalized it's all comparable and you can build a prioritized list from that now so is it one score to rule them all have I just told you the the ultimate score I don't think so like I'm just I'm just telling you the approach I'm just saying you can use these wa is Formula these are a bunch of data sources you can you can use these easing functions these functions to map to normalize range in a in a shape or curve that makes sense um but obviously that's just

an approach right but you can you can change this up you can say I don't find the CVSs very useful for impact I don't trust subjective sources there's loads of Pap saying it's got a lot of flaws I'll swap that and I just use CTI C so that the number of times the CV comes up in um threat intelligence uh threat cyber threat intelligence reports let's say I want I think the social media is a bit vague let's use a you know natural language processing of the social media to get like some semantic value out of that and they use a score like that instead of the audience okay so you can just swap things out you can come up

with your own data sources you thought impact obviously you want to get impact likelihood in there but from there where you get those sources from it's up to you and how you map those just make sure you know you use the right mapping curve and then you put the through those right to come to that normalized range and then we haven't even got started and won't really have time to go into like local scoring context like I've just talked about CVS on a like generalizing this CV is 0.75 risk but is that important to your organization on your organization so what I'll call I'll say like when you have a vulnerability on your device let's call it a

vulnerability instance Pi um you have your cve in general it's this RIS score but when it actually comes on to one of your devices how impactful is actually going to be for your organization right you know your servers might have a really high technical impact you want to allocate to it but your CEO's business laptop is going to have a very high business business impact and these are you know the way I'm describing these are essentially their values you'd assign to each device your like kind of manually um so you want to feed this General risk score for runability into a calculation for your vulnerabil vulnerability instances so so yeah I can imagine it like this

you know we just did that risk calculation up here for a global risk calculation let's build never t for our local context for the ruil the instance calculation so we could just imagine putting that Global risk calculation to like another layer of a weighted um formula but then multiply or weight it and then add it along with the business impact and the technical impact

of okay so my laptop might just restart in a minute and then you're going to get a prioritized list of uh yeah and you know we can go about remediating the top ones on your prioritize list first right but doesn't have to stop there why not optimize your prioritization you know vulnerability risk exposure is one thing but what about your Personnel hours to get to them the cost you can use like a optimization algorithm like a gold programming if you can actually get estimates or real hard values for any of these like kind of metrics you can build out like real optimiz priority prioritize lists to um to to actually get a remediation plan that's the best like

you know bang for your buck like with the cost involved in the Personnel hours or time to remediation you get the most reduction and risk associated with those other goals that your company or your team might have so quickly uh some useful resources I'd like to just press on that come out for that you can use if you want to build your own risk formula uh cv3 so K is we have a get repository uh we uh routinely on weekly publish all the latest uh vulnerability information we have to this repository with the schema that looks like this um and I would say this has all the vulnerability information I've just covered in the slides that you could

build a you know risk formul yourself from this is just on GitHub you can go and get this data it's not live and dynamic but every week you'll be able to get a decently fresh risk calculation out of the data provided in here the company also uh K also have some free services online one's called CV Shield um so we you come on on here and this is it this is our like Ops dashboard for looking at trending you know on social media trending vulnerability with the tweets that you can go and look at but also like to shows you you know the score we built with that risk formula approach called the vcore um cbss all the private all

the kind of details provided on here you can come on there for free and then if you wanted to come and have a look at like more detailed breakdown of all the vulnerabilities it's it's again free you can sign up for account and go and search it's basically like a a database or search engine for vulnerabilities you can go and explore and yeah it's the end of the talk but I'm really interested to open it up to questions because um you know this is just like this is the approach we're using it' be really interested to know what other people think um yeah I was just really excited to hear what other people think about this approach

or if they've done anything similar or there's any recommendations this kind of thing so yeah if anyone yeah um really good talk loved it um how are you measuring the verality on social media is it just like the amount of followers an account has versus because obviously it's quite a complicated thing to measure accurately right morality rep I was just wondering what the methodology behind that part it was yeah so that social media is is kind of the max exposure so if an account so for vulnerability if an account tweets it we add all their followers their follow account that's on Twitter anyway for Reddit there's a there's a different process that's a little bit more

complicated because they have like Karma scores and um and a community has a number of followers you kind of have a breakdown of the percentage using that um yeah for for Twitter it is just adding all of the people have tweeted that vulnerability adding their their follows up OB L without duplications of the same account tweets twice you don't add it twice that's approach we use for the social media audence yeah it's kind of like a Max possible audience okay because obviously not every follower is going to see the C yeah yeah yeah I really enjoyed the talk to um I'm glad you got the bit of context written down what about the business context so it's quite a

difficult one to Al in my experience is lots of different factors um I showed on the slide few few point you got anything else for me the context is ranges from whether it's business critical function it's not personal information it's those other things beyond the technology it's the time yeah you got any other thoughts like what else uh not loads of thoughts I I I kind of leave it to that point where like you know I've show that approach and like you can reapply that to the local context with the weighted stuff right um so I think with unbid scoring it's obviously so important about the context is so important but it's so contextualized it's like like it's just different for

every organization I think and that's why I think the approach to building that out but I think it's so varied between what organization you go to what's important yeah yeah and it's not just so I've got the same vulnerability two or three places one it's not exposed to extreme it's not at business level it's at the the thing level so you know I might my team might not be worried about it but another team in my organization should not one score for all even within a business be the same score across the business oh I see what you mean yeah like yeah yeah I see what you mean like the people's like responsibilities when they company between each other

different as

well that's the trying even in the business that well starts up here it's not

where the other one I certainly yeah yeah another one we usually rate for the context side is like criticality system criticality might cover something like that um but going back to your point about like uh the position like on a network say of a device if it's not even on the internet we actually did do like in the company we've done some like experimentation with like a network simulations and that could be another component in the risk calculation right or this risk calculation could feed into that that's actually how we do it so we there's um really interesting project called yawning Titan uh that uses like agent based AI models to like attack a network and it's an open source project

so we used it and then use like this the risk calculations we come up with to say how like vulnerable these nodes so if you wanted to build out your organization's Network you put all your no and say what the connections are between them then like let loose this AI to this AI simulation to attack it and based on all the risk scores associated with each device like how like easily they could be compromised and exploited in that simulation to get like an insight into uh that kind of network configuration side of things where and the position of your organization how that affects the the likelihood it will get

exploited oh Sor yeah organiz Cur youest first all immediate stuff but what point do you you say now we start this or do you think that's the first do yeah I think like in the past would have been difficult to but I think we're in like a space now where it's quite easy to to do to to prioritize it and start remediating with like a with a plan or informed reason to go and remediate something over and another for small businesses obviously like they don't have a person who's even familiar with this space so it is difficult to get into but I think that's why I don't really want to press that this is a

solution for small to medium priz Enterprises this is like also useful for um for like a community driven effort like this is like Empower everyone to be able to build something but not everyone's going to be able to but if a community built out like a really robust scoring system that collected all this together and then all these small to medium prized Enterprises could tap into that instead of paid solution they wouldn't be able to justify um previously because they you know they don't have the revenue to then so I I don't think yeah know like what you're saying like right now what's this what can we do maybe not there just yet that everyone can tap into that but I think

there's all the data is there I think it's just like a real accessible you know free solution that people can tap into is really important and that's when they'll be able to I think really instantly prioritize based on

that any more questions you har thank you everyone