Understanding And Using Impact So You Know What Vulnerabilities To Fix First - Chris Madden

so thanks folks for attending we're going to talk about understanding and using impact so you know what volum is to fix first AKA using AI language models for cyber security like a pro uh without the hype all right going to use these things for real so I'm Chris from Yahoo paranoids product security team and I need your help I have a security hypothesis the real reason I'm here I want to prove it I need a smart friendly audience to help me show a hands are you an Ernie or a Bert Ernie is the fun irresponsible one and Bert was the serious responsible one who's a Bert all right I'd say about 70% of the audence who's an

Ernie all right delusional thank you we'll come back to that later so for this talk another question how many people are here to learn about state-of-the-art AI what's you know is that the primary interest in this talk versus impact how many people WR to talk about the impact and risk-based prioritization cool an ai ai stadi tools and how to use them that's why I like bsize dou it's a smart audience doesn't buy into the hype so we're going to cover both cool I was worried it was all about AI the interest so we're covered so contributions the industry um so as part of this talk um I developed a taxonomy for impact which we'll look at

a data set of all published cbes to date labeled by impact so if you're familiar with CVS there are the vulnerabilities uh associated with public um let's say records public open source thir party software and there's about 230,000 of those so we've extracted the impact from those using Ai and I'll show you how we did that and then a model that can um determine the impact uh automatically from text description so the data set is published on hogging face just recently you'll see it there on the link um that's 230k cves with the impact extracted um spoiler alert this is not an accurate depiction of how it was done so risk is paracet and depends on

the impact of a vulnerability being exploited by a thread so most people would read that and thought okay I get it makes sense I didn't get that I couldn't find find a way that I could actually use that so it started my journey started with presentation at bsides last year on the top left left basically understanding the building blocks and subsequently going deep releasing a guide uh in March with some of the thought leaders in the industry and today on the talk we're going to talk about impact um and ultimately it was trying to understand impact but also me trying to understand For Real Ai and LM and how how I can use them as a practitioner and

later on the afternoon we'll talk about epss so user need so from LinkedIn somebody posted you know is there a uh machine readable version of the technical impact of a vulnerability and they give examples and shorter answer is there isn't which is strange because as a user that's probably the thing I care most about in general and in cyber security what's the impact to me there isn't a tax Hy uh another user view is you know they were trying to understand in their organization a word cloud for their impacts there is no um machine readable version of that so that's what we're going to do so the pitch our industry relies heavily on accurate timely cve information and CVS today are

manually enriched and the enrichment data is not as useful as it could be or should be so we can prioritize vulnerabilities by exploitation and that's a lot of what my talk in the afternoon will be about and that's basically what sisa and Gartner recommend separately we can prioritize by impact and that's what we're going to cover here and in order to facilitate that we're going to provide by the day set with the impacts extracted okay are you in would you invest show a hands I'll talk to you later so B landscape quick overview so we have cwes which are weaknesses the root mistake can lead to a vulnerability tracked by a common vulnerability enumeration which can be exploited by an

attacker and there's a taxonomy uh CAPIC compon attack patterns which can lead to an impact and we're going to focus on the impact all right so that's just the high LEL view how this fits together quiz cross-side scripting is a weakness attack pattern or impact so shout out the answer all together cross-side scripting is a oh come on shout it out nobody can see you you're on YouTube You're Going to see the back of your head cross side scripting is a o not getting a chorus of answers I'm getting a cacophony of different words let's try remote code execution is a denial service is a okay so cross- side scripting is an attack pattern there's the cic there is

a weakness associated with it but it's a weakness associated with the actual weakness is the improper neutralization um which results in cross-side scripting rce is an impact and the L service is an impact okay let's look at some examples and we're just going to Breeze through these noting the r Parts the impacts and the cwe just some examples log for Shell spring

shell another example you'll see that the weakness doesn't always relate to the impact and that's why we need a separate impact I like this one because there's a lot happening it's got multiple impacts and again the weakness doesn't really relate to that tell you much about it so if we look at CIS Kev CIS Kev is the the most relevant set of vulnerabilities it's the non exploited vulnerabilities and of that set there's about 1,100 there's a subset which is the most exploited of the exploited vulner ities and they release reports with International security agencies and info that they choose to report is yeah they report the CBE cwe but then they also have this thing called type which is

ultimately impact as well okay so let's look at existing definition and impacts and again it's just looking at what's out there so CVSs is what's used in our industry standard our industry for prioritizing and scoring um vulnerabilities and it uses confidentiality Integrity avail ability the problem is most of the stuff is high so you can't really use the prioritize because if everything is high then nothing is high Microsoft for their vulnerabilities they use um different labels and they relate to stride and there's an example but it's elevation privilege Etc so it's similar um to some of the things we saw actually in the highlighted text the red text from the cves but unrelated to confidentiality Integrity

availability cve details provides metadata on cves again we see on the right impact types there's five they're kind of similar to what Microsoft gives different and then have these other types again again it's just looking at what's out there the main point is similar but different not standard so n uh vulnerability ontology they talk about impact and impact type and the or impact method the main difference is the impact method is the means to an end and the impact is the actual you know reading data writing data that we're most worried about again is a research paper summarizing probabilities they give a similar but different list so there's lots of similar but different ways to

represent impact so let's look at

miter so miter are the folks that uh published the cwes and CAPIC so they know what they're talking about and basically what they're saying is hey look there's thousands of cwe weaknesses but there's only eight impacts so why don't we focus on those well that sounds like a great idea yeah why not well the problem is there's no data where do you get the data there's no as we saw standard identifier for the data so it's a good idea it's hard to implement because there isn't the standard taxonomy um what they do have if you look at a cwe in there are these common consequences and they list out technical impacts of different types so example

for cwe 917 associated with log forell um the technical impact was example read application data in terms of impl impacting confidentiality so there are some technical impacts buried in the cwe and a cwe can be associated with one or more cve vulnerabilities so the data is not directly in vulnerability but if we were to look at CAPIC uh within the spec it lists out these on the bottom left technical impacts and if we were to map those extract all the data from the cwes and Associate them with their scope we get this uh heat map or Matrix so we can see for example bypass protection mechanism is primarily associated with Access Control we can see that read

application data about Midway down is primarily associated with confidentiality and so on so forth so taking that then what we can do we can create a taxonomy and it looks as simple as this so we split it up between logical impact and impact method the impact as we saw from n bology is the end result and the method is how that's achieved and the main things we care about are reading or application data or modifying it or Deli a service of our application and then there's various methods to achieve that like promote code execution um Etc or bypassing access controls so that's a very simple taxonomy and if we take our example where we had a lot happening the reds

are the impacts and we can map those to the taxonomy so the map so whether it's bypass protection mechanism maps to bypass protection mechanism okay so now we're getting the a clearer picture um and so from the CV description we can extract out the impacts and they do map to our taxonomy which is good and looking at all the different standards that we showed whether it was CVSs Microsoft cve details all we're showing here is they all fit under this impact taxonomy which is good so they're all different they call them different things uh but ultimately they can all map to this

taxonomy drilling down we can apply the cwe and CAPIC entries there's IDs for each of these so whether it's example um cross-side scripting you know it rolls up to code injection uh which rolls up to unauthorized code command which rolls up to uh execution so we can take all of the cwes and the CICS and map them to our taxonomy as well so that's good because there's you know over a th cwes and there's whatever approximately 500 capex and they do fit under this impact taxonomy moreover within the CP schema there support for impact but it's not really used we see it there Midway down um and in the example that they provide they actually use the attack ID CAPIC

which is a bit strange an attack pattern been used as an impact so most impacts can be represented by an attack pattern but we saw things like remote code execution that can be so either way there's a place within cve for impacts if we can extract them okay so that's all cool we have a taxonomy but we've got 230k published cves how does our taxonomy fit with reality so what we want we are going to build a model that can automatically extract this information these technical impacts because there isn't a machine readable version of these and we do want them and the problem type is multi-label classification in other words a vulnerability CP can have one more

impacts as we saw just here you know there's five so multi-labels class imbalance meaning that some vulnerabilities example denil a service shows up a whole lot more than ssrf as an example it's unstructured text um it's you know people writing text with typos and errors and all that and it's a domains specific language cyber security which is very different than medicine or Horticulture as an example the constraints we Have No Label data set uh we're doing this for the first time so normally when you look at ml you know tutorials it starts with download the data set uh problem here is we don't have one uh limited human time um this was done as a background task to my well

to the guide which are published in March with other folks which is a background task to my real job and limited GPU budget 50 or dollars uh in contrast the budget to train gbt 4 was $100 million so the challenge we have these 23k public cves that's about 10 million words 66 million characters and the longest description is like 4K characters so who's with me show our hands okay on your seats you'll find 20 sheets of paper with 50 cves if you can start labeling now you got 20 minutes no we won't be doing that so we're going to start by learning MLA aai and finish with a conference presentation that's how we do it and the approach is

um so understanding the problem what we're trying to do small verifiable steps and automate as much as possible simple as that and this is pretty much lean ey okay approach to things uh it isn't a waterfall model it's very much unstructured as we do one thing we learn and we go back back and we learn some more so does reality fix fit with our taxonomy so the traditional Workhorse for natural language processing is Spacey and this is typically what it would look like and the normal method would be phrase matching redx or fuzzy matching okay uh there are integration with language models as well but we're going to use machine learning and we're going to call a

friend and that friend is on hugging face and and so with hugging face we can invoke models to our browser so looking at the URL at the bottom seport we can invoke a model through our browser um to example guess the Mas word and that's relevant because models are B models are trained using mass language modeling and next sentence prediction that's ultimately how they're trained and we can use it to extract the entities whether it's attack patterns uh Etc and impacts that's named entity recognition there's different models for different plac things so we're going to rely here on the smart audience in front of me so for the first one what's the blank think about it I'll say when and I want

everyone to shout out what's the missing blank for the first one 1 2 three shout second one out of bounds axis any others any other takers what about memory I think we're in trouble folks okay let's ask some of bir's friends so we can look at um non-security language models again this is what we would get if we just used our browser and we can see that these language models are not so sure their confidence is not so high they select different words um but the confidence is generally low so they're not cyber security trained so they don't give good answers if we look at security models and I picked a few um code the audience said

code so let's assume it's code is the right answer the first one and the second one there's a mix I deliberately picked one that's ambiguous could be read could be right folks said memory access um these models yep they came up with access as well and memory there you go um so these are the right answers so the models can guess the blank word much better if they're trained in cyber security that's the key takeaway here um the other key takeaway related to my opening question are you an N or a birth I detected a bias in generative models you'll note smiling not smiling okay so let's meet bir's friends so these are different um

capabilities that we have with machine learning we have embeddings which are basically numeric representation of words we can compare those numbers to see how similar words or sentences are we have topic modeling which allows us to get a good understanding of what data is there and when we're looking at 230k CVS we want to get a good understanding of what is actually in the descriptions um named entity recognition we saw an example we want to extract out certain things like impacts um then we want to create a bootstrap classifier as in something to help us classify uh being impacts and then we want to improve that and once we do all that we unlock the next level

which is ultimately where we want to get to a fulls scale model that can automatically extract out the impacts and classify them so we got a level up unlock on that one and shown is a very high level view of precision versus recall Precision being how many of the uh retrieved items are relevant versus recall how many relevant items are retrieved and that's is important there's no Perfect Tool and it's what was the trade-off between those two things for any tool so why birth embeddings well we saw earlier that um security models you know perform a whole lot better with security so similar with embeddings numerical representations of words um they give they have let's say

that cyber security context so any of the models that we use we're going to use you know words or numbers with cyber security edings and there's one already a attack part uh available in hugy face so we're going to use use that somebody's built that already and what that looks like is when we look at example set of cves we see the different things mapped out into uh two-dimensional space these Vector Bings are like 500 700 Dimensions which humans struggle to understand but map to two- dimensional space it's a lot easier and so we can see related things like um on the right we have path traversal and directory traversal are very close together and so um that's basically you

know there's a semantic understanding of how things relate based on the embedding so let's look at topic modeling so when we're approaching a data set for the first time we understand what's in it what are the topics and you know 230k CVS is a lot of text try and understand and topic modeling allows us to do that and it's unsupervised meaning you give it a chunk of text and it figures out what's the relevant topics or things um so via diagram you give it an unsorted list and you know without any pre processing of the data it clusters them based on their semantic similarity which is great when you're trying to figure out well what is in this massive

data set it's very simple these are the main lines of code uh as I said you know we're trying to understand these things so we can use them and this is how you use them uh we use attack part for our cybercity embeddings and we just call bir topic for the topic modeling very simple few lines of code most of the things I talk about here can be done un less 10 lines of code okay they do things like clustering so it understands things are related like at the bottom we see that stack Overflow and Heap overflow ultimately chain up to buffer overflow so we could collapse those things together and it's important to understand that we didn't tell the

model anything we just gave it a chunk of all the CVS and said go figure out the hierarchies and topics for me completely unsupervised which fits with our limited time and budget it does things like wordclouds which is pretty standard but it it as I say from taking all those descriptions you can create the the war clouds for you an example of a user need I work with Jay in epss special interest group uh Jay was trying to understand there's a relatively new cwe data set called 1400 because it has 1400 entries and they're massively complex there's a lot of you know dependencies and relationships between them very hard to understand um so I said hey I've looked

at this and i' recommend topic modeling so I took the CSV file of 1400 entries and Rand the topic model and what it shows is that on the right is the let's say categories that come with the 1400 view but the topic model was able to figure those out I I overlaid the the bold black text from the categories on the right and you'll see they're very similar so again we took a file did nothing with it gave it to the model and said go figure this out what's the relevant info grouping it together and that's the beauty of topic modeling okay and the value isn't that the headings were the same it's more that you can

interact you can zoom in and interact related things so get a very good understanding so now we can't really talk about AI without talking about chat GPT so I'll talk briefly about this different things we can do with the llm we can use it in a zero shot fashion as in ask it questions um we can create synthetic data we can label data and we can validate label data so let's look at a few examples so here I've prompted gp4 saying hey here's an example of vulnerability here's the labels that are available tell me with a probability which ones you think it is so when people talk about language mods they talk about hallucinations most people

don't understand that models know how confident they are by Design so if you ask the model hey how confident are you with the answer that will largely reduce the you know hallucinations in other words you know how confident it is it'll tell you so here we can ask how confident are you with the uh answers okay so that's cool it's classified the last command injection wasn't so sure about so here we ask GPT to now create a prompt so we say create a prompt with a few examples to be fed to an AI so we're not asking it to classify something we're saying create a prompt to classify something for Ani and it does and then we take that prompt and we

feed it back into chat GPT and it performs better as in now it's more certain that command injection is zero as an example and all we did was asked the B to create a prompt and give it back to it so we gave it a few examples and it learned in context learning we can validate label data sets so the struggle as I said at the start is there isn't a label data set I looked around there was one from 2018 three language models and I said hey please validate this data set for me um I took some copy pasted some examples and said you know tell me how confident you are and whether you agree

or disagree and then I just SK the whole file of 500 entries and said right tell me which ones you disagree with and we see the first one it's correct the others it made a good point um but of those 500 it came back with five that's pretty good and it found an error and this is training data as in this is the data that was used to train the model back in 2018 so it's very important and within minutes you know a GPT can detector so let's look at named entity recognition uh this is where we saw that we can use a model to pull out the things we're interested in and this could be you know song title artist it

could be anything any genre or topic or field and they do exist on hugging face for cyber security so an example on the bottom left we can see we give it a uh vulnerability description and we say hey uh well it's trained to actually only pull out vulnerability um the observe data and ATT attack pattern so that the model that's all it does and understands okay it's fixed and that's all it can do again you see 10 lines of code how we invoke it and this is similar to invoking it via the browser and uh it comes back with you know what it thinks is thread actor attack pattern so name entity recognition um useful and this could be

as say any entities at all cyber security or not depending on what trying to do there's models for everything so nor is cool so what I found though was that the content that came back from that previous cyber security model wasn't great it was two for both I wanted something short and sweet and succinct and I came across gliner and originally I started playing with it it was the medium version and it was kind of one of many things I tried it was kind like not really doing it and then I tried the large model and I was blown away I like whoa what magic is this uh it was able to extract out the impacts just like

this so give it some textt give it the labels I wanted to extract and this is a general purpose meaning if I give it something content about music it could and I told her pick out the song music whatever artist it could do that so it's general purpose named entity recognition I could tell it pick out whatever from whatever text so it's general purpose state-ofthe-art um it's pretty cool and I was as I say blown away by this what magic is this I thought to myself uh and only 10 lines of code again so super powerful and yeah this was one of the workhorses for what I wanted to do the other code was wrapping this up feeding it 230k cves

processing in in 1K chunks and you know that was the code you're talking about 20 lines of code there was a bit more to it than that but this is the Workhorse so this is kind of what we get given a CV description it pulls out the things I want SQL injection SS RF Etc that's pretty cool again I emphasize general purpose no pre-training not trained on cyber security specifically one of the 11 data sets is programming related uh and it outperforms the older cyber security named entity recognition tools super powerful tool super easy to use uh and so overall and I know I'm mixing my mopit metaphors here sorry about that the Swedish Chef is one of my

favorite characters in my youth um but how we do all this then the recipe is this using gliner um to extract out the impacts and then Spacey ner uh to augment that in some cases where it says Deni a service it normally has brackets whatever crash or memory consumption whatever I wanted that part in Brackets as well and that's really it then per topic we can use topic model and remove the outliers okay so using the tools that I just described in you know functionally you're talking about 10 lines of code for each one um is the core code uh that's how you get to extract out the impacts from 230,000 cves simple as that so some of the interesting things

it found um there there's a mad stuff when you look at what's in CV descriptions everything from forign languages uh the first one suddenly switches to German for whatever reason um you get typos um you get things I didn't know I didn't know hair hunting was a thing it is um Android specific but it's able to extract out things that you know if you were using redx or traditional technology where you have to know what you're looking for um you know this pulls out new things in other words it can support concept drift new Concepts it can extract them out which is pretty cool and so we have ones related to the last one is related to cryptocurrency

where just Financial damage you know ultimately it's just software and in this case it's currency and financial damage so if we look at the distribution we see that um there's a relatively small set if we look at the top 100 key phrases the things that came out they account for uh about 50% of of everything in the cves you know so you get this peretto type

distribution some things that were not um extracted cuz sometimes there's nothing to extract or very hard to understand you know what's the problem you know you get things like the first one what's the impact well it doesn't say nothing to extract and so you get a lot of those so I'm going through those at the moment figuring out well and even let's say the second last one um people might yeah people may struggle to understand what is the impact there what's the problem pastors were hash CH 256 what's wrong with that okay so let's look at set fit then so we have now extracted out the impacts from our CV data um and we want to be able to

classify them um we don't have that much data and we want to build an AI model to assist us in labeling the data so we can use set fit so what's set fit so set fit is good at fuse shot learning it's it excels when you have limited data so here we're going to use it with our limited data to go from we went from kind of no data to extracting out the impacts now we need to label them um we can use this model because it works well with limited data as a classifier kind of like this bootstrap classifier before we get on to our bigger classifier and even if we ask chat GPT which is better for our you

know purpose um chat GPT will tell us for this specific purpose you know few shart learning limit examples set will generally outperform it people have this thing bigger models work better that's not the case um ber excels because it it's bir directional it under it gets context and meaning from words because it understands words before and after that's the bidirectional bit versus um gpts so kind of some simple metrics or differences it'll work with it'll work well with eight labeled examples so example if we said Den a service I eight different examples I could train a model with eight so it's eight versus example 800 that's a big difference again the code 10 lines of code and you can train

the model in you know seconds rer than hours again you go from nothing to something with minimal code and time and we now have a bootstrap classified that can help us um you know an AI assistant if you like for classification so lion is a recent addition came in January and that generally just improves performance not by changing the model but by using the text input uh it's very clever how it does that I won't go into the details but it basically how set fit and log on work is that there's information by knowing that a given description is a class the information is also that it's not in this other class so you get this

contrast and you can compare things and work out the difference you're basically learning by the difference of things not just the thing itself um so this is pretty cool and it extends how many samples and the benefit uh of set fit set fit is typically limit to about 50 examples in other words it either stops improving or deteriorates if you have too many samples which is unlike bigger models so the verp model implementation is this is all now once we have a label data set this is all standard tutorial stuff all right um not much to say here we ultimately want to build a model for classifying and extracting out as in named entity recognition extracting out

the impacts and as I said standard tutorial stuff at this point the only hard problem is dealing with the class imbalance and that's a standard textbook thing as well so as we said some of the impacts like denial a service and orce will feature a whole lot more than ssrm as an example and that does impact um how we train the model if we want to get good results again all standard textbook stuff at this point and the reason we're using a bigger model is it can work better with more data and give more context so it's only at the very end once we've kind of started with no data um use topic modeling to understand well

what is in all this data using named entity recognition gliner to extract out the impacts using set fit then to start with limited sample of data to classify them um and then build up or classify data set now we have a a data set that's classified and we actually train a model we've solve the hard problem where's the label data set and then we're into standard textbook stuff with this point so to recap our journey um we looked around at what was out there for impact and merer says hey why don't you use impact to prioritize which is great but there's no uh standard taxonomy and there's no machine readable version of the impact so we've

heavily relied on what's in miter cwes to build our taxonomy and it's very simple and we saw that we can map other taxonomies to that and C CW's and CICS that's pretty cool and then we validated that taxonomy against reality we showed example where cross side scripting would roll up in terms of CWS and CAPIC up to that and in general all of those you know hundreds or thousands of CWS and CICS can roll up to our taxonomy which is pretty cool and as I said the descriptions do include the impact whether it's rce or denial service or whatever um but there's no machine readable vers person of those uh we looked at Bert's friends to

help us and Bert was the standard technology that we used and we used it in different ways from trying to understand the data to classifying the data to building our final models so the takeaways the hypothesis going back to the opening point was that security people tend to be more burs than Ernies right and uh I think we're we're there on that one we can conclude satisfactory with the numbers yes uh I'm an introvert BT myself so uh yeah don't feel too bad okay so the other takeaways um en reachability data on impact and exploitation allows for Effective granular risk based priorization um with 230k published cves with approximately 100 new cdes being published every day it's very hard to

keep on top of those and we need to prioritize in the talk later on I'll talk about how we can prioritize by exploitation 5% of CVS approximately are exploited let's prioritize those and now we talk about impact so it's a separate independent prioritization so the key secret I think to get on top of all these cves is risk based prioritization and here we showed how to do it with impact so the first steps to doing all this is one Define a taxonomy like what is the taxonomy that machine readable format um that humans can understand okay there shouldn't be 1,400 things it should be easy for humans to understand like we showed you know eight or 10 for

our taxonomy and we can roll things up to that and also make it easy and automated to extract this information we can encourage people you know say hey you should you know put impact in your cves but that's great but unless we make it easy people won't natur so let's make it easy and automatic to do that extract that info and get it into the cdes okay and to the gentleman who was asking about Ai and all the hype did you find Value in any of this how much of this was hype and how much of this do you think was useful

yeah so good points false positives basically so that's why we use traditional methods and we can count if if there's one of a data set of 200k and one comes out and doesn't fit with all the others is probably an outlayer okay and that's how we can deal with false poses in the context of all the data but it's that's why yeah as I said ultimately all this is about balancing the Precision and recall there's no perfect solution okay so llm tools are very powerful they will be used a low a lot more and especially in cyber security so thank you to all the folks who've given me 40 minutes of your lives to the folks who built the tools to

Yahoo for cultivating such a rich environment for people to thrive and putting people first and for besides Nadine and crew for making this happen bides is my favorite conference that I I don't choose to talk at many conferences this is the one I choose to to talk at because uh you guys so thank you

all do we have time for questions five minutes for questions folks is there a mic or shout you might have to shout so for a new vulnerability what was today would your models be able to help predict UL so no that's where epss that's its purpose and we'll talk about epss in the afternoon the exploit prediction scoring system it gives a probability between 0 and one to say here's the probability that this new vulnerability will be exploited once that vulnerability is assigned to cve it can then predict the probability so that's what I'll talk about in the afternoon [Music]

[Music] yep so yes I considered it we don't let's say haven't done it because we're this is kind of where we are but my viewpoint is our whole Dev seops pipeline end to end right right sh to B Bounty and how how to relate those things together um so yes what we're what my mental model is all the stages of the de se Ops Pipeline and how they can be used to inform each other so in let's say for exploitation we use bug Bounty if something shows up in our bug Bounty or inent response we prioritize that ter of exploitation so yes we've considered it no we haven't done it because we're not there yet but

absolutely yes we we'll do the correlations against payouts and impacts and see see if there's correlation and if there's value in that but absolutely yes one of the things we've achieved as well is getting all our data into a data lake so we can actually do this type of things and just let the machines figure out the associations and how we might benefit from them but yeah it's it's that's a really important point point that all this is is in an overall context um and trying to inform that context with the different tools is is important we're trying to like Yahoo delivers value to our customers through software through our devops Pipeline and that's what all this is about

[Music] ultim yes yes so so we had different versions of this this started off as a simple redx a few years ago we wanted to figure out rce and you know memory prop for overflows and then I guess 2 3 years ago I built a non llm model so it's better and it pulled out stuff so it allows us so wasn't on many classes of things as you say rcees and you know memory buffer overflows so we use that in production we are now now that we have let's say more classes and more impacts we're trying to figure out well how can we use those in book Bounty how can we use those in wherever and then we see well

you know what what's coming in in terms of publish CVS is it and that's what we do example with our book Bounty what are the things coming in is it you know is it cross-side scripting is it whatever all those things putting it together so we we basally you know expand that the capability that's been there for years in a you know much smaller scale

one more

question say say

again yeah so has the large language model made it easier for uh upper management in terms of cost and risk I would say no not yet um that's where exploitation is a whole lot more it's an easier sell because 5% of cves are exploited and we can say well if we only fix or let's say if we prioritize those first we can save this amount of money that's a much easier s with impact because there's a lot more things it's much harder to let's say get the granularity for prioritization so I haven't put together the business case for impact yet the business case for exploitation is there already that's an easier solve so we with all these things when you

build something in need to see how it plays out in your reality first and you know so we're yeah all right folks much appreciate it thank you very much for your time