GT - ZERO-RULES Alert Contextualizer & Correlator

hello everybody my name is asz um before we start I want to do it as interactive as possible so if you have questions please don't be shy raise your hand or just shout out the question right away and even before we start if everybody can kind of like just give me a quick poll so that I'm not talking to like different audience um anybody here did something in the sock before or like some blue team okay a lot okay that's fantastic the other people that are not in blue team can you just shout out um either like you're interested in like what kind of area like it's fine if you're a student and you're just like a generalist just

shout out generalist uh but if I can just get a quick poll like what are you into so I can kind of like cater to to your interest as well purple team purple Team all right more purple all right anything else all right okay wow this is a fantastic room none of the compliance folks none of the compliance gets me to test everything right all right any students any generalists okie dokie all right so a little bit of uh an agenda so I'm going to start with a quick who am I uh so that you know who the hell am I and uh and then I'll talk a little bit about the challenges in the industry as a

whole from security operations but since you're already practitioners I probably just going a Breeze into that I will probably put a lot of time into the analysis challenges like what is actually hard in our job today um as people in the sock and then I'll talk a little about attack but you guys are probably experts in this area so I don't know how much time is going to be spent there probably not that much and then we're going to talk about the investigations and the model of investigating that talks about the what the when the who and how can we automate things a little go for it that's a good one that was a good one okay

um and then we're going to talk a little about some models in machine learning I know everybody is all about machine learning I am a data scientist so I'll probably bore you with a lot of machine learning just as I get my mic set up and uh we'll talk a lot a little bit about examples and we'll talk more about questions as well all right I will not I will not jump into the screen okie dokie all right a little bit about me I'm a data scientist that works in the sock um I have a bunch of sock experience but I have more experience in the data science area um than than in the sock so you're

probably more experts in the sock side so if I say something that is kind of like huh this guy is not really covering everything don't be shy I would not get offended shout it out or raise your hand and be like by the way that's not really how we do it that's a better way to explain it and that would be fantastic because I want this to be as interactive as possible but I worked in the sock as a data scientist in Royal Bank of Canada want cyber defense haa and for Scout and that's about it all right a little bit of a general kind of statement in cyber security is it seems like things keep on

getting worse and I'm not sure why if you remember like 10 years back we thought like stuck net was bad nowadays I don't know how bad was that right compared to the stuff that happened um but it seems like as if attacks are getting better and even more stealthier than they used to be and harder to detect and it seems like we have more alerts than ever before hitting our sock and it seems like we also have a lot more tools so why are attacks getting more successful uh some people seem to say that it's probably just bloated and ineffective but some people are just saying that you know what systems just got less secure and if you think about

that are systems actually less secure I mean we have the fact that back then things were simpler and now things are like crazy complicated I mean even in ic things were super simple to to pull off stuck net for instance or Colonial pipeline or whatnot things were very different back then while Colonial pipeline was like a little bit easier than stuck net right um and that's because like the networks have more attack surface right so back then Stu net was completely air gapped to do anything there you had to circulate the virus for two years around planet Earth until someone by mere chance gets that virus on their USB and plugs it in now you don't really have to do a that

there's like smart grid there's like I don't know 5G um IC kind of mobile Edge cloud and whatnot there are so many attack uh surfaces and that means that we have a lot more attack vectors to pull off so anyways what that takes us to is everybody in the sock does not mind any kind of automation they actually welcome it more than anything else this is one of the few Industries where you go and you talk about Automation and people are like yes give me more please take my job I don't want to secure this job ever like please take it give me another job um and according to the sock survey that is pulled off by sanss and I don't know

where to you stand on Sans I mean I'm I'm kind of biased because I'm I'm on the gak Advisory Board and I paid them a lot of money so I feel like they're good but they do this report and they go out to all the people in the sock and they ask them what's the biggest problem that you think is out there it's a pretty good report if you want to read it it's called The Sock survey by Sans and it seems like most people just point to this idea of my investigators they take a lot of time and their job is pretty hard and their investigations are very inconsistent and all of that is to chase this dream of

correlating stuff like I just really want to understand what is relevant to a ticket without having to like jump into hoops and whatnot and that is coming at a time where we have all of the data that we need ideally in a single place either logically or technologically it might be the Sim it might whatever you want to call it but we have a bunch of alerts and events and logs throughout our organization ideally we have a good coverage but even if we don't we have a pretty decent coverage in most cases and we have all of that in one place and the way we deal with it is we actually have a lot of lookups so we actually go in

there and we look up what could be relevant to an alert so if I get I don't know like some sort of an ADR alert I go in there and I look up what is going on what could be relevant there and so on and so forth um we can use Sigma rules for things like that in terms of detection rules or what they call correlation rules or analytical stories that have a lot a lot of rules that look for things that are relevant and in this case the rule is simply looking for uh command kind of uh strings uh via some sort of like a query language or regular expression that finds these in web traffic so it's a

very straightforward detection rule um and we we like we like Sigma Sigma is like a unified language and Things Are beautiful we also like what they call um investigative playbooks anybody familiar with investigative playbooks not as many okay let's quickly jump into what is an investigation Playbook so um there is an open source thing from Splunk uh called research. spun.com if you quickly go there you will find all of the open source content from Splunk so you'll find detections analytic stories and playbooks this is obviously not the only place to find this but it's just one of the bigger uh repos out there with if information like this so if you can select like investigation playbooks you can

essentially look at investigative playbooks so for instance here is a Playbook that I don't know like what do you I don't know there's like identifier activity analysis there's like internal host SSH there's like what do you do when you find like an email what do you what do you do when you find like a threat Intel hit what do you do when you find something like that and if you click on any of those like for instance this case once a user or device is involved in something it will go and it will look up the attributes of that so that's already something that an analyst has to do like when you find an asset is

involved in some sort of an investigation you better know what kind of business unit you better know what kind of it leader what kind of manager is handling it and stuff like that ideally through a cmdb but if you don't have it you can also do just a live lookup for it um and anyways uh this is kind of like the Playbook you can kind of like see the exact um I guess Json off it on their GitHub so all of this is completely open source as I said uh none of this is to kind of like um none of this is charged and a lot of people actually take this content and they repackage it via Sigma or other kind of

like uh Source or whatnot um and it's pretty pretty common out there that we have this kind of investigative playbooks however I would like to pause here and ask anybody from the folks that are around here do investigations today and if so what is kind of the

guideline can you speak out yes so some sort of like an investigation once I have an ADR alert what how do I know what is relevant to it yeah I mean we we use a platform called logarithm it has playbooks on it and those playbooks will tell the analysts you know steps to do and it measures the time between the steps so that we can get mttr MTD type stuff and is that enough or does the analyst do a little bit more of a look that's an aptitude thing of the analysts how curious are they you know how much do they want to impress so it's it's a challenge okay all right fair enough thank you for sharing anybody else wants

to share a little about how do they do investigations or how do they find what is relevant to an incident you can shout it out but is it mainly just playbooks and lookups series of lookups and then anything other than that is completely are there guidelines for pivoting So when you say like a human could be curious got it got it so it it seems like a lot of it is really up to the human Acumen and we call that process a hypothesis and validation so they look at something and they're like you know what I hypothesize that this is a lateral movement let me go and figure out if it's actually a lateral movement or if it was just like

a driveby and I have to go and I figure that out and how do I figure that out I have to retain this acum and and knowledge that I have about the environment and then I have to go there and look for things that are considerably um enabling to a lateral movement so if you remember in the miter attack we actually have like tactics and I would be looking at the tactics before the lateral movement trying to find something that kind of answers that question was it a lateral movement did I see credential discovery that I see something in the area of maybe execution or persistance beforehand and so on and so forth so this is the state of art

today it seems like we're highly dependent on the human and unfortunately humans don't have a very good consistency so even if I'm an amazing analyst one day I'm feeling like it one day I'm not it's very hard to judge me on my quality of a ticket to promote me or to demote me you would probably just judge me on time so you mentioned mttr mttd right like just time so do you want more tickets I got you yes right think thinking they see evil and then they find logs to support their evidence instead of looking at the evidence for what it is right interesting and that's another problem CU then you're Chasing Ghosts interesting yeah so um to rephrase that

a lot of the time uh it seems like people might end up chasing the wrong stuff and that means that their hypothesis and validation uh train of thought might lead them to essentially Chase nothing and essentially waste very expensive time on the job and lookups and intelligence discussions and pinging users and whatnot after absolutely nothing while leaving the good stuff so it seems like we lack some sort of structure we lack some sort of a methodology there and that's really some of the reason that we don't really have a lot of good tools to support these analyst in these investigations the best we have in terms of tools in this industry is maybe use an

llm and hope for the best but an llm is a magic black box and I don't know how useful it is but we can you know we can actually take it on and ask an llm what does it think of something and see how would that work out so I have one of the best or one of the most popular llms out there and if I ask it um I have an EDR alert about privilege escalation um in a process uh on Windows 11 at my HR department what do you you think I should investigate in terms of relevant um events alerts and logs in my sim the llm would give me highly non-deterministic answers every single

time you hit up an llm its creativity configuration or what they call a temperature is usually set up to a pretty high value which means that a vanilla llm would give you a different answer every single time you're prompted with the exact same stuff so I I had no clue what would come up um but essentially a big issue that we have there is we're never really too sure what it will spit out so we we try to guard rail it we try to have configurations like you can easily use an O Lama for instance and have a local llama um where you can drive down the temperature instead of chat GPT you can probably use hugging chat so I can go

here to hugging chat where I can have some more configurations set up and I can make an assistant so in this case I have an assistant that I built for this and you can see that in this assistant if I edit my assistant I see essentially what kind of configs I gave it so in this config I can kind of like drive down the temperature to zero all the way it's zero and in this case or maybe 0.1 it doesn't even allow me to go to zero in this case it's as less creative and as more deterministic as possible so what that mean sorry what that means is if I give it the exact same prompt it's

kind of deterministic to a certain degree more than others and in that case I can really depend on what's going to spit out and I can have an actual experiment figuring out our lm's going to help me but I'm kind of jumping the gun I'm going towards essentially what kind of solutions are out there but this is just a little bit of a brainstorming bit so we figured that a lot of things relate back to the attack methodology so you probably already all know about attack if you don't know the attack methodology seems to be always referenced where possible because it allows us to figure out what is the answer to the what question so when you

look at an alert or an investigation you're going after who when and what remember so the what is usually referenced via some sort of a framework um or some sort of a knowledge base that allows me to have a common language for what is happening and answer that what and we use the attack methodology for that just a refresher going left to right there is M causality or sequentiality and then these are different techniques the value of having this is obviously if I'm a junior I don't have to be a unicorn at your sock cuz we don't have a talent shortage we have a unicorn shortage there's a ton of people that would love to take a

job at your sock but you don't accept them cuz they're not unicorns um but if you had essentially a nice methodology and a nice guideline where everything has MIT attak you can actually have them read the mitigations and detections and procedur and examples and they can kind of perform above their pay grade however another problem in this industry is the mitak uh labels that we have are usually non-consistent because guess what all the detections playbooks and whatnot have very very different authors most of these authors put the attack just for marketing they don't really put it for operations they put the most broadest technique that they can find or a tactic and it's not really very good

so one of the things that we should probably do in a sock is try to make sure that we have a consistent attack technique what can help there probably natural language processing and llm is a good place to start so you can easily go to hugging chat or AMA or whatnot find a nice model give it a few resources try to figure out how to get a nice um essentially a t uh label out there populated in a nice sheet and now you have this sheet that allows you to map everything consistently every single unique uh alert or signature or whatnot to a technique and that is what I would do so I would use some sort of like a

free thing and and we we built a free uh bunch of these uh that essentially they have a nice system prompt they have domains uh that can help them find out these kind of things and we built these domains essentially like a of domains that has all of the stuff that it would need to look up um what is the nice technique there it might not be the best but it's consistent so that's essentially the piece that you want to focus on when you're figuring out what's what um in terms of an answer but anyways uh you can very easily pull this off if you're not familiar with AMA you can also have uh a local one hugging

chat can also be set up offline but you still have to trust hugging chat with your data y y y but again you only need to Fe to feed the D duplicated or unique alert messages and descriptions that's it so really nothing sensitive out there it's just a rule set that you use in multiple Source types and that's really about it and you would have some sort of a consistent technique so let's assume that we have a consistent Technique we were talking about an investigation go back to the point as whatever you talking about so about that investigation attacks usually are not a single incident right they're usually a coordinated kind of story and when I'm

talking about giving me relevant data points ideally if you remember we were kind of hypothesizing is this a lateral Movement we were going back and we were looking for Discovery credential access defensive agion am I do I have any hits do I have any actual coverage of what happened in these uh tactics and if I do I can kind of like Stitch It Up in what miter calls that attack flow anybody here familiar with attack flow take care Gabe you are familiar with attack flow we got two people three people okay four people all right for those that are not familiar with attack flow attack flow is a nice project that Gabe was one of the

people that actually worked on it I know he's running out but thank you for working on that project Gabe that's a pretty cool project um so if you want to take a look at the project a quick a quick way to look at it is essentially let's open let's say Equifax Equifax was a big breach that happened in 2017 half of the people in the US have their sin numbers addresses date of births out there in the dark web somewhere so if you suffered identity theft after 2017 this is why um but essentially I can see the story I can see it started with vulnerability scanning and then uh this happened on a vulnerability scan in one

of the I guess um online dispute portal seems like it uh they had an Apache Stratus web framework vulnerability uh that was exploited out there and then there was a webshell and then after the webshell there were like some credentials harvested on that machine wow there were credentials on that machine and then there was an encrypted Channel where they quered some databases there was like um archiving of the data probably for exfiltration and yep there was exfiltration exfiltration used the proxy to to make it harder and then they deleted their their traces y they Tred they deleted Windows event logs and files and whatnot that they left okay so essentially the attack flow is a way for

people to uh instead of just read an AP note or an AP report now you actually have the Tactical uh not really fully tactical but a little bit of operational knowledge of what happened and you can see the sequence out there so it's just a nice way for the community to talk about these kind of sequences at the spot so anyways if we were to have a little bit of a task force a little bit of a workshop and you know on our Innovation Friday try to make things better you gather the team team how can we make things better how can we make next week better than this week and they say you know what let's actually action

some sort of innovation and make our incident investigation better like can we augment our playbooks can we augment our rules let's think how can we do it so one of the first steps would be like the who when what and when you talk about what we're already talked about let's figure out a consistent way to put MIT attack techniques M attack techniques is a good place to start and let's use some sort of a natural language processing to get that figured out and if we do we have a nice CSV with it we give it to the Sim boom everything in the Sim now has it um for The Who ideally we also have some sort of a cmdb

if not we can also make a um some sort of a CSV of here is here is a service account here is the account business unit here's the whatever and again give it to the um to the Sim to Rich things automatically for you and now your tickets already have like entity information as well as what is going on information so that's the first step in terms of MIT attack um enrichment for the events and then ideally we have some sort of entities attributes that we kind of figured out and ideally we want to use these in a similarity fashion so what is similarity yeah usually in the Sim you don't have such thing as a similarity in

a Sim you only have a common characteristic so what is a common characteristic well this and this share the port this and this share the OS this and this share the parent process this and this share the business unit or the subnet or whatnot but is there a similarity no we don't really have any similarity we have identicality do they share something or do they not so ideally in a common U practice like this we have an algorithm that helps us with similarity what kind of similarity you know like Port 80 and Port 81 are kind of similar right uh process uh Internet Explorer and I don't know Edge are kind of similar right uh it might not be

straightforward that it's similar but if we get some sort of a model that tells us what is similar we unlock a lot more um nicer enrichments right so I'm going to pause here any questions on this kind of train of thought the what um a little bit more on the what in terms of similarity and then The Who and a little bit more on expanding the what and the who in terms of similarity and ideally the output of course is find me stuff that is related find me stuff that is relevant any questions if not I can fly through the rest all right we go to the fun part so MIT attack techniques we talked about let's classify events with

relevant attack techniques we use an llm how do we do that we get any kind of M we give it some sort of a fine training small labeled Corpus if we can if we cannot we just thrust it with an unlabeled Corpus and we give it some uh essentially websites that it can look up and we can essentially just quickly pull something like this out on AWS and put it in a little stage maker and the output is actually pretty nice so we here we use a small BT nothing too crazy not really a big llm and it's actually pretty decent like it's actually pretty pretty decent it's not that hard and you can get a CSV out of this and it

immediately enriches your sim the second second part is kind of interesting so first we're going to take this as a big data problem I have 10 terabytes that come to my SIM every day I cannot feed this to an llm never I can feed it to an llm so what am I going to feed it to I'm going to feed it to a classical machine learning algorithm such as clustering which is very easily understood very easily explainable very easily um auditable which is very important and transparent and llm I cannot really understand why it did something clustering I can't um so clustering events that's what I'm going to go after and I'm going to Cluster them based on

similarity in the characteristics of the event as well as characteristics of the attributes um uh of the entities involved so if it affected two people that are kind of similar I need them to be in the same ticket if the events are kind of similar I need them to in the same ticket and ideally after I have these clusters of course I can stitch them based on sequentiality because guess what I have the technique and the tactic so I make some sort of a basic data structure and this data structure I can kind of highlight what I need to be the nodes which is how this data structure is going to index things for my clustering algorithm and then I just

feed uh this data structure to what we call an embedding algorithm um usually if you don't have a complicated data structure you can pull this off with a very straightforward uh clustering algorithm without doing the the this kind of data structure but if you have the time if you have like a summer intern or something like that or if you actually have like a full Friday you can do this as your step two your step one would definitely just be a basic um data structure like tabular format but if you can do this it it makes your life a lot easier and and the connections are actually after what you want cuz while you're designing this you're saying and

node is a cve now my clusters is going to be related by CV or maybe a node is simply an entity now my clusters are based on entities involved and so on and so forth um and then of course you can chain things around with what we call a markovian uh logic essentially so this is a finite State machine we call it markovian uh essentially models just because we make it fancy but it's a finite State machine and I can easily go to my chat GPT again I can generate this code and by the way I already generated it it's it's sitting in my GitHub I will have the links ready for you but essentially uh make me uh python to

Cluster alerts with 10 event attributes and five uh entity attributes right so and it will now just show me the entities that it will make it will show me the uh essentially alerts in a very tabular format so these are the five attributes for the alerts they're encoded numerically you can encode them in whatever way you want and uh essentially uh you can also get the entity attributes there and you can choose a model so the model could be DB scan K means whatnot you don't have to be a data scientist you just have to actually deal with this the same way you're not a combustion engineer but you drive a car right like you just deal

with it like how do I use it I have this can spit out code at me let me try a few things let me hack around we're hackers consider yourself in hackathon one Friday and pull this off and you would be astonished as how easy you can provide humongous value in this area and again the code is out there we have a vog about it uh this is what we got so we fed about 2,000 alerts just a quick kind of uh uh playground that we got uh the ground Truth for this was one uh cluster and then we found eight clusters and actually all eight were kind of interesting so our ground truth was

broken cuz we didn't know that one of the firewalls was kind of like letting showan and other Bots online do stuff so we didn't look for these but these were actually through when we looked at them we were like oh interesting okay um so essentially in this case we have this kind of architecture where you have a couple models uh one that gives them tactics and techniques which can be just once in your life via an llm make a CSV and then once everything has a TTP it doesn't matter what kind of data format it came in um you can also have one of our models that we uh outsourced sorry open sourced um allows you to

immediately switch any data format into a common data format via an llm and you can have a unified data model that can go into like some sort of a flow or clustering detector uh and you can get things chained up uh very nicely so again the idea is to put out attack flows ideally you're clustering stuff you're correlating stuff and then um you just put it in a flow and usually the first step is just know the causality uh via uncovering techniques uh here are some of the screenshots that we did so here's the data that we fed um we fed a ton of those and uh immediately the cluster would look something like this

so these are all of the IPS of the stuff that is relevant uh here are the entities that were involved here are the techniques that are involved and so on and so forth and here here are some internal uh internal users or internal IPS that we marked and um obviously it's so much better than the current stories and rules but it augments them because they are bettered at certain things and you want to augment them you don't want to replace them um how is it easier to adapt and maintain you can easily go and Mark in the cluster what do you not like and what do you like and immediately the cluster algorithm would learn how to

Cluster things the way you like it without you explaining too much or writing a rule uh it's actually more efficient than a rule cuz the more you have of these books and lookups the Sim is going to be slower so if you hear people complaining or ranting about oh you know Q radar is so slow man like I want to move to like Splunk or something like that you you immediately ask them like how many lookups do you have on there like how many continuous searches do you have on there like how many playbooks do you have that are constantly hitting it with searches like if you go in the jobs then Splunk you like you can have a lot of jobs but

usually when you transition into a new sim you don't take any of these rules with you so like 3 years later you're you're again like oh Splunk is so slow manly um so that's what I see a lot and then the nice thing is it's Standalone so the leadership changes the Sim they change the EDR your clustering thing that you worked on a little with your intern or in your Friday Innovation Fridays or whatnot is going to live with you for the rest of your life um and yeah you can correlate via nuances and that's the similarity piece because clustering takes in in consideration similarity it actually looks at semantical similarity of of Kevin and

Kev or HR and Corp corate safety like these are actually similar but if you have a rule or a Playbook it's just looking for a match on business unit and that's it um but yeah that's essentially the idea a lot of buzzing alerts um ideally just put them all together this is a UI that we built uh a lot of this is available on our GitHub um but essentially ideally your full postive ticket looks something like this you already have the sources and whatnot you have the resolution of the comments and if you already have these tickets historically let's talk about how do we do this via super supervised learning which I couldn't put into this uh

discussion CU this is just unsupervised for people that don't have historical tickets but if you have historical tickets that you can trust and you can be like you know what my tier three my senior guy I trust this guy any any ticket that this guy closed I can use to train a supervised learning model to notice what is a false positive to notice what's a what's a nice investigation to do to just show the analyst today what are relevant tickets and they can just quickly skim through them and be like oh yeah yeah yeah yeah yeah this is the vulnerability scan that happens every quarter for compliance and every single time it's marked as a

ticket but this is obviously false positive because every time I just ring up this guy and this guy's like yeah this is our vulnerability scan or this is our pentest or this is oh yeah the the accountant keeps on going every quarter to pull up Financial metrics for the management like this is the exact same thing every single quarter if I have these tickets uh like the tax tickets or whatnot even the coordinated tickets like the AP stuff if I have them historically and I feed them to a supervised model I can teach my new generation of analysts how historically good human Acumen did these investigations if I have it but if I don't have it unsupervised is your

friend natural language processing clustering and the rest is history but yeah um these kind of these kind of things augment your playbooks I don't leave a lot of time for Q&A so throw it at me if you have it there's a lot of a lot of stuff in that thick um these are the links for the for the hugging uh chat uh assistance that I built for this there is a gist here that I kept for you guys that has some of the clustering algorithms so this is one uh you can immediately just go to this it's public and it does essentially what the chat GPT did like just a little fancier uh and then you can see the Clusters uh

essentially like what what alerts or what incidents or what logs or what events are kind of clustered together and just to remind you these are coming from hetrogeneous Source types they're not even the same data model none of it is similar um but we are still able to kind of like get um the overall uh essentially similarities in there but anyways any questions I need a lot of questions go for it just a moment oh yes thank you I was thinking I I got to go there appr uh how much do you trust GPT to transform the are into a common model I don't trust CH GPT one bit the whole demonstration was starting with Chad GPT and then

immediately saying what's wrong with it so I the only reason I brought it up is it doesn't have temperature control on the UI so don't use it in the UI with like for anything serious uh brainstorming is amazing quick gold Snippets are nice but I wouldn't use it in the UI the API you can access the temperature for some reason the UI you cannot access the temperature so if I'm in your shoes instead of ever ever ever using CH GPT I would install all Lama and have a local instance or if you don't care about it being local I would use hugging chat so I would immediately go go here to hugging chat through hugging face and I would go to

assistants and I would create a new assistant and I would call it whatever test and I would choose a model that I want to play with I can choose a million of them and I would drive down the temperature as far as it would let me so and mistal can go to zero I would do that CU if I'm being serious about anything I don't want it to be creative I'd rather have it be wrong and me knowing that it's strong in this use case and it's not good for this use case so zero temperature means zero creativity and it stops giving you non-deterministic outputs it's highly deterministic same input same output a million times thank

you 100% hey great talk um two questions um the first one being um you talked about uh unlabeled data but how often have you found that the miter tactics in techniques are mapped accurately enough to provide a strong signal for doing some of the similarity searches uh that's the first question should I answer it first yes go ahead okay um so when we started training our llm to do this we gave it all of the stuff that we pulled and scraped from GitHub from different rule sets and people have already given them attack techniques immediately it performed really really bad and when we investigated it we found that the data quality was not consistent and we were like yeah of course a scout

Source data what what the hell did we even think so we scrapped all of that and we had two choices to either like sit down and make our own fine tuned and make sure that our people are consistent which means you hire the same person and the same person should write regular Expressions to check their consistency because every day they're acting differently and we did that but that was ridiculously expensive and I wouldn't tell you to do that um you you can just thrust unsupervised like an llm and just hope for the best because it doesn't matter what quality you have literally it just matters that it consistent if it's consistent you're good to go and

actually in your rule sets today if you go to your sim and in spunk or whatever you just type in like after you do like an index all and you do like a d up per message you how many messages do you think you have in your Sim in the last three months like maybe a thousand different unique ones you can probably just sit down and manually do it you don't need an llm but an llm would give you a quick and dirty draft off it but short answer tldr I don't trust any of the stuff that's out there in terms of techniques because they do it for marketing they they actually don't use it for operations most of the time yeah

I agree with you the other question is when you did clustering um so what kind of clustering approaches did you try uh did you try any uh basic clustering approaches like um hashing like tlsh or um did you try more advanced clustering with a vector database uh that would that can further provide some explanations Downstream did you try any graph clustering algorithms could you just give some commentary around that yep uh so if you want the tldr version of this is use something that is simple first and then build on it don't don't be a perfectionist perfectionism is the opposite of progress uh so don't tackle this as a perfectionist tackle it as an iterative kind of process like as if

you're a Founder You're Building like a version 0.1 first and so on and so forth so I would tell you to start with either a k Mees or a DB Scan they would use essentially a tabular data if you want to improve your exactly on point so the graph kind of um data structure and the graph kind of embedding that allows you to then use DB scan is so much better CU in the graph you actually do what we call Knowledge encoding so if you remember in my slides uh I had a little bit of a graph data structure here that focused more on cwe and cve because I noticed in my sock I would really like

to have my tickets focused on vulnerabilities maybe because uh the way we action most of our tickets is I don't know we we we actually fixed our software because we're a product company but if I'm a manufacturing company or I'm an operations company why the hell do I need this I focus more on entities which of my suborganizations is hacked which of my sub entities we we acquire a lot of companies or something like that which of them actually has the problem and I want an entity based graph so in this case you're knowledge encoding and you're telling the clustering what would you like to get out of it and as you see things that you don't like when you give

it feedback it can action it very easily because it will just change the weights of edges versus nodes it will change the weights of which attribute in the nodes or entities do you care more about do you care about business unit or same operating system or same it leader manager admin and so on and so forth can very easily tune this any more questions amazing questions by the way thank you can we get a question here thank you this is amazing you guys can shout it out and I can repeat it if you want we recording so um I just wanted to clarify so you wrote hugging chat and attack llama those are things those are yes so attack

Lama is just the name that I gave it um but I just went to hugging chat like any normal person and I just gave it a few links to to make it focus more on certain areas when it's doing its search so in hugging chat um you can go into to the settings and you can give it certain things so this one for instance doesn't go online at toach uh and it has a very low temperature relatively so in here I just kind of like gave it a system prompt and that was it but in this case um while I was creating this and again it doesn't take a data scientist to do this that's why I showed this early

because you can do it on your phone you don't even have to have a laptop that's how user friendly it is this is 2024 like it doesn't need a PhD in data science and I gave it the links that I thought were interesting so you can see in my links I have a ton of links but the links are focused on the sources that I wanted to look at so bug crowd varia DBS W directory like rocket reach even like for Recon on the attribution uh URL void like all kinds of um essentially links that I thought were interesting to my um to my essentially use case right my use case was threaten attribution software deop

fation I wanted to figure out what's the attack in any kind of event that I give you any kind of power code that I give you anything that I give it this assistant you better give me um an attack technique that is relevant and this is a public bot uh I can share the link I can put it in my LinkedIn this is a public bot and there's a million other public Bots out there and you can look at my settings nothing here is like a secret per se I can activate it and I can go talk to it this is the direct URL system instructions you can see my prompt there is no secret at all in this

case and then if I dislike the the the command plus that I used here I can use another one right like and it's all via UI if if you want another UI something else that you want to do local because you don't trust hugging chat oh Lama okay and then my uh second question is you know from an operational perspective as as someone who manages like a sock team how do I tell them to use these tools what's the Tactical do they go to these is this something we install ourselves or do we use it online or how I'm just trying to understand tactically yeah how do we make hugging chat is online or or can we link it to our Sim

is that what you're proposing and that it it's its own system that spits out something add us I'm I'm kind of lost on I I think I understood your question you're saying where does the sit um I would it it depends on your use case but if you want the most generic use case right at the Sim you have essentially an export to S3 buckets you probably have it already for cold archives for compliance but if it's sitting in an S3 bucket the world is yours cuz if it's sitting in an S3 bucket you can just open Cloud shell until it access this S3 bucket for free compute and do whatever you want with it or you can start a sage

maker you can start a million things one of your engineers can probably do this like assist admin kind of thing and they don't have to write the the code for data science they just have to operate on that humongous database of Json and they can do a lot a lot of things if you give them the capacity now you can give them some directions you can tell them listen like you're going to have a lot of hetrogeneous data sources here cuz our Sim is sending stuff our assist logs are sending stuff we have a lot of different things that are going out there and that is completely fine so you can uh go to our uh

GitHub um and you can go to the data mapper moduel for instance and this llm is going to quickly map everything into one uh data model and now all of these Json are just like one data model based on this llm that map everything for you and guess essentially the the Integrations and now you have essentially some sort of a CSV that they can do with hugging chat or with ama if they want a local instance if they don't want a local instance it's fine none of this is sensitive information it's the rule set of whatever you use use crowd strike use poo altu use whatever you use get the rule sets from the Sim give

these rule sets to any llm that you trust there're probably less than a thousand you can actually task one of your senior people to just sit down and give it techniques on an Excel sheet and that would be the best use of their time and this CSV is now going to go back to the Sim and it will enrich everything so on the Sim when they are doing notable alerts they will just do group by whatever this exists in the data model and they will add a data model for technique and you should probably do this as well for entities attributes like you probably don't want everything in the cmdb nobody has a perfect cmdb

but at least tell me when was the last time it was patched when what software is running on this who is the leader of this when we want to call someone and tell them is this fine is this a pentest is this a vulnerability who's the leader if you have these very little things per entity and these very little techniques per alert now your sim is a treasure throve of information take that information which is already on S3 give it to the simplest clustering model your team can use not try it from scratch use and at its worst iteration it's more valuable than them sitting there and maybe the invest instigate maybe they don't cuz when they find a cluster that

they like that's already a blueprint for documentation of a guideline that you want in investigations they're already saying oh when an EDR happens we probably should go look at the Powershell commands before it I didn't really think about that but now you know what I'll actually write this into a Playbook or a detection CU then you're not tracking them by time you're tracking them by how many things can you improve in the process in every ticket how many contributions do you have to our content did you contribute to our content did you in this ticket when you closed it did you suggest a better tuning for the detection about they're tuning for the Playbook did you

suggest a new one and if they did fantastic if they don't encourage them and give them an llm to support them while they're doing this but make sure that their llm is not creative 100% yeah but Des three I think is the best way is you already have a cold backup and it's cheap if you want to delete it you can delete it but it's the cheapest storage in the world we have one more I'd love to got a million more in terms of the clustering I'm wondering about whether you're doing smart clustering or weighted clustering so that you know maybe we know hey this particular IP address whenever that's involved this seems to be tied to that attack it might

not be the only Factor but it could also be um anything involving a certain port number or I don't know what else you could be doing in terms of waiting or smart clustering where it's like hey we got if it's got eight things in common it's more likely to be similar than so are you doing anything like that is yes I love it this is a fantastic question so for most people start slow for the people that are interested in like getting better stuff I don't want to I don't want to make it sound complicated for the people that didn't start it yet so always start slow cuz the smallest kind of element of you have there is

valuable it will reduce false positives and most importantly it will reduce data you have a problem of Big Data reduce data even if it's bad at reduction it's consistent at reduction and it's explainable at reduction so anyways to answer your fantastic question your question was how do we make sure uh our clustering is smarter than you than regular for instance like is there some sort of knowledge encoding in similarity for instance like is Port 80 um as close to to 81 as it close as it is close to 79 numerically it is but as a cyber security expert no 80 and 81 are alternative HTTP ports 79 is garbage so I can come up with an embedding or an

encoding where essentially and again you can do this with chat GPT like I can go to chat GPT I want to show you how easy this is cuz I I don't want to ever sound like a data scientist like a robot develop an encoder for ports that give them um an encoded value representing their ports Association where alternative ports are closer to each other than ports that are not servicing the same protocol right and it will give you a function like that I don't even have to look cuz I know it will make up something so immediately it goes for like some alternative points right it didn't even get you the correct Port Alternatives but https has like

8443 and a bunch others now that I have this example I will make them have closer numerical values to each other where I no more call 8080 I call 81 I no more call 81 81 I call it one and I call 79 106 it's a way for the machine to understand your knowledge you don't have to worry about how we talk to the machine you just talk you just worry about your knowledge and if you tell your users or if you tell chat GPT give me a list of all common points and their protocols give me all of the alternative points boom give me a an an encoder hopefully it actually gave me the code

yeah it did some sort of code for the encoding um I didn't check it but yeah it looks like it did so all of the stuff in HT in FTP is now called three so no more 20 and 22 21 and 22 I just called them three so they have the exact same numerical value which means exactly what you're talking about like there is there is now some sort of knowledge that we encoded to this clustering and now this clustering is just not doing crazy stuff I can also go in there and tell it HR is very close to business unit Corporate Safety I can go and tell it our suborganization XX analytics is very

close to XX solar power because they got acquired from the same company and they have similar systems this subnet and this subnet are very similar because they have both have same OT systems whatever it is right and I can give that knowledge that is stuck in the brains of your analysts stuck in the brains of the folks that actually know the operation and I can give it to the data so that it will outlive all of us it will outlive your documentation and it's easier to do than your documentation and it will be the True Legacy that enables your people not to acquire products but even if they acquire products they go into these products uh discussions very

intelligently when they go talk to secur anex or drop zone or whatever they tell them I here's my use case I actually know how to test this like are you going to give me tickets that have stuff from the same business unit and these vendors would love to take these knowledge bits in their info Gathering sheet and use it for you as an mssp as a vendor of a product or whatever but have the knowledge feed it to the clustering you're completely on point and the clustering is going to be smart the other question that you asked was what kind of algorithms as well um DB scan and K Mees are fine but as I said before

like you can also use the data structure to encode the knowledge and kind of cheat a little bit what do you want and in that case uh you can kind of make sure the cluster goes the way you want it to go um but again there is a piece that I wanted to talk about at the very end about supervised learning which is very very important if you already have your tickets historically don't start with unsupervised learning start with a classification problem like classify things and show the analyst the historically relevant tickets that were closed before that you trust of course not tickets so you can filter the tickets by who do you trust and who do

you not and use that as a classification uh problem data set any other questions I love this room by the way you guys are freaking brilliant like these are amazing questions give me more the more the merrier one more quick one if y'all have it we could do one more quick one come on I like it they're awake they're awake come on we got a lot of innovators a lot of good brainstormers here oh here we goes right bringing up the the old tickets as you were saying could you not add those back into the llm to increase the learning for any future analyst that might come across a specific investigation it's a very smart idea I

would definitely I would definitely do that in the rag but it would be like a really Overkill to use an llm for this because an llm can do a lot more and in my limited use case or my limited thinking here was simply a classification problem while an llm can grasp a lot more than just a simple classification problem I I consider an llm a magic box that I can never audit and in that case I wanted to be able to audit more so if I have a shitty classifier like like svm classifi like the most basic classification problem uh sorry model um I can very easily Trace what is going on if I go to

chat GPT and I tell it listen Chad JP here's the code of the model this model keeps on telling me stuff that I don't understand can you tell me what's from wrong with it CH tell me by the way run this code and you would know which part of your data contributed to that classification that you don't like and now I have complete auditability that I can go and be like oh this data I thought was all nice but this guy messed up this one ticket let me just delete it and it's very easy to explain an audit small models than an 8 billion model running on a humongous GPU that is going to obviously cost me more so if ideally I

also want to be cost efficient so that the these kind of programs don't get killed when your leader is like oh we got to cut budget cuz there is a recession cuz you're not actually using a GPU like dude like this is like five bucks a month on AWS to run like a clustering algorithm every single day if you want like it will never get cut it's a legacy that you leave and it will never get cut um especially if you have a like the historical data a small classification problem allows your team if they are not data scientists to do very easy explainability traceability transparency all ability of what is wrong when something goes wrong and they

can very easily delete the data they can give feedback feedback for humongous models is very hard like can you actually tell an llm don't do this again uh I don't know it's kind of hard even as a data scientist but if I'm just an intern at your company and you tell me do a basic classification problem I would be stoked and every time I I get like an explainability piece from shgp I will try it and I will get success and you now have an employee or an intern that is looking forward to innovate because you're giving them easy models to work with and they can see their success in every step if you give them

an llm they have to read the paper and they don't really have a lot of success and they hit brick walls and they're like H I don't know all right give it up for EZ smart guy talk all day about that thank you so much