Paul Ihme - Building a sustainable detector development lifecycle

right so yeah as she said we're going to be talking a little bit about some detection engineering topics today but just real quick uh background about me my name is Paul Amy as Kendra said I'm a co-founder at soteria I started off my career in in the air force uh as a software engineer and then moved over into the cyber security side of the house later on in my career spent some time at the National Security Agency doing some of the offensive Mission there spent some time at JP Morgan and in the Internal Revenue Service in the U.S uh doing uh hunt missions and security operations work and I've been a long time ago I used to work in fast

food and uh and did some Landscaping so if this whole cyber thing doesn't work out I've got to fall back there but there's a uh over my career I've had really great opportunities to work on what I've put here offense defense and special teams I've done some GRC work as well I think that's that's been really beneficial to me to kind of help me think about how um how to go out and find bad things happening in corporate environments so I want to talk a little bit and you know I can't really go fully into the history here and really do it any justice but just talk a little bit about the the history of how we detect bad things um

happening and in our corporate environments these days so it really started out with uh with antivirus and firewalls and things like that so uh somebody like John McAfee would write a piece of software that would scan the files on your computer and there were you know a handful of of known viruses back then it would look for characteristics of those files maybe some some strings or bite patterns that that were unique to that file and it would flag up and say this is a virus and over time as malware became more prevalent and threat actor started doing stranger and more sophisticated things that just wasn't viable anymore so there's been a very slow Evolution and

again this is a a big jump but really what's come to be the the sort of state of the art in the state of the industry now is that organizations need to collect Telemetry from from various systems and help uh identify strange patterns within that for themselves and you can buy products to do this you can buy an EDR product to collect a bunch of endpoint Telemetry and it will come with a bunch of rules and it'll say that it'll stop a bunch of threats but there's limitations to that and and such that uh you know they can't really write great rules that are going to be specific to your organization because if they if they write some rules for you

know everything that's in the miter attack uh you know framework they're going to throw so many false positive that nobody's going to look at it right nobody's going to pay attention to the 2000 alerts they get every hour and they're just going to turn it off and same thing for network security monitoring for for Sims so really what organizations need to do is they need to to have this capability where they can collect this telemetry understand what's normal for their organization and what's not normal and and understand how they can write rules to to find uh bad things happening and that's really what detection engineering is to me this is still a very uh immature

foreign within the cyber security ecosystem I think this is this is something that is still kind of poorly defined compared to some other areas of expertise within cyber security but the way I look at it is outlined here so a detection engineer first you have to to make sure that you understand what you're looking for so you have some sort of of threat intelligence capability whether that threat intelligence is a very organized threat intelligence team that that spends all their time doing this or whether your threat intelligence is you know scrolling through Twitter and reading through you know bleeping the computer every day and just trying to understand what folks are doing and there's there's different levels of

maturity but but you need to have some input to help you understand what it is that you want to catch right and then you need to make sure that you design systems that can can capture the Telemetry you need you got to get that data right you got to have something to look at in order to um in order to find the bad things happening and have some way to uh to parse that data and write rules against it and then the focus of of today's session is really going to be on these on these last four bullets on the slide so once you have all this data coming in you've got to be able to write your own

logic to to find bad things happening you know for whatever definition of bad that you might have within your environment you have to be able to test and verify that that you're actually catching the things that that you are that you think you're doing right you can say that I've got a rule to catch this thing but how do you really know uh maintain these things because threats evolve systems evolve you know when you upgrade from Windows 10 to Windows 11 there's some changes to how the operating system sends that Telemetry to you so you have to make sure you're accounting for those types of things and then I think it's also important you have to to know when you have to say

okay it's time to retire this because uh we're not really getting the value out of this this capability that that is required for us to continue maintaining it so that's at a very high level how I think about detection engineering there's people who might look at it differently or have different definitions but I think that's a good sort of Baseline that everybody can agree on So within that there's there's a lot of people who and and I'm sure that anybody who's ever worked in a sock has experienced some of these things there's a lot of of pitfalls that can happen when you start writing your own detection capabilities and and I've just put down a couple of things here that

that we see folks struggling with right so first and foremost if somebody comes to uh to your sock right maybe you've got a CIO or a ciso or a CEO or director or uh somebody in your leadership comes into your security operations team and they say hey I just read about this thing on the news um can we detect that can you answer that like with confidence right can you say well I hope so because you know these tools should be maintaining those rules and I would hope that they would do it or can you confidently answer that question and say yes we have this rule that rule this capability to to look for this threat

and we have it documented we know it's there or conversely if if you realize that um that you don't have that capability do you know how long you've been unable have you never had that capability or has that threat evolved in some way and you can go back and track like oh we had it and then last week something broke and now we need to go back and fix it and and we know that we have a gap of one week where something bad could have happened and we wouldn't have seen it so uh really it comes down to a handful of things you have ineffective rules people will write rules because they think that they work but you know they

have no way of testing them or uh well I've seen a lot in my career somebody comes up with this great idea they write a new rule and now you're getting 10 000 alerts because you accidentally wrote a rule that that sends alert every time a new file is written on the disk or every time a new process is kicked off um and then then on the other side sometimes you have this This brilliant detection engineer who's writing this this great Next Level you know detection logic and then they go home for the day and your overnight analyst gets an alert on their screen and they have no idea what it is right so this is a another

problem in this practice is that there's not a lot of great documentation and a lot of folks don't really go through the rivers process to make sure that you know the knowledge isn't just living in their heads what to do when this bad thing happens and then finally uh I've seen this a lot when when looking at security operations centers where somebody goes in to make an update and they realize that they broke the rule but they have no idea on how to go back to the previous date right like I thought I'd just change this thing but now if I try to undo that it doesn't work the way it used to and I don't know what I

actually did so these are these are common problems and and there's ways to to solve them right so the the approach that we take and that we recommend uh that anybody you know should take if they're doing this is detectors this code this has become a a pretty popular uh approach to these things and I think there was another talk today that was talking about policies this code so kind of the same thing there's a there's very well established uh and defined software engineering practices that really solve a lot of these same problems and and our belief is that we should be applying these these uh these techniques and these uh these ways of thinking to our detection engineering

just as you do with with software engineering so uh first and foremost how do you track a requirement all the way to that implementation so how do you know what's in your backlog of here's the things that we need to develop and we need to have these capabilities and here's how you know we're moving along uh that practice and then also from the from the other side of it if you look at a rule within um within your detection platform whether it's a Sim or an eer or whatever can you go back and say what requirement was this actually solving like why does this thing exist and let me track it back who requested this and why and when

and so on uh the second thing is is having good coding and documentation standards that way whether I write a rule or you know one of my colleagues writes a rule it all looks the same and anybody else can come in and read it and kind of understand what's going on and they know what to expect so having that standardization is is massively important uh having a way to control the source code making sure that you have a repository that you can go back and look historically of every change that's ever been made to this and sort of see that life cycle uh unit testing you know when you write a rule how do you know that it

works before you just shove it into production and hope for the best and then finally how do you control that deployment process so you know that it gets deployed everywhere that it's supposed to so ultimately what's happened in the past is you know click Ops right somebody will log into the platform they'll make a change they'll hit save and there's no record of that happening nobody really knows that it happened and you know you just hope that it doesn't destroy anything and that's that's caused a lot of problems for a lot of Security operation centers so I want to talk a little bit about the way that we approach this we we run a managed

detection and response service so we're this is a big piece of our business and we've learned a lot of lessons along the way because we've done things the wrong way and uh and suffered the consequences so we just want to talk through our approach of how do we go from from an idea to a detection capability and and uh and get that into production in a safe way that's that's actually scalable so I don't want to read all these bullets on this slide but at a very high level it's it's a four-step sort of continuous cycle somebody will come to us and say hey we've got an idea of something that we need to be able to

detect and that could be somebody from a threat Intel perspective that could be one of our pin testers who just came across this new technique or maybe they're developing their own techniques it could be somebody who just read a news article somewhere and just wants to know you know do we have coverage for this or could we have coverage anybody can do that and that could also be a tuning request so maybe you have a sock analyst who says we've got a lot of of a lot of false positives that keep hitting on on the system and we're getting way too many alerts can we can we tune this out or make some changes to this rule so

somebody identifies a need right and then it moves over into our detection engineering team where they will actually you know create the logic uh they need to make sure that you know they're they're documenting it and putting it into the platform the way that it needs to be done they're going to perform some testing because we don't want to just you know write something and hope that it works right we're going to put it out there we're going to test it and make sure that it's doing everything we think we're doing and then we're going to deploy it and monitor and and make sure that we keep an eye on that to make sure that it's continuing

to do what we want it to do and it's not just overwhelming our team so that's a very very high level overview uh and so so here's how we do it right so first and foremost creating a new issue we use jira because again we're trying to approach this like software engineering uh jira you know gets a lot of hate from a lot of folks but ultimately it is pretty good from uh doing the things that we want to do here and we have a template that has to be filled out in order for anything to get done with this so somebody can't just come in and say uh you know OneNote and then like leave a a ticket there that

makes no sense to anybody else so we want to know why why should this activity be detected put some labels on it right is this Windows because we have folks who do windows it's a Linux is it you know some Cloud platform is this an EDR tool whatever it may be and then just links to reference material so it's not you know we're not trying to to create a massive amount of work for folks but there has to be some rigor in this and I think that last piece is is really important of having some sort of reference we don't want somebody to just go post an idea somewhere um this needs to be validated in some point and that

can be a that can be a tweet or it can be a mastodon uh tude I guess is what they call them or it can be a blog post from from some other organization or it can be you know white paper but there needs to be something for our our detection Engineers to be able to look at and say okay what's the actual threat and how do we actually want to to be able to do this so we learned we used to have a ton more fields we would have to you know fill out and you know effectively what it did is nobody wanted to do it because it was too much work so they would just you know not not do that

right so we learned that we really have to try to keep this simple and what are the things that we actually need to have in order for this to be useful and then just get rid of everything else we also learned that we can't just do this for new ideas right we had to do this for for tuning requests so if you've got a rule that's firing too much and you want to make a change to it any sort of change to our code any sort of change to production has to have a ticket associated with it and we'll walk through how that works later and then we've also learned and we'll go through this a little bit later using smart

commits that way if you look in our GitHub repository you can link anything back to to the ticket and who requested it and when and things like that so um so that's the first phase we need to have an idea this is where it starts and somebody says we need to detect this thing and I need somebody to go figure out how to do that and they'll pass it off so the next phase is is the detector development piece so this is where we're going to put in the logic that says if these things happen on the system or if these things happen on a network or if these you know log entries get created

In This Cloud environment then we want to trigger an alert to our security operations team right so there's there's that that detection logic piece but but we went a step further and we said we need to to create a template that captures more information because ultimately if an analyst gets an alert in the middle of the night and the person who wrote that rule is not around they need to be able to know what to do with that without having to pick up the phone and call somebody right so so we created a very simple tip it's all just uh it's all in yaml and we just give it a a name right just a human readable

name here's what's happening uh a status so we can it can be active you know in testing or it can be archived in metadata and this is where it's really important so we want to tag it with you know the miter attack framework IDs that are associated with but we want a plain English subscription of if you see this fire here's the things that you need to go look at and here's the um you know here's some some maybe cases where this could be benign activity and here's a Surefire sign so this is malicious activity and it needs to be actioned and here's some steps that you should be taking in order to investigate because

if anybody from our most Junior analyst or our most senior analyst needs to be able to get this alert look at it and know what to do with it in order for it to really be useful and and in order for us to do our job uh the actual detection logic and then we also have built a capability to do unit testing so we want to make sure that anything before any detector goes into our into our production environment that we have ways to test and Ensure that this thing is actually going to fire when it needs to or if it's a tuning request we want to make sure that it's going to stop firing in the way

that we want to while also still catching the things that we that we wanted to to test so if you're not familiar with unit testing in software engineering if you develop a piece of code you'll do some sort of unit testing and basically what it does at a very oversimplified level that says if I send this input to this function this is what I expect back right and just do that over and over with a bunch of predefined data and if it ever gives you something unexpected then you know something broken it's not working correctly and we'll go through an example of what that actually looks like in practice but ultimately that's what we're trying to

do is we want to make sure that no matter you know what changes somebody makes it can't get into production if it's not working the way it's supposed to and we can't trust humans to do that accurately every time so we we automate that and make it part of the machine so again going through the testing phase uh does it detect what you want it to detect does it not detect the things that you specifically want it to not detect and is it resilient enough to to withstand evasion tactics right if I just change the capitalization of this command or I you know put it in quotes or something like that does that completely break the

ability for me to detect this logic so we need to make sure that we're not writing what we call brutal rules that are just easily broken and bypassed so uh some some interesting lessons we've learned here for folks who've not done this before we learned that we need to just have a generic false positive alert on every um on every detection because we have definitely written uh rules that will fire on every network connection or every new process and when that happens in production it causes all sorts of fun and exciting problems so we've learned that anytime that we put something in there we're just going to have a generic false positive that way we know that

it's not going to to trigger on every possible event that comes across uh and when we started doing this we originally had a whole process that would spin up you know these networks and it would go and automate these uh these attack paths and we would use that as part of our testing and it was one of his expensive but it was also really cumbersome and it took our team a long time to put new capabilities out there because they would have to spend a day or a few days you know designing this attack and then figuring out how to replicate it and spin it up and then every time we we push something new to production we

would have to test it and it just got super super cumbersome so we had to really work with with our team and our partners on the vendor side to to really close those feedback loops and help our detection Engineers know if their if their tools were working um more quickly and we'll talk a little bit about that here in a second and then then the Final Phase is the deployment of these rules so once uh once somebody has written a rule they're pretty happy about it we want to have a very controlled process of how does this go from from a developer's workstation into uh into the production environment and for that we use git so if anybody

here is familiar with GitHub or if you've ever done this in an Enterprise environment you know there's very simple things you can do of just being able to commit your code uh you know to a repository and it gets saved there and you can see the entire version history of that code but in an Enterprise environment there's a lot more things you can do that that are very useful from a security perspective so uh first and foremost you can have an enforce verified commits so we have this set up in our in our repo where if somebody doesn't digitally sign their code then it just doesn't go up there right so we want to make sure worst case scenario I

do a lot of incident response work and I'm always paranoid of what happens if somebody breaks into this account or into the system or whatever so I'm always trying to think about ways that we can protect that so you can make sure that you have to to verify those commits smart commits is a feature within GitHub and jira where you can actually link the two so you can see any you know version history of of your code you can link it back to that ticket we talked about earlier and then you can say okay well this piece of code was written to address this requirement and be able to map it back and forth and it's useful

for troubleshooting but frankly it's mostly useful for Auditors so if anybody has gone through a sock 2 type of assessment Auditors want to see tickets for everything and have this evidence and it makes that a breeze and Auditors think that it's the the greatest thing ever so that's that's super useful again our unit testing so we want our developers to test things on their systems before they deploy anything but before it actually gets into production we want to be able to automate that and it's just not allowed to go into production if it doesn't pass all those tests right here and then we have code review so we can say this can't go into production unless

two or three other people review it and give it the okay right so no one person can go and make changes to our environment there's got to be a multi-person verification that this is a good idea and then the automated deployment so once all those things happen and once all those boxes are checked then it just goes right and you don't have to go through and and click around in the in the interface uh several times so that's that's kind of what it looks like I was going to do a demo and then uh then we realized that uh that I wasn't going to be doing this off my laptop so I recorded a video and I think this is

going to work but I'm going to ask uh AV folks here to play it and I'll just kind of talk through it but I wanted to show a uh an example of what does this look like from start to finish so um I chose a uh an example of uh of a very current attack and if you can just pause it here so uh so one note has become a very popular Threat Vector here recently the threat actors have realized that Microsoft put all these controls in place to prevent people from emailing Excel documents or Microsoft Word documents and using them for malicious behavior nobody did anything with OneNote some bad guys realized that and they said hey

we should totally use this to send phishing emails and they've started doing it so um this is from a while ago but we recreated this so it starts by creating our our ticket right so we've got this Jiro ticket that says we need to catch one note spawning shells which is one of several ways that threat actors can abuse us and you'll see here that uh I put in a quick description of what we're actually trying to catch and then uh some references at the bottom so there's one from proof point one from four core that talks about you know how these attacks work so you can go ahead and start playing again and then uh as I do that I'm a detection

engineer so I'm going to mark this as in progress and I'm going to create a new branch in our in our GitHub repo so again this allows us to make some changes to the code and then be able to merge them in later so this is uh this is vs code and I'm going to you know create this new branch and now I'm gonna go start making changes to the actual code I know that this this particular rule right now it has a bunch of logic in it to detect shells spawning out of Microsoft processes but we want to add one note but before I do that I'm going to test it once so I'm going to fire up OneNote

in our detection station and I'm trying to figure out how can I create a malicious attachment without having to go in and download actual malware and again this is a very uh compressed version of this but the easiest way I figured was just to put my own malicious link inside of the uh inside of the document so I'm going to create a shortcut to the command prompt on my desktop I'm going to look at the uh the properties here and what you can do with uh with link files uh and I'll just let this go and then we'll go ahead and can we go ahead and pause it here so if you create a link file on

Windows this is a fairly common uh tactic these days uh you can just create a link file to the command prompt and then tell the command prompt to do whatever you want to and it you know it looks normal to a user right so they click on it it's going to do this thing so I modified the shortcut to to just run the command do evil I need sandwiches and that doesn't actually do anything but it's it's a close enough approximation to um what an actual attack looks like so you can go ahead and play so I'm going to make these changes to the um to the thing and I decided I was going to change it to Ampersand here

because that's better you know formatting for for Windows commands even though it's not a valid command but anyway I'm just going to drag that into into my OneNote and I'm going to click it because I'm a user who got this and I was told to click it so I click it and I open it right now I've got a EDR sensor on this and I'm going to move over to another window and we use a tool called lima charlie as one of our EDR vendors so if you pause it here so right here what you're looking at is our Dev environment and the system that I created this this test attack on has an EDR agent running on it that's

collecting all this Telemetry so when I clicked on that you'll see here on the right there's some Json and it's kind of hard to read but basically this says that uh the command cmd.exe C do evil on these sandwiches was run from the Microsoft OneNote command so Microsoft OneNote as a parent and that command prompt is the child and that's what we're going to look for here so another another big piece of how we think about detection engineering is we want to look for these weird patterns of behavior that should never happen in order for us to find bad things so I don't care what's in that one note document I don't know I'm not going to

write an antivirus rule to catch every possible version of this but I know that if somebody puts a malicious link inside of a a OneNote document and a user Falls for it and clicks on it it's going to cause a command shell to spawn and it's not perfect because it could also be Powershell or mshta or there's a handful of other vectors but um but in this example this is what we're looking for we we really want to just know if one note causes this General process to spawn how will we how will we catch it and we can see an example of it right here that we can use for testing so go ahead and start

playing again so what I'm going to do here is I'm just going to copy this event and this goes into our into our unit testing capability here and I'm going to go back to my uh I'm going to go back to my vs code window where the the actual logic is stored and this again this is an existing Rule and it's checking for um it's checking for all these different office documents to do it but I'm going to create a unit test here at the bottom and I'm going to say this is one note spawning a shell and the lima charlie EDR tool has a really great feature that allows us to replay um a series of events in order to determine

if a rule is going to fire so I just paste that Raw event there and I say this should fire and hit yes and what we've written uh oh wait now I've got to go up and actually modify the rule right so I'm going to intentionally do this incorrectly and I'm going to put some logic in here that says for any new process that's windows I'm we're looking for all these different versions of Microsoft documents but I'm going to put it in the wrong place in the rule that's going to cause it to break everything right because I want to demonstrate what happens when things go wrong so I'm going to save that I'm really proud of

myself I change Outlook to one note my job's done and I'm going to run my unit test here and we've created a vs code plugin to automate all of this and you see here at the bottom it says we're running the verifiers for for this uh this test and you can see that you know the verifier failed because now our logic is completely broken and it doesn't do anything so ultimately um then you see every use case failed so if you can pause it there um what what I did here and the way this rule is supposed to work it's supposed to say if you see any one of these Microsoft Office types of processes

that are running with a um with a child process of Powershell or you know command prompt or whatever it may be then fire an alert but what I did when I changed this logic when I was you know acting like a sloppy detection engineer is I just pasted this new thing in here that actually changes the rules to say if a Microsoft Office document spawns another Microsoft Office document or um you know and then it sponsor shell right and that's really not what we're looking for so I made a mistake but this this unit testing framework caught this mistake and stopped me from pushing this out and breaking our ability to detect all this so now I'll go through and I'll change

it begin to play again I'll change the logic to to do what we actually meant for it to do which really is just putting it in the right place um within this rule

and we'll tab that over because the animal cares about tabs and now I'm going to run the the verifier test again and now it's going to go through this and you see that it tells me everything actually works all my verifiers passed and now I can Commit This to our to our source code repo so now I know that this is actually detecting the things that I want it to detect and it's not breaking any of the old rules that that we had in place before right so again here's an example if I was editing this Rule and I just went and um and like broke something like I accidentally deleted a random character somewhere because I was not paying

attention to my cat like jumped on my keyboard or something then I can see now again the verifiers fail and I have to go back and fix it before before it passes again so it's a really powerful capability too many people will just go and put new rules into production with no testing and it really causes issues that sometimes you just never even notice until a breach happens and you realize that this thing was broken the whole time so here I'm going to go and uh if you're familiar with uh git I'm going to go and commit the uh the uh the change that I made that way it will go into our source code repo and

it can be reviewed by other members of our team and because I'm doing um because I'm wanting to track all this I make sure that I put the ticket number that's associated with this change within my git comment so that basically allows anybody who's going back and looking at this and saying why did Paul make this change they can go back and say oh this is from ticket det-301 and they can just go search for that and see yeah the whole process right put a note in there and then I made it too long so I had to you know find a way to make it shorter uh and then this will commit and then we'll pop over

to our to our GitHub instance to take a look at that as well so if I go to GitHub I see that now there's a uh there's this new branch and I'm going to create a pull request and I'm going to leave a note here and I'm going to request that Tommy G from our team you know take a look at that and review it I'm going to assign it to myself and put some labels and really this is at this point it's saying hey can somebody else on the team go back and check my work and make sure that this is valid and I didn't do something stupid before we push into this or do

you have a better way of doing it or is there a more efficient process and this will create a pull request it's going to send an email to to our team and it's not going to let me do anything here because it tells me that this review is required and I'm not allowed to do anything else until somebody else gives me the thumbs up right so you don't want to have one person who can go through and modify that so that's that's really it when when Tom gets this email he'll he'll pop into the uh into the uh into the code take a look at it and it'll you'll see the changes that I've made and then he can either give me

the thumbs up thumbs down and then these unit tests again we hope that everybody runs them themselves and that they're doing their job but before it can actually go through and and make the change GitHub is going to verify for me that um that it actually passed those tests so if we can go back to the to the slide so I think I've just got one more and my time is almost up so this is putting it all together this is the process that that we um that we have in place and everybody doesn't have to do this right we're not saying that this is the only way to do it but I think that some of the core tenants here

are massively important if you're going to be running a security operations team and you want to have custom detection capabilities for your organization having something like this that can help um you know standardize and control how you're putting code into your production environment and and make sure that you're controlling how you're creating these new capabilities it really allows you to scale if we didn't do this then we would have no idea you know when a customer asks us hey can you detect this new OneNote thing we would have to just kind of ask around like did anybody write a rule for that or let me go scroll through all our rules and see if anybody did it whereas this allows us to

just go through and say hey let's go just search our tickets and yes it looks like we put this in place and we developed this capability and we've had it in place since you know January 15th which is when you know I'm just making up these dates but since January 15th when these attacks first started occurring you know we we created this capability or in an ideal situation we can say oh well we actually have had a rule for this since 2018 because we anticipated that it could potentially be abused then so we were ahead of the curve but having having this process instead of you know just smart people doing smart things and hoping that

everybody's doing it roughly the same way it really adds a lot of control a lot of uh and a lot of assurity right if you're a leader in an organization you want to know that there is some process that's happening and that you can actually verify what your people are telling you and you're not just hoping that they're getting it right every time so that's it any questions

hey um quick question about so I've noticed a lot of detection engineering relies on yaml based structures instead of like other structures like Json from your experience working with the ml is there any particular advantages yaml has over Json or is it just syntactically preferred yeah I I have a deep hatred for yaml I don't know why that's uh I mean yaml and Json are kind of the same thing it's just a different sort of formatting but there's all sorts of converters that will convert from one to the other uh I don't I don't know of any specific reason why that became the the language of choice for any sort of detection engineering but but it just is

so I don't have a good answer besides I I don't like it I also hate yaml I don't like anything that relies on on like spacing for syntax because it causes issues like what we just saw right there um one quick follow-up question you mentioned before that you did unit testing with containerization um how does the previous process differ from the GitHub actions because isn't GitHub actions also containerized yeah it is uh the what we had to do though because GitHub actions is good for very short-lived um sort of processing steps and what we would have to do sometimes is also and I think GitHub actions you can only use like Linux based containers to to run

scripts in a handful of languages so what we would have to do before is if we wanted to to test some detection capability on Windows we would have to spin up like a Windows VM with um with a bunch of uh uh I can't remember the name of this world right now but a bunch of scripts that would kind of initialize I can get it set up and then run the attack and then make sure that we called it in order to to do that so limit Charlie the the tool that we use there has a has an amazing replay API that just says if you capture these events and just put the Json and into a uh into a list of events

here and then we can run the um the logic against those events and it's just a quick like rest API call in order to trigger that so we found that's been massively massively hopeful and and it took our deployment time it used to be when we would deploy a new rule it would take us like three hours to get through all the the unit tests because we had to do all this this work and now it takes about 45 seconds to a minute to deploy it across all our organizations which is massively helpful especially when something's broken and you need to fix right now there's nothing worse than just like sitting there waiting on the

Wheel to spin for three hours so but but anyway that's that's the reason why we couldn't use GitHub actions is because we were doing stuff with all the different operating systems and we would have to like come up with hacky ways to um to test it and uh and also we were using Jenkins at the time which I'm very happy that that's not the case anymore absolutely thank you

foreign thank you everybody foreign