Detection-as-Code: Scaling SOC Operations

there we go hello all uh my name is AR wison as I said there and contrary to the fact that I do have quite an extensive range of shs you're about to see in this uh presentation that they mean absolutely nothing so bear with me here folks um just want to say big shout out to bsid spell fast uh this is my first talk ever and it's fantastic to be able to do that at the same place that I went to my very first conference so happy days anyway we're going to talk about detection as code my background is incident response so if anybody in here um is in the blue team worked in a sock

instant response this is going to be the uh the talk for you today so just a quick shout out to a couple of people in the industry who are absolutely fantastic so if this talk tweaks your uh tweaks your gears for getting into detection detection engineering these guys are absolutely awesome so Alex desera is an absolute wizard when it comes to detection engineering um SPL kql integrating detection engineering into organizations across the world awesome guy really really knowledgeable and in David French uh this is kind of the uh the inspiration for this talk uh David did an absolutely amazing uh talk on detection as code from soup the nuts at bsides Dublin this year if anybody

was there and had seen that talk you'll know how amazing that was so if you're interested you can just have a look at these guys after this talk highly highly recommend it so I'm going to be a bit selfish and I'm going to talk about how amazing I am start this out thank you come see me after the show we can talk about more uh yes so I've been in the industry now for about seven years I've worked in a number of different industries from Finance manufacturing chemicals um I've worked a little bit in the uh intelligence Arena as well and Consulting um but my primary experience is inent response threat hunting malware analysis and more recently it's detection

engineering so my company at the minute that I'm at Oria I serve as that lead incident response analyst there and more recently have taken on the additional role of building out threat detection and overhauling our detective capabilities so in my spare time I like to find missing people All Around the World leveraging ENT scripts custom ENT scripts and websites out there if anybody's heard of uh an initiative called Trace laabs it's where you go out and basically gamify open source intelligence highly recommend you check them out because they do a lot of good really really good charting uh and as you're about to see I absolutely love memes so and take nothing seriously whatsoever so uh um memes automation

it's kind of the uh the name of the game there so it is so let's start out with kind of the problem problem that we have here right we have a problem in the sock and if again if you've worked in a sock or you work in incident response you work in thread hunting you'll kind of know what a lot of the problems are when it comes to detection right so the first one is scalability right if you have a small team maybe you have three or four people in your stock maybe you have a hybrid model where there's only two or three people internally but also you're working with an MSP or you're even just

outsourced an MSP all together you'll know that adding custom detection adding um anything new any new content uh is an absolute pain in the ass it just is it's really difficult to do that so kind of hard to scale uh inconsistent detections right if you're going to build a new detection control the way I build one is completely different to how somebody else in this audience might build one and my peers might build one we have different skill sets right we have different backgrounds and different organizations we follow different processes so when we have an output of a detection control it's just going to be completely different there's no standard there at all cost and overhead if you have lots

and lots of detections you need lots and lots of resources to be able to maintain those and that costs money lots of it costs a lot of resources lot there's just a ton of overhead with that especially once you start getting into sort of 800 900,000 2,000 to detections in your environment that that starts the add up quite a lot fragmentation of tools detections don't just apply to Sims they apply to EVR NVR insert three lad nomenclature here uh wafs three-letter nomenclature um they all have different ways of implementing detection controls right they all have different field mappings they all have different uh interactions with their apis everything is all just different there's no one consistency in

the cyber security industry as I'm sure you all know it's a nightmare so that's a quite a big problem that we've got Paper Trails and this is in my opinion the worst one all right I will share commission to connect my device to the TV very cool um Paper Trails is the worst one so quick show of hands if this has ever happened to anybody have you ever been working on and been woken up at 3:00 in the morning because your friend in America changed the logic of a detection control which then blew up the console that ever happen to anybody I think it was you that thought we're not going to shove blame on anybody okay it has happened all

right it's a problem it happened to me so karma got me back all right um this is a big one who can we go to to see who made a change to our detective capability what was the change and how can we then remediate it from there paper trail is massive especially when it comes to playing with edrs and Sims not a lot of them have that sort of Version Control capacity uh and then I did a detection capabilities as well and this isn't really one that comes up very often but imagine you have a detection that it's really really high fidelity it always throws you out your true positive it's absolutely incredible right but 18

months down the line your organization changes you have new log sources coming in you have Old Log sources being taken out of your sim or uh your EDR vendor has taken away some detective capabilities of their logic your detection doesn't work anymore and you don't know that so that's just overhead and burden that you don't really need especially if you have a really really small team right this is where detection is code comes in so hopefully you all have kind of a bit of an understanding of why we're all here today to talk about this s there can all be solved by detection is code really biging this up so let's all get on to like the same

wavelength with this with this um detection is code is still still kind of newi is it's been around since by 2019 people really started to talk about it a little more around that sort of time period but not a lot of organizations adopt it right so because of that there isn't really a clear definition on what this is so for the sake of this presentation I'm making up what my definition of this would be but it's very similar to what a lot of people would Define this as and it's a structured and strategic approach to integrating it with software development life cycle taking your detections out of your sin pling them into a Version Control

repository where you can do a bunch of testing and Automation and things like that but we really need to kind of also talk about what it isn't and I don't want to give any false pretenses I have said that it will solve all of your problems but it won't necessarily solve all of the problems okay detection is code is not a silver bullet okay if you have 20,000 alerts that your sock has to triage every day you're still going to have 20,000 alerts that you s got TR every day um as well you're not really going to increase your true positive yield just from implementing detection as code you're just going to be recycling the detections that you

already have I'm putting them in code detection as code format so it's not a silver bullet morph say it's not a silver bullet here's an example of kind of what the detection is code workflow is going to look like so kind of the five broken it down in the five phases here right so the first is what everybody is normally familiar with is development of your detection content so we'll start out by building our search query doing the research we're going to be looking at integrating it into the SPL or kql eql whatever your EDR platform is using xql I forget what Crow strikes is called but yeah whatever it is we'll do a research

there we'll then Define our metadata as well and I'll show you a little bit later in this talk what that metadata might be but uh detection name and description as an example and then this is where we start looking at the detection is code side of the house we start putting it in our code based format so for the sake of presentation we are going to be using yaml to get another markup language uh you can ALS you can use a whole bunch of other things as well I've seen people store their detections as python um in my opinion a bit more complex to do that but we're going to be using yam just kind of give it a highlight uh from the

next part here we can then move into testing and this is where we start to solve our problems a bit more here so that outdated detection capabilities we can now start to solve this so we can lint our files right using stuff like yaml lint we can perform integration testing to make sure that that test and that detection actually works in the platform that we want to put it into um I'll strugle back to use case testing in just a little bit from there we can then move into the review phase so we can do some peer review on all of the detections we've just created one it looks gorgeous it's stored in yaml it

has all the meta data there it looks amazing and then we push it to the review phase and our senior goes actually you have 50,000 changes go do the thing before we deploy this to our platform uh so it's very very good that way Branch protection as well this is absolutely critical any software engineers in the room all right then you'll know the importance of Branch protection here uh if somebody tries to push a broken detection directly to a main branch in uh giop which is the kind of the most important Branch arguably in the in the entire thing uh and then that automatically gets push to your detection platform and the content is broken you're going to have a bad day

and in my case you're going to have a bad night cuz I don't want to be woken up at 3:00 in the morning deal with that so branch prote is pretty uh important there and then artificial intelligence I'm going to kind of leave you with this as a bit of a thought process but it's not going away we all know it we hopefully love it um but it can help us in the review phase in a bunch of different ways right we can look at um cold styling so is our yaml formatted and styled in a way that we expect so that when we pass it to our cicd pipeline it actually works um other

things like code summarization so if our detection content has a bunch of metadata in it and it looks incredibly convoluted and then the query itself is 100 lines long can we use it some sort of llm to then go okay sum up all the metadata sum up all the query and then give me an output what is this thing doing which could definitely help in a review process you still need a human at the end of that but the time saved from that would be awesome um but then also maybe something like the duplicative code clean up maybe I have a yaml file that has multiple multiple of the same field and so maybe it has two detection

names or two severities uh maybe the AI can then go okay well I've identified this problem go fix instead of the AI just doing itself I don't know if we want to rely on it there just yet but maybe I don't know maybe in 2027 um once we've done all that we've looked at the detection it absolutely is incredible we love it we're going to get it reviewed and then we're going to have the senior approve that we're going into the deployment phase and this is where kind of the full automation starts to happen right our C cicd pipeline is built out here it's then going to do a bunch of stuff that we wanted to do we

then push it out to all of the rest apis IDR platform or our Splunk instance or whatever whatever it is that you want to push it to we start pushing it out there um and then we can do more testing here as well so we can start to do some use case testing and say okay well here's my detection here's something like Caldera or the atomic framework Atomic red team framework uh find the use case and then run it make sure the detection works as we want it to and then if it does deploy it to production we move into monitoring our detection is not in production it's in the Sim it's in the EVR it's in the threel

nomenclature tool that you're all using and you all know it well uh alerts are firing we're having good days we're having bad days um from start to finish that's effectively what we're going to build and what we're going to the problems that we're going to solve with detection is code so from here I'm going to kind of break down at a very basic level what sort of things we can incorporate at the phases here so we need to kind of figure out all right Aon you've talked about all of this stuff and it's cool and you're going to solve all these problems with detection is code but what is the technology you actually need to do you know to do this

we need a Version Control System that's kind of the most critical thing here if we're going to be leveraging detection as code we then need to solve a bunch of our problems and most of our problems are solve with a version control system that paper trail thing that we talked about at the start there you go um we can solve this with GitHub there's gitlab bit bucket there's a lot of different ones out there um you can pick your poison and it really just depends on organizational requirements we need our cicd pipeline this is kind of the critical part it does all the automation does some of our testing or all of our testing or any

other uh requirements that you want to build in to your pipeline um and again there's a fair few of these that you can use for cicd there's Azure devops there's GitHub access there's Jenkins like there there's a lot of pipelines out there again organizational requirements pick your poison automated testing Frameworks this is going to help us massively when it comes to making sure that all those detections work they're all structured effectively so things like py lint and yaml lint to test your uh the structure of your file uh and also uh as I mentioned earlier things like Atomic red team framework and Caldera so kind of automated detection uh testing Solutions uh you can also integrate those at the

pipeline level as well and then I've put this part is optional but really it shouldn't be it can be because you can kind of do everything that you need to do and a detection is code cicd pipeline in GitHub actions or in Jenkins or whatever you want to do it in you don't necessarily need a low code or a sore platform but it's kind of strange how detection is codes moving now right um the security industry is moving towards low code is amazing s is incredible it solves all of our problems every single day uh but detection is code is effectively I know detection is code is effectively um going in the opposite direction where you know in not in one

side you're moving the L code and sore to reduce code complexity but detection is code is giving you more code to work with so and that's kind of where integrating it into low code and sore makes things a lot easier it's really something that in my opinion we should be moving towards if we're going to be looking at detection as code um can you just use GitHub actions on its own and do everything yes can you use times and do everything on its own as a low code platform yes um but for the sake of this I'm going to show you kind of a bit of both so then you can make your own determination as to what it is that you

want to use in your pipeline um there are a couple of different Lo code Solutions you can use or Source Solutions so we're going to talk about times today uh but you know exor Phantom Chronicle by Google there's you can kind of pick whichever one you want there they all have the same effectively the same capabilities for the most part when it comes to cicb so let's start with what we need at the very beginning right this solves our inconsistency problem right we need a schema in yaml to effectively just have a consistent standardized detection across every single platform that we have so think about what Fields do we use in the Sim that we use in the EDR that we use in

the wa that we use in the ndr like what are the things that we can just standardize normalize around and then we'll just plug that into the Amo file and we'll just use those for every single detection that we have easy um you need to Define this scheme very very well so do so here's an example of what that might look like so I will say it doesn't include everything that you can include this is very much whatever it is that you want based on your ORS requirements right so for your metadata you'll have your detection name your description um these are pretty straightforward what they are uh the author of the detection so who who built

this who did the thing um the severity that that detection is going to trigger at if it does trigger um and then maybe you'll want to include some references as well right so if we're building a detection for the newest CV that come out let's say it's called I don't know log 5j all right let's say log 5j comes out next year I'll be taking p with that day don't worry uh we want a reference to that we may want a reference to a breakdown of the attack vector and full chain so that if we need a bit more information on how the detection works we can just go and reference that that URL somewhere um also references your

minor attack techniques and tactics in this day and age should almost be mandatory if you're going to be going down the cicd pipeline route cuz then you can use that Downstream for a coverage assessment of your entire environment which would be awesome so you can kind of store those in your references there uh the version of the detection so I build the detection for log 5j it's incredible but all of a sudden I've missed something out doing my research and this detection isn't viable anymore uh my friend comes along edits it we need to you make sure that the version is updated there we're using a Version Control System but that doesn't mean you can't have you know

version 1.1 version 1.2 and then tags as well this is another thing that we can kind of use Downstream so if we want to tag this detection as it's used for endpoints it's used for Network it's used for mobile applications anything like that uh and then once we do that we can move into the detection phase so what's our query this is kind of important we're building detection is code you kind of need the query there otherwise it ain't going to work um the event name and I will just point out as well detection name and event name two different things not a lot of people think that but they are two different things so event name is what an analyst

at the other end of the detection is going to see when it triggers so Network log 5j the detection name might have some additional metadata like a uu ID unique identifier of some sort for that that protection so it might be 11 41265 uncore Network log 5j exploitation attempt so something completely different there we need some next steps okay alerts triggered what are we going to do with it and then some drill down information as well so if there's an analyst in there and they say all right well I need to go triage this I know what the next steps are click this button and the query that I need to triage with just pops up so very

important when you're creating detection content to give the analysis as much context as possible and then time stamps as well we need to create time for the owl file we need a time that it was last updated preferably if it's just standard UTC uh we need a query start and end time because we're not creating detections that are going to search our entire our entire historical database uh that would be long sometimes you will but for the most part it'll be sear to last 15 minutes schedular type is it a real time detection is it a continuous detection is it scheduled uh what kind of detection do we have uh the review cycle so if this is a critical

sity alert that we need to be on top of all of the time like log 5j will be um how often do we need to check to make sure that this detection is not outdated 90 days 120 days a year probably not a year you know what to do year and in the CR schedule it's pretty straightforward how often is this detection going to look for bad every 15 minutes every 30 minutes we all love C so do so we've built our detection we have our yaml file in place let's push it and do some fun things with it so this is where GitHub action starts to come in is anybody here familiar with GitHub

actions or has used it in the past all the software Engineers anybody who's not a software engineer one two okay two and a half okay awesome so get back is is amazing so is it really really is in software engineering you'll know exactly why this is amazing um you can effectively just build automation workflows that's what it's for um but the cool thing about it is that it integrates so well with your get repos so it does you can have it trigger based off various actions right so I've just created my detection file it's absolutely amazing it's been pushed what I can then do at HUB actions is have a check to say okay if any new

yaml file is pushed in this part of the repository that's a new detection so run all these things do all the things with it push it to my S push it to EDR you know you can you can get as granular as you want here right um and the nice thing about GitHub actions is we built our detection in yaml GitHub actions Works in yaml as well so it's not too much of a jump to go from learning yaml to build your Det to build your GitHub actions with the animal as well very very cool here's an example of what your GitHub action might look like and I want to stress this is a very basic level

just for the sake of this talk you can get as advanced or as deep as you want when it comes to building the detections so what we're going to do is our yaml file has been pushed the GitHub action is going to spin up what's called a runner uh and that Runner is going to do a bunch of things so it's going to check out our our detection repository to grab all the data out of it we're then going to use a script to pull back all of the detections that have been updated so it's going to search for all those yaml files that have been changed or added or anything like that uh we're then going

to just do some basic linting and testing just to make sure that it's structured and actually works uh we're going to link the files and if they are successfully linted and tested we're going to push them out to our 's web hook to do a bunch of other cool things with uh and then once that's done clean everything up shut down that's it so this might be a bit tricky to read but I'm going to walk you through it just real quick this is the backend yaml file of a GitHub action whenever you're creating your cicd pipeline so as I mentioned it's going to enable once you push a yamamo file let's see does this

work it does not okay well it's right and it's going to push uh start this GitHub action up when you have a yo file that's been pushed it's going to run and a b to uh based Runner and then from there it's going to do a bunch of the different steps so we're going to use a checkout GitHub action to check out the repository we're going to use GitHub scripts to find all the updated detections and this can be uh JavaScript it can be typescript it can be python it can be bash it can be vely whatever you want uh so we're going to grab all of our updated detections once we've done that we'll run through and do some

things to run the Amo lter and test them to make sure that the structure is working and it is exactly as we want it to be as defined in our criteria and then we're going to send each of those to our uh to our time weap so we'll receive a b64 encoded array from the lint detection files action previously and then decode that and then for each uh thing in that array send it to times so again it's very very basic what you can do with this or what I'm doing with this right now just as an example but you can have a bunch of different tests all thrown in here you can have a bunch

of different checks to say hey is my detection working as intended yep absolutely fire it off um you can send it to multiple different places uh and just do a lot of things with parallel work streams um but I'm going to show you a little bit more about that in times what's times does anybody here used times before I love this guy this is great software engineer you know everything that I'm going to doing here today it's great so full disclosure I'm not paid to say this I love Tides I will shill tides to the end of the day the platform is amazing the people behind the platform is amazing and they constantly roll out features all the

time um it's a low code platform uh it can effect you can effectively call it a sore solution as well um but it's a low code solution and we build automation that's that's its job it's what we do with times um and one right how can this how can it possibly boost my detection capabilities well we've already talked about a lot of the testing that can be done with it but one thing that times is really really good at is parallelization of all of your stuff because it's a Lo code sore platform um so we can do all a bunch of other cool things right we can do some health checks to make sure that

our detections are actually alive and working we can have have some deployment metrics to say all right here's my cicd pipeline I've just clicked Deploy on this detection how long does it take to deploy 100 detections to all of the platforms that I've just made updates for oh three minutes great love that love a good bit of time save and as everybody here knows leadership love a good metric so they do so this is absolutely fantastic to have it's already built in in times but there are other platforms out there like torque and swim Le and your exor and things like that as well um but then you can take that detection also and say all

right well I want to do this remediation action whenever that comes through plug it into the remediation action I want to reset a user password I want to kill sessions I want an isolate a host I want to cut off an entire arm of my business because log 5j has totally compromised it with whatever else the payload is there don't do that I'm not recommending that by the way so what I'm going to show you here this is kind of T in action with protectionist code so looks kind of complex really really straightforward so we start off by bringing in our yaml file through the web hook here at the top and then we're going to do three different

things with this again very basic but we're going to do three different things the critical path is in the middle here we're going to create our detection and our chosen Sim or EDR I'm just using Splunk as an example here again I love Splunk um we're going to check that that detection was successfully pushed and if it was great sent me a message in slack and let me know that everything is fine and dandy and Rosie and I absolutely love it uh if not sent me a really angry message in slack so I know about it and then on the right we have another work stream where we can say okay here's my anel file extract the query from that

also extract the miter data if any exist map that to a use case in the atomic red team framework and then run on my VM run it in my container and make sure that that detection actually fires and works the way I want it to work we make our HTTP request to do that this can fire to whatever you want again organizational requirements uh and then we do another check did this work okay sent me a lovely message on slack and if it didn't work shout out me on slack um and in the third work string over there on the left is okay I have my yo file that is coming in from my web

take that and leverage AI to switch everything about it based on the uh prompt engineering within the AI action take everything from my yamel file input convert it all to Mark D in the exact format that I want and then push it out to my detection Library where I can just go and type in one word like log 5j ntds you know c/ windows or have a specific ioc whatever is so that I can then just have a repository of all of my detections across all of my source systems and be like all right we're covered for this if somebody comes ask me uh and you can build that in you know MK docs and read

the docs it's a really really powerful one to have because you can just push the markdown file and it'll just work it's awesome uh and then we'll do one more trigger here anybody know what that trigger is yeah make it work does it does it work sent me a lovely Smiley message on Slack if it does if it doesn't shout at me on slack with that really obnoxious exclamation mark that everybody is so T is really really powerful here you can have seven work streams here instead of three you can have 50 work streams instead of three um really the limit is your imagination here as well um but going back to what i' said earlier on it it feels like

detection is code really should be moving towards in these platforms as opposed to being in GitHub actions now you can do a lot more with GitHub actions just because of the nature of uh the feature set there um but it won't be long until you you can accomplish 90% of that in a low code platform so it's very awesome to see um and did I mention at all that everything that we've talked about today it's all free every single thing so yamama you can just go and learn create your own files GitHub is obviously free GitHub actions are obviously free um python TI all these things are free you can go sign up the times right now and have you know just a

story there build out your own uh detection is code pipeline but you can walk out of this room now and then by tonight have this exact thing just working at home really really cool um I have a couple of additional resources around detection uh detection engineering detection is code here so if you want to look up what GitHub GitHub actions are times and yaml go check them out um I have a really really good article from David French that I mentioned earlier on this is a really really good one if you want to go home and build this yourself start to finish excellent article cannot recommend it enough um if you're interested in detection engineering as a whole detect.

FYI has amazing articles about anything to do with detection um and then if you're starting to get real serious about integrating your own threat detection team in your organization start looking at things like your maturity Matrix so Kyle Bailey made an excellent maturity Matrix on detection engineering and that you can kind of take and either run with or enhance uh to then start to build out things like your strategy and where you're going to go from your threat detection um for your threat detection team and that brings us to the end folks I hope you've learned a lot here if you haven't feel free to come to the Afterparty leader and shout out and tell

me that you haven't really learned very much from this I'll still appreciate it um but I'll open it to you all night does anybody have any questions hello

sir I think it's a good question so the qu the question was does does this show a shift to a a sockless architecture with within within your organization I would say yes and no right so I think the thing the thing with that is you're still going to have your detections at the end of the day and yes you can have as many automated Remediation workflows in those as possible and technically you could but we all we are all kind of aware that while we tune these detections to be highly granular and super focused on a very specific indicator or behavior or in set of indicators compromise they are prone to false positives I don't think we're

really at the level in the industry yet where we can say we can go truly sockless with this for for some of those quick wind detections like brute force or anything like that absolutely we dealt with that years ago but when you're getting into very bespoke Behavior based detections I don't I wouldn't personally think we're quite there yet but maybe in a couple years I don't know we'll see how technology goes there any other questions

[Music] hello you say when itss it

does that was an excellent question uh the informal that is all the time because I always do everything wrong uh but it starts when you're starting to build it out when you're fleshing out your detection as code architecture you're going to be start to think about these things right when when in the QA process is this going to break down and the reality of it is when you're building out your detection is code you build it the first time and you QA it once and usually the pipeline itself just it just works so it does you don't really ever need to go back and change it unless you're adding another feature in which case you are of course you're

going to QA again but if you can get to a stage where you're implementing your a bunch of your automated testing for your integration testing your use case testing all the different kinds of testing you can do will keep that QA Fe to the minimum now can you implement detection as code as a entry-level sock analyst yes can you implement detection is code as an entry-level sock analyst and go through all QA processes and no testing and backend engineering that's where it gets a little trickier to do which is why I'm trying to keep very basic today it's very much a look at the proof of Life of how this can work now I go out and flesh

out all of your additional features and quality assurance tests and everything else that you want from there time yeah all right thanks for coming out everybody I really appreciate you coming