Detection as Code: The Engineering-Focused Future of Detection and Response

welcome to detection as code the engineering future of detection and response I'm Jackie Beau and I will be your panel moderator today I'm really excited to get to talk to you all I'm really excited to be back at b-sides my favorite conference don't tell the other ones don't tell them all right so what are we going to talk about today we're going to talk about how applying engineering focused Concepts and like a software engineering mentality to kind of the classic world that is pretty Manual of detection and response actually unlocks so many possibilities reduces manual toil and just makes everyone's lives better so in particular what I'm talking about in applying these kinds of Concepts is and how it makes us better is it helps us detect actual serious threats to our environments rather than what tools or things that we have no control over tell us is a threat it helps us automate a lot of the noise out of our day-to-day it helps us get faster turnaround time from detection to actual response so you can do the work that you need to do to keep your companies and organization safe and also it just means you get to focus on the things that matter and the interesting parts I have strong feelings about this topic I started my career as a malware Analyst at a government cyber security operations center or we'll refer to them as socks doing manual cue based work and since then I have done detection and response at companies as big as Facebook and as small as 100 person startups what I've always found is that the way that we approach threat detection and response we're Treading Water we have so many alerts so little of them are High Fidelity the things that are actually incidents aren't usually coming through our tooling we separate out analysts and engineers and we expect them to operate on outputs of tools that they haven't written and they don't have any ownership over uh and so I want to share one of my favorite quotes which is by Cliff Stoll if you don't know about him I highly recommend reading the Cuckoo's egg he's pretty much the OG blue teamer and he has a quote that goes the first time I do something I'm a scientist the second time I do something I'm an engineer and the third time I do something I'm a technician I want to be a scientist so that's kind of the mindset that we are applying to to this uh this space so the good news about all of this manual toil and work that we do is there's a shift that's been happening and I've witnessed it over the 12 or so years I've been in this field to apply engineering focused Concepts and that's what we're going to talk about today and so for those of you who don't know me I currently am at Asana and we have started up our detection and response program which if you couldn't guess is engineering focused and so we focus on a lot of the concepts that we're going to talk about today things like detection as code infrastructure as code automation having really high quality data Pipelines and yeah so obviously I love talking about this area but I decided what would be better is to get people who are also doing this work to talk about it with me and share their perspectives so without further Ado I'm going to turn it over to our panelists so they can introduce themselves tell you a little bit about their backgrounds and why they're excited about this topic so first I'll go to you Louie yeah hey how's it going folks I'm Louis Barrett for the past 10 years or so I've been working in both detection response as well as instant response these days I work in the AI space in product security as well as Cloud security I'm basically trying to marry the two disciplines such that we get better at building detections using Ai and get better at security securing AI using the detection work I've done in the past really excited to talk to you today where are you currently I'm currently a scale AI leading both product security and Cloud security and we're hiring yes awesome Jessica oh I gotta get closer to the mic I'm being told usually everyone tells me I'm too loud um yes uh Jessica rezine I currently work at Brax as a security operations manager which includes the security detection response function as well as corporate security um and I find my I found my way to the wonderful world of security and scenario spawns I started in tech support and then I moved into a knock kind of a junior sres type tile style role and I just loved incidents like I'm that crazy person I love a good crisis the world's on fire and I'm having fun finding out like where the burning pile of garbage is but that pivot and I realized like my favorite incidents were the security incidents and that's how I came to the detection response space it's like hopping one crisis to another um and I'm particularly excited about this because I've been really fortunate in my career that when I started out at in a knock I got to witness the transition to being less of a knock and trying to be more engineering focused and then in my security roles I've been at smaller companies and had the opportunity to help build a program program which like lets me do more engineering focused stuff so I'm like super passionate like I have experienced the benefit of detection engineering like engineering first rather than the sock model so that's why I was really stoked to talk about that here today awesome thank you Julie yeah I'm Julie Sparks I'm a security engineer at Brooks I currently work with Jess and you can tell we align on a lot of things but I've been in security for about seven years started in GRC and then transitioned over to detection and response what really interests me is the fact that you can apply these things that we like learn about from other Engineers software Engineers SRE and we can take that and we can use those tools to make detection and response more efficient and give us more time to work on threat detection and work on security incidents some of the things that I'm passionate about something else I really love is that these engineering first detection response teams we tend to help each other we love to open source things we love talking to each other about these engineering problems and how we can work together to solve them so that maybe we all have the same size team but the end of the day we're able to work on more exciting problems and really optimize what we're doing and that's just been a Love of Mine love it thank you all right let's get into the nitty-gritty what are we talking about when we're talking about engineering first detection and response so first we're going to start with detection as code which I'm sure many of you are familiar with this is a switch to really owning your detection signatures writing them into a code repository using classic CI CD pipelines and the the benefit here is that you don't have to deal with signatures that you have no idea who wrote them was it a tool was it an analyst was it someone who left your company six years ago and put six random IPS into a deduction signature we've all been there um so there's a record um and so Louis I'm wondering if you could talk a little bit more about like detection as code and why it has been revolutionary yeah so we lost that segment we had a pretty small team and we had to be detect um engineering first otherwise it really wouldn't scale so for us we've really benefited from having our detections in Version Control allow the engineers to comment on detections learn from what people are building and have more visibility across the team and what was actually being done while we're trying to detect it also let us know at a point in time what were we able to detect so you could actually go back and say hey we would have caught x-ray wouldn't have caught X based on the detections we'd actually submit the source control and you know even with that being in place we could say like hey if we have an incident come up we know that we had a gap and we know how to fill that Gap because we can literally see you know what that Gap was written into Source control additionally it kind of like helps to deal with crunch as well a lot of teams are under this kind of crunch work style when you have small teams but if you're able to kind of push things through CI and test whether that your detections actually work you're less likely to rent an issue in production where you're then having to say okay this detection is failing and we don't know it's failing yeah alerts that were just generated exactly why like knowing why is crucial to detection so like we should definitely be better at that when we're building alerts yeah another point I think is really important is when you are making pull requests for your detections and you're storing everything as code you get the collaboration aspect that other software Engineers benefit from we're able to provide feedback work together become a better detection engineer because everyone on the team is looking at it or at least a subset and you guys are working together on what makes a good detection and then all of that is like historical like saved in the code like the repo request and can be referenced by people who are onboarding or who want to learn more about what you're doing yeah and I think like the big thing about detection is code is it offers consistency repeatability collaboration and things are just better when you have that like you you know exactly what's in your environment it's like just the next step to being more mature in the space is like you're not you're not expected to write something sit look at all of the data in the future be like oh what would fire and in a traditional model that's like kind of what you do you don't have the tools to test what it's going to do in sort of your seem prod and still until you do it um and it's just like a massive step forward in maturity to be able to store detections in a repository anywhere and not just in whatever logging system is running them for you totally and speaking about maturity I think I would be remiss to not talk about infrastructure as code kind of this concept where we're actually building all of our Cloud infrastructure our data pipelines our product in code so we have again this like source of truth of what infrastructure is set up what is it going configurations and I mean one of the big benefits of that is disaster recovery right like things get blown away you can send them back up but I also think for us as Defenders it gives us so many steps forward in um like our processes because if we know what infrastructure should look like when it's drifted and also like we can accurately threat models so much better and so just yeah this is this is something that's also like a little near and dear to my heart is like infrastructure is code it's just what you do now and if you as a team are not doing it you're behind your team's not technically mature you're not really sitting to play at the table with the big dogs if you're talking to another team that does infrastructure as code and you don't it's like this is just a core table Stakes for an engineering team to do and like of course a detection Response Team should be doing it yeah but there's and there's so many ways you can do it depends on what works best for you but you need to be doing it be an adult put on your big boy pants yeah it also comes out like getting the respect of the engineering teams you're going to ask to make meaningful change if you're not kind of like you know being toe-to-toe with them and their capabilities let's be frank like they're not going to respect you as fellow engineers and we can't really call ourselves Engineers within security unless we're building something contributing the way a software engineer or infrastructure engineer would right yeah I think there's also like the ability to scale up and down um depending on like your pipelines whatever alert alert payload your handling um there's a lot of these benefits that come with having infrastructures code and being able to add and subtract resources as needed yeah it makes makes your life is an engineer easy too because you don't you you have an idea of what's going on in your environment you have those yeah absolutely I remember one Friday evening of course because you should make breaking changes on Friday evenings always I blew away all of our IM configurations in AWS but thankfully terraform so it's really a savior oh yeah I mean I don't work for Hershey court but um so when you mentioned pipelines and so something that we have to talk about when we talk about detection and any kind of threat detection is we're operating on so much data so many different logs from different places and so there's this concept too that Julie we were talking about about data engineering and having like a really clear process on how you're processing all this information and storing it and how you use it I wonder if you can yeah so maybe this is my hot take but I think that detection and response should have more data engineering skills whether that's having a data engineer on your team or just having detection response people that are familiar with data and can work on these pipelines but at the end of the day you're not going to have good quality detections if you don't have the data to write those detections on so data quality where you're able to ingest logs from understanding what those logs are and for a lot of teams that also means understanding the load of data you're going to be bringing in and whether it's cost efficient for what you're going to need in an incident or whether you're going to need for a detection further on so I think focusing on data engineering as a problem and something that can really up uplift your detection work and making that like a foundational piece is really important at Brooks we we open source substation which is a way to deploy serverless data pipelines to collect logs we are able to collect logs store them pre-process and post-process after they're enriched and so that gives us the benefit of being able to store the data have it raw but also be able to perform any Transformations we want on the data so maybe we want to add certain role enrichments we want to like make custom hashes based on certain patterns and then we can utilize those in our detections but all of that can only exist because we have control over our logging pipelines and what happens to our data and does anyone have a story to share of like what it would have looked like trying to do that if you had a scene instead of like controlling your own data pipelines it wouldn't have happened because you just don't have access to like some vendor has sort of given you this box and they go here you go it does all your logs it does all your detections but they don't give you that control to get in and actually see how well they're handling your data and they might not be doing it very well and you could just be stuck with whatever their slos slis are yeah and I feel I feel like we've talked a bit about kind of this like build versus buy um mentality and I personally feel like there is a balance between sometimes you do want to buy vendor Solutions but you should have a framework or a way that you're thinking about what you're what you're building versus what you're buying and so I'm wondering for any of you have like stories about where you made that trade-off how you think about it yeah I think there's this underlying concept that you have way more flexibility to build if you have teams who have also engineering like like skills otherwise if you're just if you have a team of analysts and they're working within a tool you're kind of just going to buy a new tool for them to work on but if you also have Engineers on your team who are building the tools that you use to do the detections like that's kind of where the magic happens and build isn't even possible in a situation where you're not focusing on engineering first no 100 true I think we also need to like kind of nudge vendors to build tools that support the ability to extend and integrate because without that capability we're never going to be able to build anything that's hybrid and we'll always be in this situation where we're you know at the mercy of these vendors to give us you know data fields like strings where they should be IPS and everything we know that should be right and they're just not doing this for us so they need to kind of move in the direction of detection engineering and you can just fix it yourself if you have yeah not a string should be but yeah let's do it I'm going to give like a big plus one to that extensibility the ability to work with a product and have it be able to be molded to your environment and have control over like what's going in and coming out is just revolutionary rather than I mean I'm not going to name any names but just like buying you know your classic big name Sims and you have to learn their proprietary you know query language and you have to do everything in this very specific way that probably doesn't really have anything to do with your environment and additionally like those skills don't transfer right like exactly you get this like lock-in or where you as a professional now can't grow because you've only used vendor X and vendor X's skills are not transferable to vendor why critical think we're starting a revolution I think we're like looking at fictional response engineering teams are going to take back the power from vendors because if they're not giving us what we want we can build hybrid solutions that take over that and take note vendors in the room yeah get scared or work with us or open up your platforms so we can really yeah really customize them yeah I think that's one of the successes I've seen a lot with like our open source like toolkit is it doesn't do what we want let's change it like oh we have a cool new idea like we have uh checks like you can't ship a bad config for this environment anymore but we could only do that because we built it ourselves not because a vendor would have allowed us to do that yeah I did want to go back to your comment about how sometimes we have kind of this ceiling that we put on when we have analysts come in and they're only working on these proprietary tools and they become experts in one you know query I'm really trying hard not to say a vendor name right now but as a hiring manager and I know we have like other like people involved in this when I see a resume and your experience is nine years working with one proprietary scripting like query language that just has to do with this one tool I can't see how I can slot you in to the like the current kind of like engineering focused approach and I would say that is not dunking on the analyst that's dunking on the companies that don't give the opportunities for you to have extensible skills [Laughter] word all right I want to talk a bit about the noise and prioritization and so we know that working and threat detection response where our scope is so uh it's so large and I mean Jessica you talked about you own corpsec as well so it's not just incident response it's not just threat detection it's also corporate security and asset control and the the issue too is right we're not working on systems that are static or not changing we're not working with threat actors who aren't changing what they're doing every day and so there's so much that we could focus on and there's so many things that we could focus on we have vendors we have organizations like miter telling us what we should be focused on we have Frameworks that were written 10 15 years ago that tell us we need to be focused on certain things how do you operationally know what to focus on when you're building out your detections and I'm gonna throw it to you Louie so I think it really comes down to threat modeling your environment and using Frameworks internally that makes sense um one th