Josh Brower - Applying Sysmon-type filtering to Elastic Agent Process Auditing

all right we're going to keep the morning going uh we've got our next excellent presenter Mr Josh Brower he will be presenting on applying sysmon type filtering to elastic agent process auditing and uh I know nothing about that so I'll turn it over thank you so much and I apologize I need help with titling my talk so next time I will try to do better that was not the most uh fantastic title in the world so uh again my name is Josh Brower um first off thank you so much besides Augusta for wonderful conference uh besides Augusta is always one of my top security cons to come to every year and I really appreciate the opportunity to come and speak to you all this morning uh you can catch me on Twitter at defensive depth or X whatever it's called um I've been in infosec a little over 15 years and uh primarily focused on network and endpoint detection with some training on the side uh I joined security onion Solutions about four and a half years ago focused on engineering of the platform as well as training really enjoyed it Security on we have a table I think over here so feel free to stop by so today we are going to be talking about that long title if uh you don't really get where I'm going with this no worries I'm going to lay out some context first and then we'll get into uh the elastic agent process monitoring so let's flip over to the miter attack framework everybody is probably familiar with this library of tactics and techniques that adversaries can and do use against us as Defenders and if we drill down into something random I don't know let's do remote external Services we get a little uh some context around this particular one we scroll down we get examples of how it's been used we get how to mitigate it and we also get how to detect it as part of detection of a particular Technique we have data source we have this data source column data sour are you know where can we pull these logs from to then write detections against to detect that particular technique what's really interesting is uh Roberto otherwise known as cyber came out with some fantastic research and visualizations in the last few years and he took if you can see I don't even know if my pointer is working here let's try this yep it's okay so on the very bottom we have the count of sub techniques all right and then along the other axis we have all the data sources so up and down right here we have all the data sources and what I find really really interesting is that the top two command execution and process creation all right you can detect what this is saying is that vast majority of the techniques in miter attac you can detect through process um creation events and command execution which is all part of this overall data source of process auditing or process honoring okay now you can't just use one data source to be clear context is King context is always a part of of uh detection so we want to have more than one data source we want to have more than process monitoring there but process monitoring is very key is that making sense so far yes I love I love interaction yes thanks Wes Perfect all right so that being the case what are some tools out there for process monitoring we have uh ones that are built into the operating system like for Windows we can enable process auditing with uh gpos the newer endpoint security through Mac as well as audit D with Linux we then have third party tools like velocerptor and O query free and open source tools that can do process monitoring process auditing I kind of use those words interchangeably and then we have free but not open- Source tools like sysmon and elastic agent with the um endpoint defend integration do anybody here deployed any of these tools in production okay someone tell me and this is for a giveaway what is one of your top challenges when deploying a process monitoring tool in production tuning okay what do you mean by tuning okay so much noise so here is the first giveaway this is lockpicking all right so perfect that is exactly what I was looking for and I think this GIF kind or GIF depending on which way you go uh I think this really sums up what happens when you enable something like process monitoring okay it's a spill W from a dam and it is just a massive amount of data flowing through your system and when that happens it you know it can cause performance issues on the endpoint it can cause performance issues and trying to bring those logs back to your backend system you have to parse those logs and then your analyst has to try to sift through all those logs and there's a lot of uh legitimate you know benign logs in that but there's also a lot of logs that may be malicious and it's hard to sip through all that when we have so much data coming through okay so did I hear question yes I'm sorry yes yeah I see so elastic is on the right so you're saying it's too much data I got you and you know I gotcha I gotcha so I'm sorry I'm very slow sometimes so um so from here so from here what is one of these tools uh someone already mentioned it who pioneered a particular um feature set to help with this uh this overwhelming amount I already see a hand that's not fair all right yes who pioneered a feature and what was a feature to help with this overwhelming amount of data back here yes yes so Sison you want to come on down here because I'm always afraid of to get your you do I'm always afraid of like the mic feedbacking or something um yeah you're welcome so cismon pioneered a feature that allows you to build a set of configuration rules to include or exclude data this is just a small snippet of what that looks like okay it's written XML format and you can simply say like command line condition is this exclude that data and I don't have to have all that data then in my backend system and there are three well-known ones are configurations out there Swifton Securities Florian Ross um uh Fork of Swifton security and then Olaf which is a bit more modular it's a different take on it all right now this would be great if this talk was about cismon but this talk is not about cismon this talk is about elastic agent with the defend integration so let's go ahead and flip over to My Demo of that and we'll see what kind of options we have for filtering that kind of data in elastic agent now I am using security onion version 2.4 we integrate elastic Fleet with elastic agent so you'll see some elastic stack um as well as security onion inside the stemo while I'm doing this does anybody actually have elastic agent deployed in production right now so we have a few hands okay anybody using the defend integration with process filtering I see two hands three hands maybe okay so this is my instance of security onion installed on the local VM and if you can see this is an example of the process logs that I pulled in yesterday from a window Windows box with elastic agent on it so if I come over to elastic Fleet uh elastic Fleet is the management uh server as well as application to manage all of the elastic agents elastic agents are simply um uh installers you install elastic agent on your endpoint and um elastic agent is like the next generation of all the Beats so instead of having file beat and wind log beat and all that kind of stuff it's one unified agent you deploy it to an endpoint and then you build policies for for instance we have one here called endpoints Das initial and that has different Integrations that you add to those policies right now we have Windows and system those pull in like Windows event logs o query manager that allows us to run o query on the npoint and then we have the elastic defend integration and if I click on that and scroll down we are using the free version in um in security onion you can pay you can get some other feature sets with elastic agent but right now just with a free version you'll see that we can collect a lot of different type of data very similar to sysmon on the type of data that we can collect and that's not only for Windows but for mac and Linux as well now if we want to try to filter this data we can come up here to the uh event filters manage event filters and it's going to be great okay I'm going to add my first filter this is going to be great add event filters name test conditions Windows my field well obviously I'm going to filter out now process well let's let's try to name that right process and I cannot spell let's do process arguments okay all right what do I type in the value field I don't know like I've run Windows for many many years but I'm not a expert on what should or shouldn't necessarily be there right um and that's kind of the the right now there is no way to prefill all of these filters inside of the elastic agent event filters okay and that's I'm glad with that we have it there but we need a way to pre-fill them because you know if we see something high volume in our environment we could certainly do that oneoff and filter it out but there's a lot of legitimate stuff out there that we can filter out that I just have no idea because that's just a lot of stuff and I don't expect all of you to know all the internal workings of Windows Mac and Linux so you can build out all of your filter exclusion list right that's just not feasible for all of us so let's think for a second where is a Community Driven set of filters that have been used in production for many years and contributed by over 100 people what's out there already now you can just shout it out I'm sorry yes the cismon configuration right so if we come over here to here is a floring Ross uh cismon configuration all right problem though it's an XML format you know how in the world am I going to like copy and paste that into uh the um the web interface for elastic agent that's just not again very workable okay so what I did to solve this is I wrote a script that um that grabs the all of the XML all of the filters that I want in here and converts it into a way that we can import it into elastic agent into that exclusion list and here's how that overall process works so we first oh we uh extract the filters because there's a lot of different filters in those XML um configurations so extract the ones I want I convert it to yaml so there's a yaml document with all of those filters and then I take um all the Amel clean up and convert it to Json specifically for security onion and elastic and then I create the filters via the elastic API the reason why I converted to yaml originally is so that in the future if you all want to take that yaml document and convert it to another backend um to another system out there you can do that without having to worry about converting the XML originally okay so that will be available that yamel document you guys can take that and use it wherever you would like so let's take a look at make sure this actually works live demos always fun all right uh so I'm going to first run it locally so I have a cismon export right here export. XML and I have my convert filters uh script so I'm just going to say uh the convert the filters and you got to make it look like you're actually doing something right so I just put a bunch of stuff know it's actually converting it it's going to take just a second to run this I think it it pulls out it'll tell me all the way at the Top If I was smart I would have put this information at the bottom of the script but you know all right so it extracts all the filters all the patterns it finds it extracted 286 filters and then uh converted it to yaml so if we look at the yaml document there should be a process filters. yaml all right so it's just a simple anal document that has um image Dash you know random uh ID and then the Target Field um image condition is the pattern so you can see what are you know what the filter is the image if the image condition is this pattern then go ahead and exclude it the source comes from the cismon config and the context this is the XML uh comments that was existing in there and then from there we should be able to look at the uh converted filters this is just straight up Json or Json uh depending on where you come from and uh this we can import directly into to the elastic API and we should see a bunch of filters uh show up I'm going to come over to Security onion and I am logged in I'm going to go ahead and run the so elastic Fleet load event filters uh yes we got this it's going to take a little while because it's each for each uh object it's going out and creating that filter in uh elastic so we should start seeing filters show up over here so I'm going to refresh and so far we have a few so we created two seconds ago so we have a comment Microsoft Office licensing if the OS is Windows the process process executable is right here and the v. data set is a process event then we go ahead and exclude it all right now there are some caveats here um this was fairly easy to do for the process filters because uh they are only exclude filters in sysmon uh elastic agent with the defend integration their event filters only support exclude right now whereas cismon does include and exclude so you have a little more flexibility there so for some of the other events like file events or DNS events process events uh those cismon configurations include both include and exclude filters so it's going to be a little more dicey to try to import them you know cleanly but the process filters we can do pretty easily and again I implemented this um a couple weeks ago uh on some of our production systems and our Windows uh Windows laptops and you could very clearly see the difference in the volume of logs that are coming in because it's just filtering out some of those very common benign processes any questions or comments so far I know it's almost lunchtime we got this we got less than five minutes all right so takeaways from this morning from this talk first process auditing for the win all right process auditing process moning is a really key uh data source especially from a detection perspective so we should be trying to do this as much as possible but you know what's even better than just process auditing is process auditing well first what's the challenge right lots of data coming in process auditing plus filtering for the win but what happens if you're not running something like sysmon then let's go ahead and add in process aing plus filtering and reusing some of the Community Driven filters out there and bringing them into whatever platform we're using all right I do have all this on a GitHub repo right there that URL feel free it's obviously proof of concept right be careful running it make sure you look over the code that sort of thing um but it's all available right here on the GitHub and um you can take it the amble document is up there already you can take it and reuse it I think longterm what we're going to do um we will probably ship these um filters in security onion by default so that when you deploy elastic agent through security onion it's all going to be there you don't have to do a thing and the sysmon configurations at least for Windows process creation events are fairly stable at this point so we will update them as needed but I but I don't see a lot of um a lot of churn happening with those at this point and that's it for my talk yes I see a question so your suest sure yes now if they I have no idea about the product but if they have the ability to somehow filter it and then import filters you know finding a way to get the the these cismon filters in a way that's consumable by that yes still using sure yes yep yep yes another question don't know probably not sure sure yeah I wouldn't be a good Tech person if I didn't say it depends yes thank you for the reminder the question is there's always push back by the way feel free to leave grab some lunch thank you very much for coming we're got a couple more questions here the question was it's easy to get push back when you try to enable something like process monitoring because of all the added data inest that you have especially if you're paying something like Splunk fees or just in general the performance uh requirements right and so the question is what is um you know how much extra data do you see if you're using process monitoring with the filters okay how much extra I would say generally speaking um I can't speak to your normal Baseline but I've seen a reduction uh in using cismon in production um of at least 30 to 40% if not more so the normal process filtering data that I or excuse me process data I would be shipping um I've seen a reduction of 30 to 40% in most use cases with the cismon filters now now your Baseline who knows what that looks like and that's significant when you're talking about larger installs of it any other questions comments or snide remarks we can do that too yes so the point was and it's it's a good point is if you're if you have problems with um uh retaining that much data the the point was even if you just retain a couple weeks worth that's better than uh that's better than nothing because of the the the level of visibility you get with process monitoring all right yes and again the after yes yes yes is still a big yes yes yep all right is that it for time are we good okay all right thank you all very much appreciate it