Linux Monitoring with Elastic Security

thank you thank you uh so I've met most of you before uh if not I've been out at the table all day so please stop by even after and you come get to know me my uh so Cliff worth and there's a few other people from elastic here so we're always willing to answer questions and we're here for you guys today we're going to go over a little bit of Linux and elastic and specifically we're going to be going over extended Berkeley packet filter and how that is helping elastic security and how you can actually see more inside of your Linux distros in your Enterprise when collecting with the scene so here's a little bit about me I'm a Consulting architect for elastic on the federal Services side um 14 years in the Army eight of it doing incident responses cyber security mostly focused on the ICS scada side hmm I'm a data scientist by trade I really really enjoy python rstudio I've recently got into Apache spark been doing different things in in the data science area here you can follow me on Twitter you won't see much there except you know non-controversial opinions about you know Dallas Cowboys are the best team in the NFL mma's the best sport things like that nature and then of course here's our a link to our company all right so ebpf before I get into what extended Berkeley packet filter is I wanted to ask if anybody else here knows what Berkeley packet filter is I saw your hand first purple purple shirt I think yep yeah exactly yeah so one of the you can obviously get yep hand you one of those so one of the best parts about Berkeley packet filters what it does is it's a kernel level program that will essentially sift through different packets as you absorb them on the network very very quickly because it's at the current kernel level well what we have today is called the extended Berkeley packet filter not really related at all evpf is written in a completely different code has nothing to do with BPF whatsoever outside of BPF uses it and you have to call eppf with a Sysco from Berkeley packet filter so it's all of the codes that you would run are written in C they are loaded with different syscalls you'll see right here on this slide is just some technical information about how ebpf can be called by BPF so that's essentially what this slide is is just to kind of walk you through it you can different ways to call different compiled C language that you can write your own there's a community of dis there's distros all over the community both in elastic and outside of elastic that are using eppf as well so you'll also notice that my name is not Tammy and Tammy could not make it because of hurricane Ian so occasionally I'm going to reach over here and I'm going to just Mark out where I am I found out 24 hours ago I was given this brief so give me give me a little bit of a break here Okay so this ability through eppf a quick little poll here who has done an incident response and inside of an incident response who has encountered a Linux box okay so not not many of you what are some of the different logs and things that you're going to want to grab just just give me one yeah for sure for sure but specific to Linux audit logs is what I was looking for so traditionally with Linux before edrs and stuff started making their breakthrough into the Linux distros you had to go through audit logs you had to go and collect different data sets with like just running PS Tech elf tracing those pids to ppids and it was just horrible it was it was awful so what does ebpf do what does it actually do so ebpf is is a tool that actually hooks into uh into a process as it's running inside of the Linux kernel so it's compiled in C and one of the benefits that it does is as the process is actually being executed it is caught in the Stream of the execution it is not caught afterwards so typically with like audit D or any other kind of logging the event happens and then it's logged by the OS or some kind of third-party application logging whatever that may be with eppf as it's occurring it is being logged through a stream in the hook and we're going to go over that in just a little bit try to keep it lower level because that's where my knowledge base is on it so hope points yes so this is what we were talking about you can see as different different syscalls inside of the kernel are executed you can actually pass eppf programs to it and again these can be different things inside of Linux there are a ton of companies that are starting to get into external BPF uh or extended BPF programming and there's a ton of different ways to actually use it so what we're going to get into here in just a few slides is I'm going to show you how elastic is using it and some of the benefits and Pros that I've seen uh that I've experienced on an actual incident response so what do I get with this so you're going to see here in the few slides that the system call data and the uh is combined with packet information so what do I mean by that so you're going to see audit data process ID data and it's going to be in human readable format another problem that you've also seen in Linux is sometimes it has an issue with uh metadata sometimes you will see the the aggregation of metrics is a kind of a problem so these are kind of some of the issue solutions that you're going to get as you as you implement this inside of elastic so one of the things we didn't talk about is performance so with abpf as you compile it and see and you run it there's a couple of things that will happen either a it will run and it will do what you told it to or B it will error out and break however it does not break your system it doesn't run in a part of memory that will actually kill your your system so one of the benefits is that it's safe as you were running it it will hook into whatever process that you called and the kernel and it will actually just either fail or succeed so that's kind of what we get with the reliability and compatibility issues um so yeah minor attack mapping here is where it starts to get a little bit confusing so there is a couple of uh Engineers known as Judd Salazar and Italia Rica ivanko both of these guys have mapped out where evpf can solve these initial techniques you'll see that a lot of different organizations including uh maybe it before in the brief before they use the golden standard that is miter attack Matrix to map out different attacks and when you're mapping out how you're going to do your security operations including scene what you want to utilize is some kind of methodology to capture as much of what the adversary actually does on a network so whatever it may be your seams your edrs whatever kind of solutions you're trying to build your defense and depth you always want to kind of use the miter attack Matrix or some other kind of methodology to map out and get as much of this as possible well what we can do with this eppvf is actually map most of those miter attack Frameworks inside of the Linux distros and that's what this is breaking out and that what those two authors have done for us and again if you at the end of this if you need their names or the book I'm more than welcome to give it to you in at the tables okay so audit B one of the we've already talked about it anybody that has been through it knows that audit D is a pain in the butt uh I mean just look at it right you can only grip so much so this can be really really difficult to look at it can be really really difficult to process the data what of the metadata here is actually usable which of the metadata is key these are things that just make it really really difficult to to process this and I'm going to show you here in a second why using extended Berkeley packet filter is going to be usable or more useful than this right here so this is just your straight output of these logs and it again if you've ever done this it's incredibly difficult to process okay so this is where I can start to brief you a little bit on what we're using inside of the elastic stack so what we have here is a capture of some of the processes and the ppids and the metadata associated with PID and ppid inside of your elastic distro so what makes this usable what makes this better right so one it's it's in a GUI format you have different uh you have the metadata there you have alerts that you can create inside of the elastic stack as it relates to this alert to this event and that can be for anything uh right what you see right here is just somebody ran an LS that created its own entry into a process and then was listed inside of elastic stack uh again metadata that doesn't require PhD metadata can sometimes depending on what it is that you're using can be very very useful or completely useless so it just depends on what your organization is using it for but it's essentially right here I just have a quick print out to show you the host name the ID IP things of that nature these are all things that you can configure inside of your extended Berkeley packet filter here is a session view so this is an investigative Tool uh that we actually have right now inside of elastic stack so this will break down essentially all those different logs that we were talking about inside of Linux and it will break it down into a streamlined format for you to show you exactly what objectives an attacker or maybe just a bad user you did on an endpoint so you can see here by just this example we saw that somebody and this is an alert that we actually created somebody created a hidden file and directory if you click on that it will actually give you in the metadata below how they created it what commands were Crea were used to create it and essentially what processes what users anything of that nature so before Linux doesn't really do a good job you're not bringing up process hacker or a process Tree on Linux right it ain't happening with this you can actually do that inside of the elastic security stat foreign human readable process yes [Music] so it depends on what your organization wants and how you're going to collect that data so you can use we are actually in the process of developing the elastic security agent that will actually collect some of this data for you you could use a beats it by all means if you want to it just depends on how you want to configure it whether you want to do Beats or an agentless uh you know like uh EDR solution so it's it's up to you really yeah because most of the capabilities are things that we typically had right right so those right now uh for the elastic security agent are coming by default however there are additional things that I'm going to point out to you at the end we have a GitHub it this project will be open source for the foreseeable future and there are many many use cases because it's going to depend on what kind of data set you have but people are already starting to write different different C code because that's that's hard to just kind of do unless you have a background in it so uh like some of this dysfunctionality that's already built into the security agent some of it's not so runtime security is your security camera so what we're getting at here is this is a little meme to kind of point out that elastic considers this a runtime security issue because the event is happening in stream we are considering that a runtime security thing so as the event is occurring we are ingesting that data and making it available to the analysts so in conclusion uh audit and compliance logs have come a long way ebpf is something that we are using to develop future capabilities inside of our elastic instances because of the power it has with the endpoint Linux distros so one of the things I kind of like didn't really discuss with you guys is right now there's only a couple capabilities inside of Linux that give you the capacity to look at this kind of data set right so d-trace was one that is that only works with Mac OS instant Solaris I don't think anybody in here is rocking a Solaris machine right now that there's a few others like peat race things of that nature but there was nothing that encompassed the entire Linux distro that kind of gave this ability to look at a process right so if I'm an attacker and I inject some a malicious something into a process inside of a Linux box how is that going to get seen how is that going to get viewed by the analyst right outside of doing a memory capture and doing a hash of that individual process how are we ever going to see that this malicious process was violated and so that that's essentially what the power of this does is in real time it will actually create and I'll go back here real quick right here it will actually create a fork of that process so when I inject something in malicious into process whatever that is hostname right there it is going to absolutely create a fork and I will see when it changed what the user did and what the command line entry was for that and this is pretty cool for Linux because before like I said none of this really exists natively to Linux uh another thing is is that we talked about eppf is that it doesn't utilize a lot of memory it doesn't if it fails it it fails safe right it doesn't fail open it doesn't cause any outlying effects on your system so as there's no big deal there's really no effect to otherwise your your actual Linux box and the business processes you have for that Linux box so you don't actually incur a lot a lot of performance issues um again thank you for bearing with me as I struggled through that presentation uh again I'm my name is not Tammy it's Mikey if you have any questions please by all means go ahead and ask them and I wanted to thank b-sides for letting us continue to give this brief thanks but his point like how crazy to look at that second so ebpf is actually written into a lot of different tools You by no means have to just go out and get elastic to do it however I work with elastic so please download it uh but by all means you can use any tool to use eppf it is a industry-wide tool and it is an open source program I believe we own the company that created eppf but right now from what I understand it's going to remain an open source tool for a while here yes agents yes and then we have what you just said the endpoint security or thank you uh we have elastic agent and then I think you mentioned the word uh elastic endpoint security can you clarify licensing and the functionality you're talking about here how does that fit into elastic agent and what does that all look like for sure absolutely so elastic offers primarily right now for a security tool we actually offer what's called in-game so elastic purchased in game that is our traditional EDR that is what we are offering as an EDR fully functional out of the box right now we are in development of an elastic agent whereas elastic security agent that will function with the elastic stack so that it's kind of like all in one if that makes sense um but that's kind of what we're getting at uh I believe that has started to be offered is the security been offered I don't I'd have to check and see if we're actually offering that out loud out wide to the community but I know it's in development at the moment yes that is how we're going to use it and it'll like I said it'll be by default that some of these tools that I destroyed today will actually come with it by default yes and I saw the slide up there and I messed with not a d a lot it's already on okay the sequence numbers that built that come in audit D will paint a picture of an initial injector whatnot how is I don't understand how the uh the uh the ebpf is is different from audit D I know Oddity is very convoluted to look at and it's hard to understand but are you guys just kind of pulling that all together and producing it out because you say you're no longer using Oddity anymore right so let me clarify that statement I am by no means telling you not to collect audit data please leave that logging on that would be bananas if you turn it off uh but let me clarify so what I'm talking about is as the event is actually occurring typically the OS at a certain layer will after the event has already occurred we'll create the log correct this is going to be in Stream So as I actually do anything on this system inject into notepad.exe right it is going to immediately create a sub log into elastic that eppf I'm not saying it won't be logged into audit log yeah yeah I'm just saying it's a lot easier to look through audit log oh so uh the polar different there's Polar Opposites between like the really popular Splunk and elastic and the complaint is sort of by people who are used to Splunk when you try to introduce them to elastic is oh this isn't even a seam this is a log management tool and I don't have any features is elastic looking for any kind of way to like bridge the gap like default dashboards and stuff like that so inside of security there are a ton of different use cases for dashboards right and by default you could say hey look you should bring up all anytime a user uses FTP or whatever right there's a million use cases one of the benefits I'll say when it comes to elastic is that there are enough people that by now that have used it to where there are plenty of githubs for you to go out and grab uh formats for dashboards I just I can't see saying that elastic is behind in that only because so much of the community is already using it for security but you're right out of the box it doesn't it is an empty elastic instance right it's on you to configure it that is by default what the because there's a million use cases for elastic it's a data science company right yeah yeah well again thank you very much thank you for bearing with me as I struggled through it thank you