Scale Your Auditing Events

thank you hi everyone let's see what we can take this so often I guess this is kind of arson we accept that the world is not perfect and security is not really perfect and we say this is fine but at some point maybe you reach this conclusion where everything is on fire and everything is terrible and there were enough recent events when this happened and it's kind of like we don't want to get to this point and obviously there is no silver bullet to avoid that and everybody trying to sell you a silver bullet to make that happen it's not it's not going to work but I want to look a bit at what you can do

around auditing and figuring out like what has happened to your system and probably what hasn't happened or ideally what hasn't happened to your system so if you have any questions in the middle since time is not so long just post them to slide oh and if we have time at the end I will answer them live otherwise I will just tweet out the answer so if you go to slider and my Twitter handle then you can just post a question there just write any sensible question to that and I will come back to that at the end by the way did anybody figure out why I'm using that Twitter handle and that name for slider it's kind of like in the

first slide as well did anybody see the connection to my name anybody old enough to remember rot13 where you rotated the letters so if you take my last name and rotate the letters by 13 this is what you get and the nice thing about rot13 is if you rotate by 13 again you're back at the original that's kind of how I got to that anyway so let's jump into oddity who's using oddity already actively very few okay so oddity is this is from the main page of oddity all it is the user space component to the Linux auditing system and it's writing out records to disk what has happened and you'd basically define it to configure or to

collect various pieces of information that are relevant for your instance so for example you could access or monitor the access to files or to the network or to system calls or whatever user action people are taking all of that can be collected by oddity and can then be locked what oddities by the way not doing is it's not blocking anything it's not like SELinux it's not securing your system it's just collecting what you are doing is just keeping track of that and it pretty much looks like this you have an application in user space and it calls the kernel and in the kernel you have these system calls and you have user task and exit

those are the three hooks basically which you can use and if you have you have an exclude rule and if any of these actions passes through the exclude rule then it will be collected by the auditing daemon so basically you define your rules and everything that passes through your rules will be collected by Oddity and will be written out as a log file which looks something like this let's make this slightly larger so you have a report if I would be in with the right user obviously a non root user will have a hard time with that one what all report basically is doing is it's telling you when it ran so this was only

running like a minute or so but you can still see it's collecting a lot of informations basically parsing that file the audit the log file and you can see for example we made three configuration changes we had 14 login attempts while it was running we have 3 uses in our system in total we have 5 terminals we have 3 host names all of that is being extracted from the logs what how do the logs look like you can use all search and then for example raw to see the raw events that you have in the background and what is going on here so for example here this is one entry and you can always see type some demon ended and

then you see there was a message oddity this is the UNIX timestamp and this is the unique ID of the operation and then you can see the operation terminated some process and the whole thing was successful you could by the way have the same time stamp and ID if multiple exclude rules would let your process pass through it so we saw the three things where you could pass from the colonel to the logging if two or three of them match you would have the same time stamp and ID for that and that is one of these elements and you can see this is another one here the user started something and here we had some

some reference that users should have been referenced something and all of these are kind of the raw events that you have you can also use that to filter for example you could say I only want to see the successful events and there will be quite a few successful events you could also say you don't want the successful ones I see for that one we will need sudo as well and then you can see we didn't have any unsuccessful events in our log here so all of this has been collected and this is working reasonably well but you can see the lines are always very different like this line has totally different parameters than this line here and if

you want to centralize that because you probably have more than one instance it could be a bit of a pain to actually do that so how to understand the logs there is a documentation from Red Hat which actually describes like all the elements that you can have in there I could basically show that quickly as well so if you had over to the right of documentation they have a pretty good description of all the elements in that log and how to see how that's working out and if you want to define the rules in the github repository of Oddity you also have a couple of rules which you can define what oddity should be doing

for you for example we will get back to that one for a power abuse where basically a root user is accessing stuff in the home directory of a not root user and this is one of the rules that Oddity could be working with for example so you can see always when they process exits we want to monitor the home directory if the user that is calling that has a low user ID for a user for in the developer group or like in a high user group has is the owner of that then you want to basically lock that event and we put it under the tag power abuse and this is how you could work with that and filter

down on that later on so this is one of the rules that syntaxes may be slightly arcane but you have a couple of examples in this repository here that you can just reference and try to get started with one thing that is still a bit work-in-progress are containers so if you have a namespaces and are using docker saying like which event came from which container is sometimes a bit tricky and it's a multi-step approach and it still work in progress for all Italy here so did the general idea now is we want to collect all of these things and we also want to centralize that because you probably have more than one instance and you don't want to have

it just in one place so how do you centralize that that is where I come into play so I work for elastic the company behind elastic search logstash Cubana you're probably using us somewhere for logging already and we also had that neat where we wanted to centralize our security events for that and how to do that I build highly monitored hello world applications and I want to show you how to do that very quickly today so I'm I assume some of you are familiar with the infamous Alex tech elasticsearch la Cubana which is like one sitting on top of the other we've added some more components later on so we have something called bits now the problem with Beats is there

is no B nelq so that's why we call it LTP or Belk which looks something like this you can see it's a B and it has the elk horns but marketing didn't want to stick with that because we always are about scaling and this isn't very scalable if we have another component and a letter we need to redo the entire branding so we just call it the elastic sack but doesn't really matter here and you have this stack then and what the idea is you have the beat which is like a lightweight H in the shipper which is written in gold so you have a native binary to a forward your events store it in elasticsearch and then you can

visualize it in Cabana to centralize that and all of the stuff that I'm showing you here is open-source you can just use it change it to whatever you want the first thing we try to do is we try to write the module for file bits so we have something called file bit which is basically tailing files forwarding them over the network and centralizing them the main thing is we always need to parse the log events and if you look at the raw log events that we had here you can see these are probably very hard to centralise because all the lines are totally different and how do we actually parse out the rate at the relevant

information we write the regular expression to do that who likes writing regular expressions anybody okay it's the Stockholm Syndrome right where you got so used to writing regular expressions that is just part of how you approach stuff and for this one here we we did build that module and we collected some good information out of that but it was always very painful to write those regular expressions um just to give you a quick idea of what we are collecting in there by the way so let's head over so this is based on our auditing event since I've stopped all it D and I only ran it like 12 hours ago we need to switch to time so that time

frame and you can see this was when Oddity was running and we collected a bunch of information here you can see for example these are all the things or the kinds of events that was the type field the very first one that we have Horst out you can see which users have been causing how many events you can see these are the raw events down here here I do have a map how do I get to the map so in some events we have an IP address and then we just do a geo IP lookup to draw that out we could actually try to see how good the map is today because GAAP always depends more more or less I guess

I was in a hotel pretty close by so more or less you get the city you get the region so this is where the events were coming from so this has been parsed out just from the logs and by the way you can combine that with other logs so for example you have the auth log and you could figure out for example if somebody has to try to ssh into your instance since this is just a public instance you can see ssh login attempts you can see over the last 12 hours this is when I set up the instance this is when somebody else try to log in because you can see those were failed login attempts

and you could see what usernames for example they tried to log into that instance and you can see where they're coming from so for a change it's not the Russians but the Chinese and the Koreans let's see it's I this South Korea at least and you can see this is where the SSH locking attempts were coming from but this is just a reverse lookup and this is just in the auth log so you can combine the audit d log and the auth log for example to see okay somebody tried to ssh into the instance and suddenly they were successful and afterwards we could collect events with Oddity so all of that can be collected and it's doable but it's still kind of

painful to do that we always try to kind of talk food that stuff ourselves or maybe you like the phrase drink your own champagne better because that's probably nicer so we have a cloud service and for that cloud service we're kind of picky to find figure out all the security events and that's why they came around and said like well we want to use all the tea but all that parsing isn't working out so well and we want to get more information out of that so what we came up with we kind of wrapped audit D into a beat and we call it audit beat so it's using the same syntax and it's using the same base library but it's

being wrapped and you don't need to write out the file and then parse it back rather than that we can't just collect the information in a more structured format directly also we've added some other events around it that made our life easier so we can correlate events we can resolve the user IDs to usernames directly we don't need to parse it out or write it out to a file and parse it back but we can send it to the elasticsearch directly it doesn't use EBP F but it's kind of closer in features than a lot of other things that we have the good advantage of oddity is that it works on all the kernels as well and it's easier to

configure and since it's written in golang it should be reasonably secure hopefully and it does support doctor and kubernetes all the way so you can enrich all your events and know where they from which container or from which connait this namespace they've been coming from so what we've done is we took lip audit and have a goal library around that and that is communicating with the oddity framework that is basically the base library that we are using to extract from the kernel the messages and then just write them out in a structured format directly that's the general idea mom and what you can do then is let's just head to a dashboard so we have

audit beet this by the way are all pre-built - but I'm I'm lazy I didn't build them myself and you don't have to either you can just use whatever is available you so you can see at night I started my instance we collected a couple of events right here and since then it has been a bit more quiet then I did something again here so you can just see the general spikes off for example connections and executed tasks and everything we have collected here you can see which rules were basically collected so you can see user logins user space etc all of this has been collected from Oddity and you can see the raw events down here and you can see

how they are doing and before I talk too much about that let's do something else let look into the actual configuration file so let's say we want to use an elastic admin and we have apps on it aw did this is where the configuration files should be living and this is just the configuration of how to put this into place so you have a few things which are kind of the configurations around it for example you can rate limit the events because otherwise you might be generating too many events and might be overloading your instance you could rate limit that I set it to zero so we don't rate limit right now I'm also not collecting the

raw event messages because it might just be too much traffic and after that we have the actual raw rules that we want to collect so for example here we are watching a file PEM Khan so every time somebody reads writes exercise or changes an attribute on that file we want to collect that and we write out an event that is called PEM access with the tag and then you can filter down on those to find those probably PEM that conf is not a super valuable file to monitor but whatever important configuration files you have on your system for example that you want to monitor if somebody changes or exit system that is something that you could

put into place so we can look into the raw events for that one here so I'll head over to the raw vents you can see in the last 12 hours we had 180,000 events because well this is a hello world application I'm just collecting lots of stuff here you would need to fine-tune that for a production setup probably let's switch to over to the last 15 minutes to see just what we had here so for example I could say here I want to have I am accessing my file and then you can see we have all of these events here and then you could just either in the search bar search for Pam conf or in a more

structured way you could also say since we have these tags and then you need to remember what was my rule I always forget what I call my rule it's Pam - XS and then you can just say this is the rule I'm interested in and ideally if you run that you can see this with just me where I've accessed that file and in that one event you can also see a couple of other pieces of information here so you can see the auditing information so you can see the primary actor who actually did that was the elastic admin user that I've been using you can see how I did it I ran cat and what I did so the primary object

that I was monitoring was the Tamkin file that I've shown you before and you can also see there are some other pieces of information around it first off for example we can enrich every element with the host so here you can see where is this even running on what kind of post so for example if you know we have some security issued just on one specific operating system or at one specific version of it you could totally filter down to that so for example here I could say I'm only interested in Ubuntu 18:04 - if you click on that plus it will add an additional filter and it will only filter down to instances from that

specific version of the operating system which doesn't change much I only have a single instance which is running that version but you get the general idea how to drill into that and the second thing we are enriching here is the metadata from the cloud provider so if this is running on a cloud provider you can't just see this is running on AWS you can see the instance ID you can see this is running in Ireland on 80 - micro instance all of that can be enriched if this was running in docker which it's not but if it would be running in docker you could get the docker metadata as well so you would know which base image you have for

Yutaka container or which labels do you have applied for your Tocker container if you are using kubernetes you could get the kubernetes namespace and similar pieces of information so that's all pretty easy to collect and show since I've shown the power abuse before let's quickly run through the power abuse to see what we getting out of that so let's assume we have our elastic admin user and he's curious so he just looks into the home directory and he finds the elastic user and he assumes well maybe the elastic user has something interesting I want to see and then he can see okay there's a secret txt file from that user maybe I want to take a look so will this work

hopefully not because of this Domitian permission denied let's try with sudo again I am and you can see the file actually contains my secret which unsurprisingly there was a secret in it and now we want to figure out who has accessed that file so we have been tagging that with I think it was power abuse so what you can do is here you can just switch the label and say there should be something called power abuse and if you filter down to that you can see there were some before but this is probably the last one that we are interested in and you can see again this was the user doing something so you would know where to look you can see how

the use cat this was the thing that we were monitoring the secret txt file and then you can see the various pieces of information around it that might help you search for example you can see the process ID and you can see the actual command that I have been running and we have added various tags to that so you could filter down so you can see who has been doing what and what they have been abusing or not abusing in your system so that's pretty easy another thing that you might be interested in let's say is somebody opens this socket and wants to yeah run something you don't want to have their so for example if I run

netcat on 4025 anybody has their her laptop handy or something on their phone to talk to me so the instance name is you can just tell MIT to it's just like my twitter handle WTF which is like the right domain any demo if you just tellin it here you should be able to connect and then you can write whatever you want and it should pop up on the other side so if anybody has a laptop or anything handy you can just tell that to that instance on port 1025 and the messages should then pop up and we will be monitoring both which sockets have been opened and also which connections have been coming in into that instance here while you run

that let me quickly jump back what to disrupt the process this is not where I wanted to go let me quickly jump back to the configuration file here so for example to log which sockets have been opened I'm just saying the system call is - s socket and then I just tag that with socket open for example if you connect to assist them out you could just you just need to know the right system call and then you can monitor any system call on your system so the open socket is the one that we're interested in let's get rid of this one because we don't need it let's say we want just the open socket

and ideally this my browser is hanging ok this refreshes and then you could for example see here these are all the sockets that we have been opening and for example in that one here you see where what when has been opening sockets for example here you can see ok this Beach that is also monitoring my instance has been opening a socket when you could then just drill down into those um but because all of that configuration is kind of a bit arcane we have added another kind of module into that same thing this is not based on oddity this is kind of like a nicer syntax and just something we check in the background with audit beads

automatically but we can for example check which you have which users packages logins and for example every two seconds I check which processes do I have running and which sockets do I have running and for those we have proper dashboards pre-built where you can just see what is going on on your instance so if I had over to the dashboards again and rather than looking at the raw event I'm looking at the audit beat dashboards again and you could just say like let's look at the sockets you can see these were all the sockets that were opened and actually does not the dashboard I wanted or did be if I could type audit beat I want the socket dashboard is the

one I want here you can see these are all the sockets that have been opened and the connections because of the screen scaling this is slightly cut off now but you can see which IP addresses have been talking to you which destination ports I have been working here and you can see our 1025 is already sticking out here and I could just click on that plus here to filter down on that one and you can see we have in the last 15 minutes for sockets we're open and you can see this is when they were open and you can see first here we have the listening command when our netcat opened and you can see this was the actual

command that has been running and then you can see inbound connections where people connected to that and you can see from which other IP addresses to our system people open the socket connection and you can just monitor okay something bad started running on 1025 and you can see when was it started you can also unfold that to actually see okay we have a socket opened and you can see all the details of like who has been doing that you can see the elastic admin user has been opening the socket and afterwards you can also see like which inbound connections have been coming into that one socket so if a remote shell is starting to pop up on your

instance on one specific port you can then figure out which specific user opened it and which other systems connected to that pretty jillee so this has been going on did anybody write any messages no maybe the Wi-Fi is also blocking port 1025 I've seen it a couple of times that people try to connect and we're not able to but generally you should be able just to tell that to the instance and then it will show any other messages if I type other here it will just pop up and you can have those events okay since we were running a bit short on time [Music] the final thing maybe we could touch on is every now and then you have like some

sensitive files you want to monitor if anything changed on those so what the file we have for example I have a very professional website here which is just welcome but maybe somebody wants to change the website so somebody could just walk in and say like the W in HTML we want to change that site it's read-only sorry

let's change something on here let's say welcome to [Music]

and suddenly your professional website is not so professional anymore so somebody just changed something and you might want to figure out what what changed on your system and to do that file bit sorry audit bit has another thing another module we call file integrity basically group basically you can define watch this folder exclude any files like these are the temporary files VI is creating because I don't want to create any noise from VI changes and watch for any changes basically what we're doing is we're hashing every file in there with sha-1 which is the default if it's smaller than ten megabytes and we're scanning up to fifty megabytes a second of files change and for that we

can also see in a dashboard which should be a file integrity overview this one here this one will show you am we deleted and moved one file this one here you can see it was done by the root user because we were using sudo and you could see this was on this host and you can see these were the two files touched why do I haven't moved and deleted of that numeric file here because that's the pattern that VI uses to change the file it's actually creating a new file and then replacing the original one so you can see here this was the file that was deleted and this was the file that was changed and you can see the timestamp

this is a way to track like what important files on your system are changing over time so depending on your operating system you will have different methods to keep track of file changes audit the oddity features obviously only work on Linux because it depends on the kernel to do that there is a port to Mac but that's very it's slightly different in Bakke so we haven't implemented that yet for the file integrity that's supported on all operating systems so you can monitor all the different operating systems and these are the calls we are using for that and you could switch out the hashing algorithm as well for example sha-1 is the one we use by default but this one here the

last one would be the most performant one if your they're worried about performance in that okay we've seen that one to wrap-up I always compare this a bit to a block of Lego like you have a lot of rules you need to put together yourself but once you have the right rules you can pretty much monitor whatever you want on your system so it can be executions of processes it can be sockets it can be user actions it can be logins it could even by now be stuff like if I go back here we could even monitor for example packages here in the last 12 hours it checks you have 620 packages installed on your

one instance you can see when we're packages add it to your system and you have a full list including their version for example you could figure out like do I have audit bit installed and then you see we have one package installed called audit beat this is when it was installed and this is the version that we installed this can also be very helpful if you know there is a vulnerable version of one specific package out there you can just collect that from all your instances and see like where is that package still installed or which packet or which instances have already been updated with that so stuff like that will come in very handy this is

just part of what audit beat can collect for you but you need to kind of configure the right rules whatever makes sense for your environment so we've seen oddity for the Royal ents and the base lab the basic kernel module that we were using and then we are wrapping it in to audit beat to make collection of events a bit easier and then we also have thrown in logs and dashboards to kind of like tie it all together and see a bit more what is happening on your system we're actually using our cloud infrastructure and we have a couple of thousand VMs and bare-metal instances and all of those are being monitored with audit beat and we have been doing

that for quite a while sometimes people ask for the specific rules that we have or are using I'm not sure the colleagues want to put them out because then you could try to find something that is maybe not caught by those rules but maybe they will yeah that they plan to write blog post on the actual rules that we are using in production just to give you an idea what might make sense for us or for you if you want to try anything else out of that I have all the configuration files in the setup it's fully automated that's why I just could it set up tonight it's just a bit of ansible and terraform it

will just start a cloud instance create all the configurations and then you can just go wild and try to do stuff if you are looking for alternative solutions slack has something very similar which is called go audit to collect auditing events and they are also using the elastic SEC to collect all the security events into that there is another project on github from scrubber e which is called all shape so like all report and all search they wrote the binary called all shape that can export the audit d-logs to Jason and XML if you really want to use XML that's the only tool that can do that from what I know so those are your options to get to

those audit the event logs and with that I think we have one minute left for questions any questions in the room or did anybody post anything on slider are there any questions for Philip no questions here no Chris is there perfect then we are perfectly in time thanks a lot then thank you very much Philip