Quick And Easy Forensic Timelines Via Sysmon, WEF, And ELK

invited me here so my name is Aaron Jewett I'm gonna be talking about forensic timelines via system on window F stands for windows event forwarding see that acronym and ELQ elastic stack so Who am I I'm a threat hunter this name is my life right now just about every alert you see on the network about 90% of them are assist admins doing weird things so this is what I do so right now I work for Booz Allen working with the US Army up in Wiesbaden as a threat hunter I spent about ten years as a rock operator at the NSA so I've got a lot of experience on both sides the defense and offense as far as info set

goes living in Frankfort so yeah the new job with elastic I start next month so I'm going to be transitioning from a Windows centric network to an entirely cloud-based network that I'm gonna have to defend so it's gonna be a bit of a learning curve for me I think no adventures and once again you know all opinions are my own not anybody else's in case I say something that makes somebody upset um so what this talk is about so it's gonna be about basically using Kabana or Splunk I'm going to focus on Cabana because it was free and easy to use it my house we use Splunk at the office they're both great so we're gonna be

talking about building dashboards that make the response easier so you know so there's multiple portions there's the alerting there's detecting the adversary there's hunting there are tons and tons of blogs out there and talks out there about using this type of setup for that so I wanted to focus on what do you do when you see an alert how do you respond to it and how do you make it quicker easier and more efficient to do the response so what this talk is not about you know the anomaly detection the threat hunting I definitely you know I could do a whole other talk on that but like I said lots of people have done these talks so I

didn't want to focus on that portion of it I'm not gonna go step by step walkthrough on setting this up I've got a lot of instructions but you know there's a lot of stuff out there on the internet showing you how and and yeah absolutely zero blockchain or AI in this talk you know blockchains a scam and AI is way too complicated for me so we're gonna go on this scenario you know your CI so just got a call Brian Krebs said he's got indications that you're compromised he's got a host name a user name and it's Saturday you know how quickly can you get forensic information from that host about that user you know one of the

users and hosts in New York it's an international company in New York Singapore imagine it's you know 4th of July weekend and some holiday in Singapore you know it's a Sunday in Germany so nothing's open and you've got to get all these answers you know so how long does it take to collect this data you know how long how many people in your company are capable of doing this you know these are the scenarios that is traditional InfoSec you know you get a lot of people they get a phone call and they got to get on an airplane to Singapore to go image a hard drive and it's just not a good way to do things so

problems the attacker is moving quickly through your network you have to be able to keep up with them you have to contain them and find them getting data for investigations is hard if anybody who's taken the SANS forensics course is the first two or three days of most the forensics courses is all about gathering the data about imaging hard drives about you know getting the forensic information off of the device that takes time and expertise if you don't have the expertise in-house you got to pay expensive consultants to go do it for you so this is this talk is about you know getting the getting the systems built so that you don't have to call

these people you don't have to spend this money on that more problems you know traditional seems only contain alerts they said the the antivirus says something bad happened and that's all it tells you something bad happened and you've got a time and a host and that's all the information that the antivirus gives you because they suck attackers delete the logs you know they compromised user accounts and they spread so you're trying to quarantine a system but it's actually the user account that's compromised invading antivirus like I said it's super easy I lost a lot of respect for any virus during my 10 years on the offensive side and I haven't getting any respect back for the three years on the defensive

side at stevia so what's the solution so we're gonna create components or Splunk not you know should be yes or you know either one whatever you want to pay for and so you make these these dashboards and it makes this easy to do so then you get a dashboard and then you have you don't have to spend three days of sans training teaching somebody how to image a hard drive they just go to the dashboard and type in an IP address and boom there's all your information they type in a username you can do the investigations extremely quickly like this elastic and spunk they are great for forensic science but everybody does the talks about data science about AI

about anomaly detection you know they put up these you know crazy formulas for the data scientists that I just they blow my mind I don't understand any of that what I do know is forensic science and all about you know most forensics is creating a timeline of events you know what happened when who did it while you're doing an investigation I don't need you know complicated math for that I just need a time so how are we going to do this so this is what I'm gonna talk about in this talk so it's three-step process super easy well it's a lot harder than that but it's you know this is not I'm gonna go out over a lot

of stuff but it's it's pretty much these three steps so you got to enable the verbose logging so we're gonna do that with sis Mon on the windows boxes elastics auto beep module which just came out on the Linux and Mac OS systems which is basically like system on the for Linux and then you get all those logs you centrally collect them and then you feed them into a elastic stack or Splunk whatever you're using and then you build your dashboards and you're good to go and then after that you got to keep growing and adapting and changing your stuff so it's not like that you do that and you're done now you know hang up the mission accomplished banner

alright so step one getting the right logs so sis Mon so if anybody hasn't heard about sis Mon if you're running a Windows domain you absolutely should have system on deployed it's a free utility so big emphasis there on free and the config file lets you it allows granular control over exactly which logs are created most people they just say oh it's too much it's too much data it's because if you use the default configuration file for sis mod is you know probably a gigabyte a day per host if you tuned it down to only send you the stuff that you care about you can get it down to just a few megabytes per host which is a much more manageable

manageable amount in an enterprise this is a quick overview of some of the stuff if you haven't heard of system on that it logs you know every time a process starts and stops any time somebody changes the file stamp on a file you know attackers love doing this when they put down their root kits and implants every network connection and network connection tells you not just the the source and destination IP like a firewall log will but it tells you the exact process gets spawned that network connection registry changes file downloads named pipes drivers etc you can see this it's it's an amazing amount of data once you tune it to work right alright so you can you manage it via

config file so if you haven't you know looked at it Swift on security put out a config file the community's been adding to it it's a great place to start and I say start is and you will have to make changes to it for your domain no matter you know you cannot just take this and plug it in every domain is unique you're gonna have scripts you know you're you know as I mentioned from that first meme some sysadmin is doing some weird out there and you're going to have to whitelist it because it's just going to flood your logs umm constant testing and changing it never ends somebody's gonna go buy some new software and it's gonna

wreck your config and you're gonna have to you know tune that up so here's some config tips so that top line explote exclude the good and include everything else a lot of people will try to just they'll treat it like it's an IDs like it's Splunk err suffer like it's snort where they say I know this is bad so show me this that's the wrong way to go because then the attackers will say oh this is all you're looking for I'm gonna do something else if you exclude it if you include everything and only whitelist that no noise you know chrome dot exe making a connection on port 80 to the Internet okay I'm going to exclude that there may

be times the attacker uses that and it sucks but it's gonna fall it's gonna flood my logs I want to know every time chrome talks to a port that's not 80 or 443 you know or you know maybe then I whitelist my local you know internal servers that use weird ports whitelist with caution with sis Mon if you whitelist an event you will it will never show up in the logs and it's like it never happened you know inevitably you will be doing an investigation in something and you'll run into a dead end and you'll know that that the logs keep happening you just don't have a copy because the attacker moved somewhere like I said maybe they

migrated into chrome dot exe and started connecting on 443 over the Internet you're not going to have that connection and your system on locks sorry just part of the price of doing business management tips so sis Mon config file every host when you you can point that you can host your config on your net logon server on your domain and then just use Active Directory to tell everybody to go back to this config make sure it's read-only of course don't you don't want your attackers messing with your config well best practices we found over the last year or so doing it three separate config files your workstations your domain administrator your domains controllers and then your servers on a

domain controller you're going to want to log it every day and you can generally whitelist everything that happens on that because it's it's a domain controller you shouldn't have chrome connecting out on the internet so you on a domain controller you may want to log that so have three separate configs you know customize it for your domain group policy powershell make they're great for deploying this stuff once again i'm not gonna go too in depth over this there are blogs out there on how to do all this stuff oh and yeah don't make changes on a friday just don't elastic ought to be module this just came out January 29th so when I actually did the call for paper this didn't exist

so yeah every host gets a unique gooood when it's deployed and so you can change the host name the IP address everything about the host but that unique gooood stays the same so in your you know in your stack you can track all the changes to a system over time other than that yeah process information socket creation user logins etc and so anybody who's ever smashed their head against oddity logs knows that they're just just a nightmare and Linux logging in general is just a nightmare so this is you know a great thing to have yeah for some reason ready for vs. Red Hat 5 they changed the schemas and everything looks different it's just

yeah no standardization additional data so in addition to the windows event logs you may want to add some extra stuff so Active Directory information is great for enriching your dashboards so like get ad computers get ad users on a daily basis and pipe that into your dashboards then you can add all the you know you can you know when you look up a hostname you'll find out exactly what desk they set out and it's displayed on the dashboard through the phone um what the phone number is for a user so you can pick them up and call them say hey was this you autoruns Palantir made a cool project where they run auto run it this

is a PowerShell script that will run auto runs and write as a Windows Event log so you can just scoop that up and all the rest of your collection and then any other vendor Windows Event log however Sony so step to collecting the logs if your logs look like that you're getting too much data um so different ways of collecting logs you've got agent based and then just using the built in functionalities most time you're gonna have to use a hybrid approach in a Windows domain the Windows event forwarding is a great way to use the built-in capabilities once again it's free and if a system gets the GPOs then it gets the event forwarding agent

agent based technology a lot of times your agent will die and you have no idea I mean how do you know you didn't get a log that you know it's just sitting out there so you you know relying on the agents gets a tricky as to seeing a little bit of diagram later at some point you do have to use an agent to get it into your your digestion system windows event forwarding so like I said it's free any well mostly free any of you just need a Windows server on base install Windows server VM works can be a Windows Event collector so GPO to tell the workstations what to do everybody has to have win RM on in your domain if you

don't have it on just turn it on Jessica pain at Microsoft has a bunch of great blog posts on how to set this up how to use this for threat hunting once again that that this could be its own to our talk so I'm not gonna talk about it the subscription so every system gets a subscription these are all the windows events that have to go to the the collection system so everything is system on your logins these are just some other recommendations of things that you want to send for collection USB events is great if you're looking at like insider threat or things like that you know every time that's the kernel PNP have it the time and devices plugged

into a system you'll get a log event for it what do you do once you've got it in your Windows Event collector then you can send it so in this example we send it to Kafka so I used hunting elk help for this talk to set this up at home because it was easy to do and we use Kafka at work as well it's a great place to send your logs and everything can subscribe to it so if you have Splunk and elastic and a traditional seam and whatever else you can send everything to your Kafka instance and from there send it out to whoever needs it it's a a subscription model alright so step three

we're gonna build things with these logs this is the big thing that I'm you know the big part of this talk alright so the alerts I said I wasn't going to talk about this but I got a you know put a quick shout out Sigma project by Florian Roth it uses a lot of these system on data to make alerts so if you see run DLL 32 executing with no command-line options that's bad that's an attacker most of the time a lot of these rules like that I think they're about 130 system on specific rules in a sigma project and Hulk has a script that will just go to github get the latest rules and

create a last alerts with them so that in your elastic stack you get alerts whenever these rules trigger and you can add them in your dashboards or you can send them to a slack channel whatever you want to do with them alright so the dashboards so the dashboards the the big advantage dashboards is you can have one person who learns how to use elastic use Qabbani or Splunk and really build out these nice easy-to-use dashboards so that the rest of your forensics team doesn't have to become a data science specialist most of these places will you know most the time i've seen businesses they they get their logs they throw it into Kabana and they just hand it to their security team

and say have fun and it's just a mess and nobody you know these they're not data scientists they want to know what happened and when so the dashboards can make this easy so host investigation dashboard so i'm going to talk about three big dashboards and i've got at the end of the talk i've got a link to my github project and all the JSON files to import these in your Caban instance so that the host investigation investigation dashboard who when where how and why for a single host so you enter a host name IP address select your time frame and boom here's your forensics timeline if you've ever done the sans classes they talk about like logged a timeline they've got

lots of these scripts did like take the logs you move them to a server you run python scripts you export that to an excel spreadsheet and then you do some other things to it and then you get this timeline and here you type in an IP address or a host name and you hit enter and it's done depending on how much horsepower your elastic or spunk instance has each panel should answer a question you split it out so it's not too confusing and then you know once again whitelist annoys your SCCM regular recurring tasks um so this is just a kind of how i've separated it out when it's pretty easy to do user events um

show me all the active users who was logged into the system at what time line successful and failed login attempts you want to grab the logs from the domain controller as well to enrich this elevated privileged account logins 4672 these are great to see on if you see a normal user elevating their privileges this you've got it you know probably gonna have a bad day commands executed by system if you ever see system user execute who am i dot exe you just got owned because red team guys and i found guilty of this as well the very first thing you do when you pop a shell on a box you type who am i to see if you got system and that's an

indicator right there so you know everybody's guilty of it so if you see who am i dot exe running a system you know it's for sounding alarm downloaded files this is a great one if you know when you're trying to find out why of an anti-virus alert popped and the user says i have no idea why that pop and then you can go to the downloaded files and if nid fifteen you say hey i see here the chrome dot exe created this file and the downloads at this exact time while you are logged in while you are surfing to this dubious website oh you know now the user has a little more trouble arguing the case or do you see

outlook dot exe create the file you know great things to know when you're doing an investigation process execution every process has a unique process to it I'll go into that later but you get the full information about the process about you know the the full path what the current directory was who then who's the user that ran it everything you can imagine with that so you make these dashboards so if I see the parent process how it also tells you who the parent was it's a parent's command powershell w script c script you know put those all in a little panel so i can say hey the user just opened up powershell and ran all

these commands and they work in the finance section so they shouldn't be doing that strange network connections this is you know things that not chrome cut talking to 80 and 443 things of that nature just you can include everything but it's going to be noisy if you want to get into the big investigations you're gonna if you've got more data coming in just the windows events you can add in your proxy data your firewall your bro data once again make this really easy to do for your users you can have a junior person you know very quickly identify you know events inside your firewall logs correlated with your windows logs modifications to the host registry changes new files created so if

you see somebody dropping files in system 32 that shouldn't be new services getting installed drivers loaded USB changes as once again insertions you can also find this if you look for registry keys with star Enuma star every time you plug in registry or a USB item into a Windows box it makes the registry key and gives you it gives you the name of it so you can see you know Jim's iPad plugged in you know at this time on it you know it's in the registry and then the W my subscription changes that's a popular persistence tech technique for attackers all right so on to the user investigation dashboard once again users get compromised just like systems do

sometimes the users bad sometimes their account gets compromised there's no way of knowing really the difference just by the behavior of what's happening on the computer most the time that takes like a law enforcement type investigation or somebody's going to fess up to it yeah get a B user is your friend if you can enrich this data then you can have the users phone number and so then your junior analyst on a Saturday morning see somebody logging in and they type it into the panel and it's got their phone number there they just pick up the phone and say hey was this you yeah I'm working on a Saturday morning okay great and they say no I'm with my kids at the

field I'm not I'm not at work at all didn't you got an issue this is just a quick screenshot of the dashboard I made a home on a over the weekend on qivana like I said we use Splunk at work I use Caban at home either way it's not too hard to do you can see the dubious PowerShell command line spawning off there the user was a running PowerShell Empire because was me so this is just so you saw the time line there that that was the time line of every computer that's logged into so when you've got a computer user account that gets compromised you know this is great because you can just put in and

say hey you for the last seven days show me everything this users done every system they logged in to all these events successful and failed logins definitely it gets a good it gets a bit creepy though because this dashboard is capable of showing you everything users done on your domain you know anywhere so this kind of because of the power of this and these dashboards in general if you do this at work make sure you multi-factor authentication lockdown these systems because there is going to be some sensitive information in here it's not going to show you the contents of files but it show you the name of the files you will have admins typing the password

into the command line and you'll have those passwords in your logs so you know bug people about that it's bad habits but you're gonna see that in your logs this is a good selling point to the compliance team so because you know I live up in Frankfort you know regularly the bullet I'll walk into the Deutsche Bank offices and ask for you know escort a user out and ask for everything they've done you know so a dashboard like this would come in handy for them all right so let's say you're gonna learn it's talking about a very specific process that's happening and you say what is this process doing every process in sis Mon has a unique process gooood

this process good is unique every so if you spawn for calyx dottie exceeds every four of them are gonna have a unique good so you just go to the dashboard you see search for this do it it will show you who spawned it who is parent was every single event related to this process will be in this dashboard and in the dashboard you can add multiple goods and get a whole hierarchy chain of events of everything that happened so if if you see a user doing something shady with PowerShell you can see every child process and the parent process you know chain together this is just a quick instance that the shady user spawning our shell empire again so you can see uh

I think yeah I think I ran who am i in there actually that ran automatically that was just part of power show and fired I didn't even have to run Who am I it just does it on its own so uh I can find out you know I can just trace it back so this is great for junior analysts to really get an idea of every time you see an alert you say okay is this something bad and I've seen in countless organizations where you know analysts will just say oh I've seen this alert hundreds of times it's nothing well how do you know it's nothing well one time it could be something well it's

it's just so hard to investigate individual alerts they just write off oh it's nothing with this you can say okay show me everything that that account actually did everything that that process is done and now you can say definitively that it's nothing or it is something so I think I probably went a little too fast but so any organization can build this very little money so that the software is free you just need basically the hardware to run this on and everything can be done on virtual machines the dashboards are not just for the security team this whole system is not just for the security team so if you are trying to get this built in your organization you

know go get the active directory guys on board go get the DevOps guys on board on all of this data is great for them as well and if you get them on board then that helps the funding and your your CIO so we'll be happy as well to share the burden of this system when you can use this to audit changes to Active Directory or do a quick query of all crashing processes or find out you know things of that nature if the DevOps guy is trying to troubleshoot why their system keeps crashing they can use these dashboards so are there any questions start that one a little faster than I expected Oh question yeah so we we spoke with

some some of the German German Army Bundestag people came and they were taking about this and they liked the fact that when the logs the logs come in they don't have the contents of any of the files so you have the file name so if somebody puts sensitive information in the name of the file that may be an issue but it's easy to justify look we were collecting the names of the files on our domain so they were pretty confident they could sell it to their lawyers I haven't had that argument but you know once again it's you just have to have the right lawyer I guess oh sorry yeah the question was how does that work with

German labor laws with the sensitivity of the information etc and yes any system like this absolutely must be locked down like don't put this out on your AWS cloud with a crappy password or no password you know two-factor authentication keep it protected any of the in India this will fly with a big company with a decent life behind them because you can basically see if someone was working what they were working on yes so the question was about a big company with it with a union and yeah monitoring individuals and yeah the data is already there in the net work we're just centrally collecting it so yeah once again I'm not a lawyer in the German

labor laws I'm definitely not familiar with them to that extent but you are just you're collecting the data that's already there I mean any of it any of this information you could already you could find out with the PowerShell script as well but this it just makes it a lot easier and makes it a lot more simple to to view and see I see them myself the team's yeah my team then push it back because yeah okay just adds up yeah I'm sure there's always anything like this there may be a maybe pushback because the users would have zero visibility in this I don't know how much pushback the pushback would have to come you know from someone who has access or

somebody who knows what's in the system and knows more about system like Microsoft teams makes it pretty obvious I think and normal like normal managers in the company should not have access to this system this should be like the IT guys and security guys and maybe some compliance teams regular old manager no they should not access the system in any way yes coverage necessary from this so 100% of my service and on Senate McFly's power yeah so the question is what percentage of coverage to profit from this I would aim for a hundred percent I mean there's it's because it is free and it is just a GPO that you push out to the systems yes it does have to be rolled

out and yeah we've been doing it for a year and we haven't gotten to a hundred percent yet there's always there always going to be you know pushback from admins workstations I definitely think a hundred percent isn't it is doable we've had red team's come in and before we had this they were able to do PowerShell Empire and living off the land techniques and we didn't see it after we built this we kicked their butt we would see him immediately we get the alerts within 10 minutes of them coming in and we'd kick them out within 30 and so yeah they look on a red team's face when you should have a full report of every

single command they ran while they were on your system is great you know it's it makes up for all those years of them coming in to kick your ass what you can say this is everything you did and this is when we kicked you out of the network and that's a great thing for a pentesting record to have that so yeah that's as far as workstations go you need to have coverage because they're gonna go to your kiosk downstairs or you know that the waiting room area or somewhere where it's not going to be a critical access system and then try to move laterally from their servers you know I would push for a hundred percent

if you can but it's gonna be an ongoing fight that never ends nope oh yeah and yeah the question side yeah that that's the github link for the so I've got this these slides as well as the the JSON file so if you want to use these dashboards you just go there um right in your Cabana so I was using elk the hunting elk for this so your if you're it's formatting the events based on the fact that Kabana gets them first and then they move into Earth that Kafka gets them first and then they move into cabana everybody's going to have different field names but if you just do a fine if you have different field names

you can find and replace inside of the JSON files and it should work one other question we have with German and because my limits they've been something really problems because a lot of things are then suddenly German yeah and I think that's where the when I mentioned that every domain is unique you're gonna have you're gonna have some processes that are the same name and some that are completely different and so you're gonna have to just so I think I kind of skimmed over it but I said start small with your config when you're when you're building this out you know pick random hosts five or ten hosts in the network and just start collecting from those

systems do not push to everybody in the network l at once that's a horrible idea and it will break things pick five or six people and a spend a week or two just tuning the logs and getting rid of the white noise and then you know maybe go from five to ten to a hundred to a thousand from there as you build out your config take it you know you know this is it's a project you're going to need a project manager for this to get it deployed somebody to kind of champion this and push the to push it out into your domain ya get the same quality for unique systems I think so like I said audit became out and the

question was same quality of information for audit be so ought to be came out January 29th as I mentioned my next job is going to be working for elastic which is primarily a Mac OS Linux based network so I'll be learning this and trying to build this on you know these systems so prior to the oughta be you know I've tried building out these dashboards using like audit D data just using the built in Windows that net logs are the windows syslog data and it's just a nightmare you get data from your SX server and from your Red Hat server and you're a bun to server and your debian server and the logs are all

completely different format and there's no normalization and it's just is horrible so yeah so I'm hoping that I like said I haven't done it yet but I'm hoping the door work are there any other questions all right so I guess that concludes my talk and I'll be here the rest of the day and if anybody has any other questions thank you