Log All The Things! Proactive Forensics using Log Analysis

hey good afternoon everyone everyone hear me okay cool uh thanks for coming to our talk uh my name is kyle salaz and i'm aaron buring and i just have to say if i seem a little bit nervous today it's because this is the first time i've ever presented in front of my seven-year-old daughter so she has promised me that she would make at least two funny faces at me though so that should break the ice so it'll be all right all right uh so real quick we're going to talk about a lot of stuff pertaining to collecting logs from endpoints in the network and if you hear uh some things that you don't recognize or you don't

understand why we did some certain things or turned on some certain things we've done a couple of talks for uh shmukon and then in mir khan which is i think firecon or something this year and then we've uh we've gotten an article in ieee so take a look at those and i'll give you some background into why we did some things with immediate app locker and host-based firewalls so the point of of our talk today is to kind of go over the our our last two years with deploying sims or simin in our environment and how we went about collecting logs uh some very obvious goals here implement a sim uh collect logs from what matters

not from everything and do it within budget anyone here that's ever tried to spec out a sim uh they're really really expensive and actually make use of it we don't want to treat this like an ips sensor where you just buy it and then no one knows what to do with that so if you do cyber security operations uh you will likely have to deal with phishing emails it's still the most prominent attack vector uh for compromising end hosts this is one that came across our networks i'd say in september and it's a you know it's a receipt for some payment that the user supposedly uh you know some transaction that the user had done in

the past and it's a it's a word document uh but the user doesn't know that the word document has embedded macros it calls on a script that drops a an exe that then makes a connection out and starts encrypting their entire profile so this is pretty standard stuff our goal eventually after doing the basics was to figure out how we can log this stuff and then proactively alert on it and then search the entire enterprise for artifacts from something like this because this is the most common attack vector or at least the scariest in terms of how apt starts so the the point of this talk is if you listen to your logs uh they'll have some

great things to tell you uh i'm gonna go through some basics if it bores you uh bear with me we'll get to some of the more interesting stuff that we did so we had a demo period we evaluated a couple of sims i know it's hard to do a demo sometimes you just want to pick something look at the magic quad and be like all right we're going with vendor a because they're all the way up here uh obviously you want to create your own requirements and then make sure that you don't exceed your budget and then you want to document everything uh and then you want to send a requirements document out to some

vendors the usu the reason i say this is because some vendors will if you send them a requirements document they'll usually the marketing guys will answer everything like yeah we do everything uh guess what when you buy it and you have that document with their responses you can come back and say you don't do this or you don't do this well but you said you did so you could you have some leverage with the product development team come back and have them uh build some more uh some more features that you actually want uh that stuff is really expensive if you want to hire consultants to do that especially for the more higher end uh sims

so let's go through the steps uh step one you want to configure logging obviously figure out what log sources you want to log from and only pick the events that you want to parse out don't just send everything we're going to go through the order of what we did in terms of what we picked out first and and it was more of a methodical approach i know it's kind of counterintuitive to say i'm just going to send my firewall logs to this you know half a million dollar product that i just bought but actually try to do that and then figure out how that how that works with with the sim sometimes you run into parsing issues run into

timing issues and we'll get through some of those uh the other important thing that we ran into this is that you will have performance issues with some of the some of the log sources the devices that you turn logging on for instance switch acls if you ever turn those on because you have all these acls that you want a lot you want to see what what kind of traffic's reversing your network that'll if the switch is a little bit smaller it'll bring down your switch so that's some of the the things we kind of ran into the second step is actually figuring out what you're going to send to your sim uh we this is in the list of what we did

we started with perimeter firewalls ips sensors that's an obvious one sim vendors really do a good job picking up on the common events that those those lock sources generate domain controllers for obvious reasons you want failed logins you want domain admin logins you want any really any login and there are a lot of rules built in for that uh web and database servers we haven't done much with database servers there's it's kind of a tricky uh it's a tricky device to log or or entity to log but we we have done a lot with web servers and there's a lot of good logging there that we'll get into uh load balancers and proxy devices uh

proxy devices are are amazing if you have an inline proxy and if you're doing ssl decryption there are a lot of great logs that you can get out of that and then all the infrastructure stuff rsa servers vpn devices uh the one the bullet point before the last talks about netflows jflows partial packet captures full packet captures if you have a gazillion dollars uh you can feed a lot of that stuff into your sim and it it kind of augments your log sources so it acts as like a like a crude dlp or even an ids really and then the last bullet is client logs and and that's what aaron's going to cover in the second half of this talk

and that's kind of where we did a lot of the custom stuff and and that's where we want to lean in on the vendors the sim vendors and the product vendors to kind of do better there because it's severely lacking in that area the final step is actually looking at the payload as it gets parsed out in your sim so there are multiple event timestamps that could be labeled or thrown at a payload and uh half the time those things aren't uh get co-mangled like the event time versus the storage time versus the start time uh most most host-based firewall logs don't sim vendors don't do a good job parsing out those logs also look at if the entire payload is

arriving so we've had this with our proxy locks where the law the payload is literally too big and it just gets truncated so if you have like a url at the top that's really long urls can get really long and then the source ip and the username or beyond that then you don't get to see that so that's just something to pay attention to and then figure out if things are actually getting mapped to global categories uh you know they're they're native uh mappings in every product and and those things don't always translate well so depending on what you're sending there you want to make sure that like a username is getting mapped to the global

username that makes a difference because if it's not then if you have a domain controller and a firewall they both have a username field and that category isn't happening then the rules won't work because the username doesn't translate for the firewall logs quick note on ntp uh just point everything to one log source if you can uh it's for obvious reasons you don't want to have uh an issue where you have dhcp logs and let's say firewall logs that are two minutes ahead somehow your client made http connections before you got a dhcp lease that is going to mess with your um your correlation engine so and you're not able to actually do anything forensically

if you if you have uh log sources that are desperate in time so that's just something to pay attention to and then if you have chatty applications that you know send a million packets of requests within a second it gets really hard if if the timing's off

absolutely yeah and that completely depends on the application oh he said the depth of the time whether it's milliseconds or actual seconds that makes a difference uh how how granular your your your setting your ntp settings but we'll get a little bit more into timing when it comes to some of the other logs uh if you do all this you and your logs will start painting a really pretty picture hopefully so let's get into some of the network logs that we've done uh proxy logs we mentioned this is a huge uh advantage if you can get this parsed out so i wouldn't worry about system logs initially unless you're unless you're using your sim for both

operations and security so that's kind of one of your requirements or your operations guys going to be using your sim or just your security guys access logs are really where you want to spend your time here we talked about your url length being an issue most vendors will fail here so that's something you might have to take up with them or increase the the payload side on the on the proxy itself you could you could switch uh syslog from udp to tcp if you wanted and increase the packet size so it doesn't have to go over udp a cool thing that we discovered is you can get every list of every file that was downloaded or uploaded so your your

proxy sits in line uh that's not going to be parsed out i doubt it we haven't seen a sim that parses out file uploads because it's in the payload you can reject that stuff out uh i don't know if you have use for that on a day-to-day basis but it would be really cool to say that file was downloaded two months ago on this machine and it'll you know if you're keeping your logs long enough you have that data user asian strings will infer application execution so you know if i use netcat or if i use ie there's going to be some sort of user agent string associated there and then domains obviously ip address

and the destination domains being logged by your proxy because it is proxying that entire connection uh and then i think most proxies nowadays allow you to integrate with ldap or active directory so the username for every outbound connection that's going to be included in your log flows and partial packet captures there's really one one sim vendor in town that does partial peak pcaps so traditionally you do netflows which is port and source and destination ip and then the packet size or you do full packet captures which is very expensive you need storage uh processing all that stuff so this ones inventor can combine flows with packet captures and then that'll give you some interesting data

on your network because you get to span ports in really strategic locations and then you have kind of like a crude dlp sensor where you can start detecting like someone bringing up a new dns server or new sql database or a new vlan or something and then it helps you identify patterns of usage baselines don't have to be complicated initially i mean i've attended a lot of talks that talk about how you baseline something and then how you build these really cool uh graphs and how you detect anomalous behavior this is a graph from our network for three days of just network traffic as you can see from around i don't know if you can see the

time but from around 11 p.m till 5am not much is going on it's an easy rule if you see a spike in that time frame you know throw an alert similarly uh during the peak times it's pretty it's pretty obvious that if someone you know tries to do something at the peak of time you could say 20 over that that peak let me know if that if that triggers uh that's just that's some of the things we've done uh from like a very basic standpoint uh the last bullet talks about strategic choke points so if you spam a port in front of your proxy every connection going to the internet will be the source ip will be

the proxy same thing with dns if you spam a port in front of your dns servers then all the recursive queries go in your dns folder are going to be sourced by the dns server if you span a port behind your dns servers or behind your proxy then you're getting the client ip that's making that connection so that's just something we learned that pay attention where your spanning ports if you're doing flows or partial packet captures uh we talked a little bit about the behavioral rules uh this is more of we use this for insider threat kind of things or rogue admins you know if someone brings a new dns server you should have a process for

you know change control if something comes up you're aware of it most sims allow you to build a network topology you basically feed it all the vlans tell it what your dns servers your sql servers are your web servers and then if it sees anything outside of that it says hey there's a new dns server on the network there's a new web server obviously that takes time to tune early on but once you do that it's really not a lot of tuning to do depends on how big your network is new network being created similarly how many times you're creating new vlans we've had admins try to create new vlans to circumvent a lot of our interesting protections

that we have so that does happen we talked about traffic patterns during off hours those are easy rules and then once you have a large data set you can start creating more complex rules web servers uh again back to user agent strings if you have multiple user agent strings being sourced from the same ip address that's unusual you should probably write a rule alert on that it's usually an indication of someone's just trying different tools files that should never be accessed honey pots you could put a like a dummy file and if someone downloads it we do that then you alert on it it has no need to be on that server but we put it up

there to see if someone tries to download it um create alerts for access to protected directories uh same thing if they don't if they shouldn't be accessed you should alert on that uh load balancers again we have large web environments that are in the dims either usually fronted by load balancers make sure that the load balancer is uh capturing the source id of the client that's making the request otherwise you're seeing the source ip as the load balancer making the connection to your web server just just a common theme how you've seen dns logging has actually been really uh hard for us to do because before server 2012 r2 you really had no good way to extract dns logs if you're

running uh windows dns uh you have to turn on debugging and then that dumps it to a flat file and then you have to parse that flat file somehow get it to your sim debugging on a domain controller is a bad idea even microsoft will tell you that so what we ended up doing is using the proxy logs and aaron will talk about how we've enabled the client firewall that's logging all of the domains that you're visiting as a client so we've used that kind of to search any dns or you know to augment the fact that we can't really do dns at this time what was that yeah we could always do that

we tried to do that we tried to do some things with flows but uh the the parsers didn't play very well i mean it we have some ideas bro is one of them uh we do have 2012 right now so we're probably not going to go the complicated route and just get dns logs off of the domain controllers uh there are some more interesting rules that we could we want to write like the the length of the domain and uh some some other some other things that you want to do but so far we really haven't gotten a chance to do that but those are some good ideas uh finally some basic custom rules i'm

sure you've seen some of these alert for an ip address that connects to multiple hosts in your dmz across your entire enterprise sure you have legitimate vendors that might do that but why would one ip connect to three different web servers that have three different functions so that that's that's something you can alert on there any critical ips signature if those aren't tuned yet i think you have bigger problems but you shouldn't have critical ips firing all the time that's just an easy rule to set there suspicious countries connecting to your network i don't know if you do business with all 200 plus countries you might have friendly countries you might have not so friendly countries most sims have

geo identifying abilities so if you see an ip address they automatically say this is coming from china you can create a rule this is every time you see china or russia connect to my dmz create a rule or alert for that multiple failed vpn logins obviously that's to detect some sort of brute force on your vpn servers simultaneous logins from the same user that's rare alerts from demand from the domain admins group uh you shouldn't be using domain admin across your network if you are uh don't uh alerts on new user agent strings this is also from the proxy so if you have the data set for every single user agent string that's been launched on your network and

you regex that out and say all right here's my baseline for user agent strings anything outside of this throw an alert that's just uh that's an easy regex to write and then all of a sudden when you have a new security someone tries to run some new tool that you don't have in that data set uh you've got you've got some some interesting forensics there so this is the network side aaron's going to go into the client side and some of the things we did there thanks so on the client side this is where we really like to call this more of a proactive forensics approach our goal is to try and collect logs that normally to

to get that sort of information you'd have to do some sort of a dead disk analysis you take an image fire up the forensics tools dig in try to figure out what ran when what network connections it made all that sort of thing our goal was rather to get this logged as it's happening real time pull those into the sim and be able to alert on them and or if we need to go back and investigate them be able to just make it as simple as a sim query so some of the types of logs that we wanted to pull are client firewall logs if you haven't turned on the client firewall on all of your endpoints you're missing out

on a very valuable source of data turn those on feed them into the sim and start searching into them additionally getting some information about what applications are actually running on your network this can either come from application whitelisting um you know if you've got windows 7 enterprise or above you have access to app locker you don't even have to turn it on in enforced mode just log pull those application execution logs in and now you can see what's being executed and when sysmon is a fantastic utility and if you haven't kicked the tires on that you're really really missing out and i'll show you some examples of how that can be used in a moment other types of

event logs that might be interesting are when is a new service installed on your endpoints you know if you can alert on that and you know that it's not associated with a big push that's coming from your applications team who's installing some new software that's definitely something that you want to dig in on and look at as long as you're kind of being prudent about the types of logs that you're bringing in you can really get some interesting stuff out of this and by bringing it into the sim we may not even need to take that step of going to a forensic investigation now the problem though is typically sims have been focused at let's pull in from

the servers and you probably have less servers than endpoints now if you just flip on logging on all of your endpoints haphazardly and feed it into your sim the system is going to go down so we need to be a little more judicious about what we bring in so i'm going to talk about some examples of how we can kind of source filter that down to to get what really matters for endpoint firewall logging if you have one of the standard endpoint security suites chances are it has a firewall component that can be enabled and what it's going to end up doing is logging back at that management console that management console can then be configured to syslog

over to your sim and so that's the most normal way to get those events in what's awesome about these logs is it will tell you the exact application that was running that's something that even the best next gen layer 7 inspecting firewall is taking a guess at but they can't give you the exact application like this can from there you can create some very simple rules i mean would a user ever actually run ftp.exe from a command line probably not now admins yes admins yes and you can separate out those out and have different rules to trigger on your admin accounts um you know powershell you know if there if it's a a bad red teamer who's still migrating

into notepad with metasploit there's an easy detection that sort of thing so you can get pretty pretty interesting stuff um some of the gotchas though normalizing time that comes from your clients can be difficult if you have a laptop that's left your environment and it's been out for a week or so when it comes back and uploads those logs this is something that the sims don't handle very well those logs are going to come from your your management appliance and the sim is going to take those events and say these just happened despite the fact that they happened four days five days maybe a week ago uh and they'll trigger on a fence and say it happened

at this time and if you go circling at that exact time you're gonna be like uh no it didn't um another consideration is when these these clients come back you may get an eps burst when they dump all these logs that they've been offline for a while so it's one of those questions you want to ask your sim vendors is how do they deal with kind of spikes in your events per second but really the best way to to manage this is like i said source filter look at what some of the more chatty protocols are decide whether you actually need them on your network i mean some of the ones that do a lot of

broadcasts ssdp net bios that sort of thing do they really have a need to be in your enterprise if not block it at the source it's better to disable it but also go ahead and create a firewall rule that blocks it if it's going to try and go outbound there are broadcast applications that you're going to have to allow though take for example dhcp if you can't get onto the network it's not very useful so but when dhcp comes online your clients going to make a broadcast out to port 67 and say give me an ip address well every other client on that same vlan is going to drop that packet and say this is not

legitimate traffic and when all that comes to your sim it's going to be a couple hundred logs and your sims gonna say someone's port scanning this vlan that's not what's happening so you need to make some decisions about what you actually want to log and when you're creating your firewall logs it might be fine to create one that just says drop inbound port 67 and don't log it's probably not going to be you know it you are kind of curtailing some of your information but on the long term it's really going to going to pay off and having cleaner logs coming into your sim and you know just a side note whatever endpoint firewall you're using make sure

it's not something that can be manipulated by your end users it should be securely managed upstream so here's just sort of an example of a sample firewall and the colors aren't coming out too well but what i really wanted to show out here is you know the log as it's being sent to the sim is showing up as though it was september 18th at 10 55 however further down in the the payload we see that the start time was actually on september 16th and so that's exactly the issue that the the sims going to have a problem with so when you go back to investigate these things it's very important that you take in into

consideration not just the time that the event was triggered on the sim but the actual log source time as you're going through it um but as i alluded to i mean there's just gold in these in these logs you've got the domain that it went to the exact application that was run the rule name that you had created in your firewall so all these things are that are things that you can key in on and and try to to alert on within your sim now windows client event logs this is always a difficult issue with the sim windows of course doesn't support syslog natively we've got to find some other way to get those logs in there

one way that you can do this is by using native windows functionality use event log forwarding to bring it into a central place and from there fire those over at syslog otherwise check with your sim vendor they probably have some sort of an agent that will sit local on your host that's going to cache those logs and then upload them securely securely to the system um but again this is something that can easily overwhelm your your sim if you're sending every single event log from your clients your sims just going to crash now your sim vendor will be happy to sell you more eps and to sell you bigger boxes and that sort of thing but uh like we said

we wanted to do this on a budget so check with your some vendors check your your client see what you can work out xpath queries if they support it are an excellent way to minimize what's coming into your sim so this is a way of searching through your windows event logs and picking out only those individual events that you actually care about if you want some good examples of the types of logs that you should be looking at there's a fantastic paper that was published by the nsa called spotting the adversary with windows event log monitoring if you go through that and i still wouldn't recommend that you pick out every single one of those events and try

to bring it in for your organization that's probably still going to be too much do some xpath queries against your own box see what's normal what makes sense then pick out only those ones that do do make sense to bring into the sim but you know some obvious ones that you're always going to want to see event logs are being cleared key in on that users are being added to a privileged group key in on that and then app locker or emmett logs i mean if you've deployed emmett and you're seeing something triggering on a dep or aslr violation that's something you're going to want to kind of want to investigate further and leveraging application execution logs is

something that i think is very interesting that the sim vendors haven't really gotten their arms around they're starting to recognize that they need to do this but there isn't doesn't seem to be some sort of a common framework to say application execution happened at just the same way that there is to say network flow happened or network traffic happened so this is something that we really had to spend a lot of time customizing around parsing out what is application execution and recognizing it as some sort of an event now you can monitor for your white listing system to say these things got blocked that can give you some information about you know something that shouldn't have happened but what's

going to be more interesting in the long term is what actually got allowed to run because that's what got by and that's that's where you're getting actually owned advanced attackers aren't going to drop new payloads they're going to use native functionality against you and abuse it against you so you need to baseline and know what is normal execution and then you can start looking for those abnormal executions like really long encoded strings associated with powershell definitely something that you want to key in on applocker i've mentioned a couple of times that's kind of uh one of our old go-to's but it just doesn't give you the same robust information that sysmon does with sysmon you're getting the full what was

launched what it what its parent process was the hashes of it and the user that did it so going back to that first slide that kyle brought up of that phishing email i mean anyone see anything that that looks a little odd here i know it's a little bit hard to read but it's saying the parent image is windword and the command line that windward ran was it launched a batch file out of the temp directory if word is ever running a batch file out of attempt directory you've got issues guaranteed now if you are application whitelisting good stuff you won't even see this log you'll just see the blocked execution which is great but if you're not this is

the sort of log that you can collect and take to manage in management and say this is how we got owned and this is why we should be whitelisting so it's still good information to bring in types of tools that you you want to look at and this actually references some things that we've brought up in presentations in the past these are uh tools that would be normal for an admin to run but if you see your average end user running these from the command line you probably have issues wmic definitely never going to see an end user do it half of your admins probably struggle with using those tools as well powershell you might if it's integrated

into some of your normal scripts and so again you need to have that full command line and that's where you can do some great searching against that these other ones typically you're never going to see an end user run them so you definitely want to monitor for that sort of thing and these are the ones that are very unusual i mean typically even your admins aren't going to run these i.e exec you know that gets called from powershell a lot if you've been watching any of the the red team talks lately they they love these tools so we need to start looking for them and seeing when they're being used against us so here's sort of you know going back to

the whole picture of what we talked about from pulling things from the perimeter all the way down to the endpoint what does it look like when we're actually doing an investigation so we have an alert that would be triggered in our network if there's a large outbound flow of data to an ip address that we have not previously whitelisted meaning it's one of our trusted partners it's going to trigger an offense when it comes time to investigate that you know if we looked at just the perimeter firewall for that outbound ip address the source ip is going to show up as our proxy typically and that's not going to give us much information since we've sampled our flows from the user

side of the proxy we can actually figure out okay what is the actual source ip address that's great information digging into the actual proxy and client firewall logs if we just search on that destination ip we're going to be able to extract what was the user agent and does that match up with the application that was said to be running from the firewall rules if it's saying that it's user agent ie but it's a randomly generated string of letters for the executable you've got issues so that allows us to validate that it was actually a legitimate program if we have the full url which you would get if you're doing full ssl decryption you may be able to actually pick out

what was the file that was being uploaded at that point in time and then you pick up the phone and call the user and say hey what was going on um you know for us luckily fortunately to date we've only had those calls where they're like yeah i was uploading a bunch of photos to gmail and we're like well don't do that on our network so that we don't have these triggers and that sort of also gets the word out that says yeah big brother is definitely watching so it's a good way to go into it but you know this whole process since we've got all this stuff into our sim and centrally managed and cleaned up this

can be done in a matter of minutes rather than having to go out to the end point dig around and try and figure out what happened so what do we see as sort of the failures of sim vendors at this point at this point there's just a lot of payloads that aren't processed to their fullest extent especially when you look at the newer generations of stuff the next gen whatever that's doing that application identification they aren't picking that out and correlating that back very neatly um we sometimes in our experience i've seen poor communication between the vendors that are making the products and the sim vendors themselves so those log parsers are getting broken on a regular basis so

you know another thing that you need to do if you're managing a sim is from time to time do a search just for unknown events or uncategorized events you're going to see a lot of things in there that you may have thought were being properly processed by the sim but weren't that's when you need to start opening tickets if you haven't opened a ticket with your sim vendor at least once or twice a month you're probably not paying attention to it enough to be be honest um a lot of the vendors will have their own you know ip reputation list and things like that that stuff in our experience is really becoming less and less

valuable as everything's starting to be moved behind cdns or even the bad guys are putting everything behind cloudflare that's not going to be very useful at all it's just noise that you probably don't need so you know don't tell your sim salesman i said this when he's taking you out to lunch but you probably don't need to subscribe to those if as long as you have some clean traffic you can find a better signal in your your own stuff timing is still a major issue the idea that an endpoint might not be sending logs in real time is that something that they grasp with and there also still just seems to be a lot of lack of support for the

non-standard windows log types if you say i want to pull things out of the sysmon logs which isn't sitting in security or applications or something like that they're going to look at you and say why and you have to kind of explain this is what i want to do and it's really cool and then you have to jump through the hoops of trying to get it properly qualified so that they it's categorized properly within the sim and there just doesn't really seem to be any correlation around application execution at all and that's where everyone in this room comes in really i mean we're the customers of these systems in many cases and if we don't start calling our sem

vendors and saying this is what i want they're not going to implement these things i mean we we actually unfortunately have a very good relationship with our sim vendor to the point where we have a quarterly call with their developers where we're saying this is what we're trying to do these are the things that aren't working can you please make this happen and what we always hear as a response is well we would love to but what we need to do is have this come forward from multiple customers so i'm begging you guys as customers of your sim vendors call them up say this is what i want to do let's start getting this there

if you aren't spending the time to configure the basics and cleaning up the rogue services and whatnot you're really wasting time with your sim and money your sim is only as good as the signal it gets so this is why we advocate taking the methodical approach add one source at a time make sure it's looking good and clean tweak it if need be don't do anything more until you've got that source in and it is painstaking and it does take time and that's why two years later we're just now giving this talk saying we are feeling pretty good about our sim but we still feel like we got a whole lot of work to go so

your end goal just needs to be a low noise environment that enables you to actually detect when the anomalous behavior happens so if you have any questions i would be happy to answer some of them now or if you want to hit us up on twitter twitter later we're we're glad to do that yes so after you've applied the filters do you find useful for all those client event logs sysmon app locker that and things like that um about how much data are you typically seeing from the average workstation in terms of the number of events as well as like megabytes grenade okay so the question is after we've done our filtering down how much our eps per from

each individual source of the end points are we seeing or how much per day are we seeing um honestly when we we first started doing this our chattiest log that we were trying to bring in was app locker and in a single day you could get 30 megabytes of data if you pulled everything in so the first thing we said was that's not going to work if we're pulling that from all our endpoints do we really need dll application executions compared to just exe executions that's a simple i know you would want it but when you think about you know thousands of endpoints feeding it that's the type of decision that you have to make and realize that

just getting the executable happening that's far better than nothing so that trimmed that down significantly i would say on a daily basis just the endpoint logs that we're pulling in from from an endpoint is probably on the order of a couple of hundred logs and it sounds minimal but in our experience we really i mean for for event logs yes but for your client firewall that is the top log source that's the top talker yeah because you have rules enabled for every single client times the number of clients you have so that is the largest log source consump consumer but for event windows event logs yeah not that much windows event logs is a couple hundred a day so i can

expand dramatically on that so um the windows firewall logs generate two messages 5156 5158 you do not need 5158 for any usefulness process command process execution and process terminations 468 4689 those top four right there you do not need termination because you just want to know when something executes those top four right there will equal more than all the other logs combined i can get my lovely license for my personal lab stuff and all my research is a one gig per day um if i turn on all the stuff that's in my windows logging sheet very similar to what they're talking about which is everything windows firewall process execution sysmon fully fully loaded with uh module executions and all that stuff

um i can easily fill in a day one workstation one gig but if i filter and you can actually get that off my website i have an nx lav com file to give you an idea how to filter stuff i can easily get 10 clients 10 times the amount of clients into that one gig per day license and logly for example it is very noisy but you can also then taper it off over time saying well keep it's 30 days and then i'll age it out um but yeah it's it's huge but the value is ginormous and i would say you know something you can do is again and we've said this in other presentations don't let perfect be

the enemy of good start small if you need to if you do nothing more than just have your agent that's able to collect logs and able to be tweaked when you need to to increase what logs it's bringing in so if there is some sort of an incident you can go back and turn and ramp that up but you know start with just what you need i mean if you looked at the nsa guide and all you did was enable collection of one event type ntlm lateral movement with a non-domain account to show passing the hashtag as things start there it's better than nothing and then from there you can just start ramping it up and see where you get

comfortable on it at that point so another great trick is creator know you that has all this turned on okay and then you can drag your workstations you want to investigate into that who use added so that your your log will spike up so i do this for a lot for windows firewall because it's so noisy that if i'm investigating somebody i need these 12 clients added to this ou and now that's automatically turned on and then i can turn it off if i'm not doing anything there's no other indicator that i'm getting from the lower level loud logs you can totally control that from gpo to apply stuff on and take something um that was but anyway the extent was uh

create a special lu that turns all this stuff on drag systems into it as you're investigating so you can bring the logs up and then when you're done investigating bring it back down and then log for the short course stuff like a new service you see five systems with a new service for example okay i'm gonna drag those five into the bucket that turns all this stuff on collect it short period to do your investigation then pull them back out and you can get all the nice data come to a conclusion and then pull them back out so ou's with just the workstations you need are awesome any other questions yes sir do you have any problems

so your dns example of getting rid of specific ports i feel like my security team would have the field they would be disabling valid traffic versus if i have a limitation in my license to just collect that on a syslog server and just knock

so the question is basically do we ever have issues with our security team being angry of us turning off certain logs in order to try and filter up at the source rather than trying to collect them all and then process them um so you know one of the fortunate things that we have is you're actually looking at the entire security team for our organization right now so we arm wrestle sometimes if we need to but we don't have too much internal conflict or strife on that but you know in organizations where there's that you know one thing that you might consider is uh and something that even our se from our sim vendor recommended was you can always have a separate log

collector that does the filtering that is capturing everything that isn't the sim and you know it's just being a syslog destination where you're keeping it all but you're then doing that filtering there and push that into the sim where the ones that you care about are going to be correlated and dealt with that's one way to approach it it's really one of those things where if you've got a tricky organization you've just got to you know tell them well if you want to have all these logs give us some of your budget and we'd be happy to fill it in this is cheap right so yeah that's great oh yeah we actually got another chip

we got one other question i would say that uh if the item on the security team usually i'm the one pushing for more lobby and the ops team is pushing back because they're looking at um uptime and response time of what they're mastering their monitors so that becomes the uh trade off in the negotiation point between security and ops not necessarily between security um and so we end up having to figure out what's the best bang for the buck and how to alternatively collect the data we need if

yeah i mean all of these i mean all of this what it really comes down to is the very one of the very first things that we started saying is think about it as an overall design architecture from the get-go and you know one of those things where they they talk about an ounce of prevention is worth a pound of cure i mean if you're thinking of it up front of and having these discussions with everyone as you're getting to roll this out that can save you a lot of pain and a lot of the these heartaches and arguments internal strife after the fact so other other questions or over here

i can say no i don't so sorry sir

the question is any experience with deploying windows powershell logging you said from the endpoints and i think that's one of the the newer features that are you talking about the new features that are coming out with the actual powershell logging we actually haven't gone that direction yet it's something that we're interested in the future but for now we're really just digging into sysmond for that because sysmon is going to give us the full powershell as it got executed uh while we wouldn't necessarily get get the logging if it's inside of a script uh we get at least the notification that the script went if you if you took carlos perez's class he talks about the logging specifically but

it's either i think he compares sysmon logging with powershelf initially until you start talking about capturing uh transcripts so if you want that level you definitely want to turn it on but power cismod will give you enough unless you have a lot of storage again i mean one of the things that um it's no secret we love white listing and so we are doing that and so for us it's not as much of a concern that we wouldn't get the full script uh as it was being transpired or whatever because that script's not going to execute and it's going to be blocked by the white listing level so it's more that we have to worry about if they did

if they were as an admin running a power shell um i do have extreme experience i'm releasing a windows powershell logging gg for this your main focus is going to be there's two logs there's a windows powershell log under the security log area basic logs and then under microsoft windows there it i have a pretty loud voice but i bet they're recording it sorry um and then under the microsoft windows there's another powershell log so if you want to know about the subject a research what ben 10 is doing uh ben 10 actually b be in to ian he's a powershell guru and he actually slowed down my release of the powershell logging so a you're looking for event ids 501 to

execute you cannot get that unless you create a default profile in that enabling the two variables you need so every session and powershell triggers and then you'll get the command line logging most a matter of fact the the laughing thing is you know that example they gave you the menu with 10. exe that that malware in my workshop it was 9.exe so they had the next version in their prezo that launched a powershell backdoor to ukraine in the case of my particular sample that i was evaluating lab in order to capture that you must have a default profile but this particular malware actually executed the execution policy bypass no profile so no logging occurred but the fact that i got that as

an alert is going to tell me i won't get powershell logging so i have to use something else or just re-image the box but you're looking at having to do those two variables you're having a look in the the windows powershell for a 500 or 501 you won't see those unless you do those variables and then the windows powershell log has some vents in the 4000 series but they're more like a transcript dump so you'll have to parse those out they're going to be excessively long and cut off in some of the sim solutions and that you can detect the powershell executions that were not executed with powershell.exe and yes thank you microsoft there's a

way to launch powershell without using powershell x exe um so watch out for that if you want but yes there is there's some info coming and you know just some one thing i would add just you hear powershell powershell powershell powershell if you aren't up on powershell start reading up on it but also something else that i think is very important powershell should not be talking to the internet for most of your endpoints ever ever ever if you go back to our schmukon presentation we put up a slide there we showed an example stack two firewall rules if you need powershell to have network access internally one that says allow powershell internally one that says block powershell all

and that cuts it off right there so yeah it was fun with our last pen test they were complaining about how uh they couldn't do something and and he was talking about it it's gonna be difficult for me to come back and do something like i'll do it i can actually run powershell so fun when you restrict it just to yourselves so a lot of these ideas are from pen testers so pinterest come in they come in and they they beat us up for a little bit and we're like all right we're going to build these rules and then they come back again next year like oh you're stopping this stuff it's like yeah you have to take it a

step up and they do and then we're like alright build rules for that so most of this stuff comes from pen testers kind of coming in and doing their red teaming in our environment any other questions well thank you everyone for attending hope you enjoyed the talk and feel free to reach us out to us in the future