Dispelling the Myth of "Maturity" in Threat Hunting

all righty so I'm gonna start off with a quick statistic here in 2011 mandians entrance report said the average attackers well-timed environment was an astonishing 416 days out of all the IRS that they performed in 2020 that number was down to just 24 days one of the main contributions to that that everyone attributes is threatened so I think it's safe to say everyone in this room is here to bring back ideas on how to jumpstart their words or Kickstart their own knowledge so my name is Kelsey Seymour I've got about six beers in I.T cyber security overall and I've assisted with starting threat hunting programs in smaller orgs to White Smith So today we're going to talk through a simple threat hunting program at a generic company any you know third normal legal disclaimers and all that any resemblances as interesting glad to hear you think of that otherwise let's just hold all the questions to the end I do have time built in there for questions and I absolutely love questions please try and insult me I enjoy the challenge so moving on we're going to do a little quick q a here we are not going to get the right answers that's for you to figure out later in the presentation so the first question I'm going to ask here is how many people do you need in your department to start a threat hunting program so by a show of hands if you think it is a between one and five people in your department to start a proper threat hunting program raise your hand if you think between one and five okay how about B five to twenty High c 25 to 100 or D 100 plus so I think it's safe to say everyone kind of got the gist at the top and said hey congrats you're actually right I'll give you that one right now for free fun fact about Dave is the last time I gave it a similar presentation to this I had a saiso come up to me afterwards and say hey about two three years ago when I started in my current position I was talking with Gartner about it maturing our own cyber security program I had 37 analysts on staff said when should I start a threat hunting program and they said at least five years don't worry moving on is another question I got here a little bit Oddball but you'll see how it comes back Microsoft has a native centralized log collection software that is not as calm and is not agent-based who here thinks that's true show of hands who here thinks that's just an outright why it's false I'm just I'm just telling you things that don't exist so any hands for true any hands for false okay so to jump right into it here today we're going to do a couple things we're going to dispel the myths around maturity and threat honey I'm going to present to you guys an example fret hunting to detection engineering pipeline we're going to provide four practical example prep hunts and showcase some sort of infrastructure that you can use to help make this happen in organization without a lot of budget primer resources so to get right with it here what is thriving threat hunting is going to be proactive that's going to require some understanding of your own environment and require you guys to be Forward Thinking so get right to the lead of it here so there's no surprises list of spell submits first one is you need an army of 200 people guess what you'll need one or two of analysts some dedicated time in fact the 2020 sand strap Hunting report that I referenced earlier over 30 percent of organizations do not perform in-house threat money next one up here it is those animals will spend half of their work week just doing nothing but fret hunting so you're going to lose half your productivity right no about five hours a week in the programs that I was a part of is about how long the average analyst spends with this you're going to need specialized tool sets well guess what chances are you already have everything you need in your environment set up ready to go if not you have some extra stuff and you're ahead of the game and you didn't know it and finally a lot of people say you need those leading experts you know what I say a little bit differently let's turn to members I'm your team and to The Experts for your own organization so getting right to it here for an example threat hunting pipeline because one of the other big things I often hear is cool traveling is great we're safe I just keep telling everyone we're seeing for doing this we're safe for doing this that's kind of hard to sell to my upper management my upper leadership because again what what's the point you're just saying we're good we already knew we're good right so this is one way you can actually help provide real measurable tangible value out of your fret lines it is we'll start off with a fret hunt work a way through it and we'll end up as part of this debrief with some detections that you can integrate into your own scene or other platforms you have so generally I like to run this as a work week so we're going to start off with our hypothesis typically we'll do this during A Team stand up on a Monday stuff like that we're gonna all sit together we're going to talk and this is going to be a team-based discussion either split up in the groups or we have individuals that have their own ideas go their own ways and we try and guide the you know hypothesis there so threaded tells a good thing um guess what you don't even need Kate thread Intel Fields there's plenty of blogs articles resources from both vendors and just other individuals in the community what are you seeing for example uh what was it I think it was the 31st the uh progress move it I.T vulnerability for any companies that use that and I see some people nod their heads going yeah no I'm too well aware of that that is an example of something that's newsworthy that hey we might use that organization or let's look at the attack vectors behind that and try and apply similar Concepts to our organization from there kicks off the rest of the work week and that's where we're going to go into the analysis phase this is going to be that Solo or paired up research you're going to let people deviate off on their own do their research do their learning ask questions of you know more senior members of their team they're going to start realizing okay how would this threat hunt look for organization what kind of tools do we have to get the log data how can I parse it how can I go through it and then we hit my favorite part which is the testing phase so I did this right now raise it your threat Hunt is done when you hit that analysis stage so now we're under testing so this is trying to transform it more into the detection detection Engineering Process so here's the fun part does your query work so you know if you have some you know members of your team that are more Red Team friendly or stuff like that or even if it's something very basic and generic such as let's say outside connections from your servers into DMZ to the outside world those are relatively easy things to test so not only do you have your threat hunt let's make sure that all that analysis and that research and that planning and that testing actually was valid so let's make sure that it's valid in your environment then we roll into Friday for the debrief it's just another team based discussion we sit there everyone gets a chance to present their Lessons Learned hey so I researched this here's what I did here's what I found and then here's my detection and then as a group we can take that detection go past you know the last 30 90 365 days whatever makes sense that the retention we have and the type of detection if we have low pulse positives good true positives from your testing we can now move that into production and guess what we just did friend hunting once that week and we gained a new detection so right there's a nice kpi you can use to report up and say not only did we make sure we're safe we also implemented something to make sure we're monitoring this for the future so with that we're going to move into our example cases here so we're going to do some network based threat hunting now because I alluded to a little bit earlier an example hypothesis for this is our DMZ server should not make outbound internet requests the idea of posting something in DMZ is that something you're displaying to the Public Public should be reaching in you shouldn't necessarily be reaching out often if ever so we're going to move into the analysis stage so we use Network logs from our firewall so you can use data transforms on things such as netflow some things we did notice in this analysis stage in this fictional organization as we did in fact have a lot of connections coming from our DMZ servers out to the actual internet now some of you might be thinking oh no that's bad but for example let's say you are a cloud scene product and you have a agent installed on that server how's those logs getting shipped up they gotta reach back out how about EDR or other compliance monitoring Suites DLP anything that actually has to talk back your rmm tools for patching Etc so those were actually really easy you reach out and grab the vendor documentations these ports these addresses are these DNS entries guess what there's your exclusions so now comes again my favorite part how can we test this well hop on the server and he makes it outbound connections and see what happens so hopefully you know you run those same queries again and all the red lights are starting to flash and you're going yes I got it so we used netcat that's the the funny cat there um for testing that from one of our Linux servers and sure enough we rolled into Friday saying we got a new detection for all new unknown outbound network connection attacks this is great for things like web shells webshell gets dropped on your DMZ server chances are it's going to reach out to C2 and boom now you got lights and Sirens going on if you're EDR platform didn't catch it already or if your firewall didn't block it so some of the tools we used here was we had a paid scene not gonna line we already had it we used it why not and we also use Network perimeter logs that were ingested into it so let's say you're a starting organization you have a young security program stuff like that there are still things you can leverage for example you already have a network monitoring solution that ingests the netflow data and assist log data chances are that gets dropped off in a database you can query either undecider directly and then you set that up to schedule task crime job your favorite you know schedule one and there you go now you pay no money except for maybe a server to host the fastest off and you're all set and running or a good old classic is flat pile on your choice a graph set and or off you can just go through the plain syslog panels dump it on a Linux server with our assist logs as log Ng and you're Off to the Races so let's move into one a little bit more system server based here so in this case uh this one was done a couple years ago with all of the fun with lapses and every executive and board members seeing all these big companies getting hacked going oh gosh that's a teenager hyper are we good are we safe from teenagers so one of the things that we picked up on that was the use of dc6 now not to pick on too many uni-r platform vendors and everything else I've yet to actually see a out of the box PC sync detection that you at least as the customer did not have to already somewhat customize to make it work right so for analysis of this we use successful Windows Event logins and optic access so when we went to go test this we actually realized as we're going through this there's another really easy low-hanging fruit detection which is the DC machine account use so in domain controller's machine account has 90 88 99 of the same Authority as a domain admin and just by a show of hands virtually in your head if you don't want to share how many of your organizations are already currently monitoring DC account usage to make sure they're being used by the right IP addresses to the right servers so when we rolled into debrief of that we had that run to include any and all abuse of DC machine accounts based on Source IP and other classifying information via those successful event sign ins and then it actually turned into just a secondary career uh dc6 we actually found a lot more value out of the detections for abusive domain controller machine accounts because the next annual penetration test of that fictional organization went through that was in fact an attack method Tried by the pen tester and he was surprised when we texted him about two minutes later saying congrats so moving on from that again you know you haven't paid scene you might as well use it I can argue that and we just had basic domain controller logs that's often your first step of your first server logs you ingest now again got open source offerings so you can use things like you know onion security that's out there in the vendor Hall you can use elasticsearch which is kind of Premium nowadays gray Loft whole bunch of options there or as I'm going to head to earlier you can use Windows Event forwarding with powershelling scheduled tasks guess what you have a spare Windows Server that's free you're already paying for it um you can I think the last time I looked at Microsoft documentation is they say four cores and 16 gig of RAM is sufficient for 2 000 servers to be monitored with reasonable filtering and everything else involved so there we go with that we're going to go on to endpoint based hunting I will give EDR platform some benefit here they're definitely catching up on this I can still let them out typically a little bit with the amount of customization we can get with this style of button so guess what Office Products should not be spawning child processes that can execute Scripts so you think okay that's easy just put it in a flat roll right so you know we're going to use our EDR logs we're going to look at all the child process we're going to look at those job processes over the last 90 days we're going to see nothing right nope our two biggest culprits HR with the HR management integration with their HR System second worst compliance and DBS not compliance or was Finance in their VBS groups because everything runs in Excel with VDS So speaking of DBS it's time for protection right sorry we're going to test this we're just gonna put a VBS macro in a word file email it across because why not we're going to get a double dip here get our email security vendor pick it up no sweet now we're just going to do everything every security professional is trained not to do and you're going to do with the smile on your face you're going to take that suspicious email download that word document double click it and hit run now and sure enough alarm Bells went off process got killed like that because it was going through the EDR platform and we rolled into Friday with the customized detection for all suspicious children from all Office Products and we actually were able in that case to expand the scope to include Adobe Acrobat and again that's four or five hours of one analyst and we um actually turned the detection off on the EDR that our custom built one we ran it didn't catch a thing so with the end point hunting as you kind of heard we use the paint DDR 90 days retention now I'd like to Hope in 2023 don't worry you already have that you might just need an additional scoop now either for the 90 days retention either for the access to that data underneath the Hub but that's typically a very easy sell to offer leadership hey this much more and then now we can do this across all our servers all of our endpoints that have the EDR agent installed and just in case you're worried about that cost let's just quick get a quote from Splunk and get a quote from Azure Sentinel and then you'll very quickly agree to it or you can use things like sysmon and then some agent-based systems like Velociraptor OS query or you can technically do web is interesting on endpoints because a lot of them like to move for example this laptop is not plugged into a land that can see a domain controller can see member servers so it's really hard to do Windows Event porting with that which is why I tend to go more towards the agent base in that case so with that we're going to go on to the last of our four year which is going to be an application based hunting so kind of gonna blend a couple of these categories together here so this hypothesis was a pretty simple one web app should not be spawning native window shells or tools especially if they're installed on a Windows server and you could do the same thing for Linux with if you were out there on track one for the previous Talk of the GTFO base so analysis here we kept it simple 4688 with event logs or you could use syspon event number one or you have EDR installed guess what chances are process creation events are already in there too the world's your oyster there's 20 ways to do it now we went to testing so here's here's where I got fun we had a lot of false positives in our environment I'm going to pick on manage engine because it's fun if you're running AP stealth Service Plus guess what that self-service password reset for ad cell service with manage engine about once a week will run Basics before encoded Powershell what's it doing it's making sure your DC machine certs are trusted on that in case it's not a domain join device it just does raw Health app back into your domain Contours or 80 audit plus is doing a lot of that in the back end for scheduled tasks Etc that's built into that the other one was it's a long query there were a lot of apps I'd want to say the last time I touched a theory like that we had 29 LOL bins detected as part of it and it was about 15 or 16 exclusions but this also covered our entire environment for any Windows server to include IIs Apache and all that fun stuff so you roll into the debrief there all excited because we had new detections from suspicious activities for web servers we not only had our DMZ servers covered from the network perspective but hey let's say it didn't reach back out it was just a one-way in web shell we were still pretty good and if your organization were to have done something like this in let's say 2021 and hopefully not too much PTSD by that getting shown on screen and a lot of us hopefully have already forgot about lock4j and the the months and months of where in the world is this in my environment for example if you were a VMware Horizon customer and running unified access gateways exposed to the public as they were designed to be as a DMZ web proxy server boxer J would come right in through the UAG hit your horizon connection server and install webshop there were plenty of awesome write-ups about that now if you're reading threat intelligence and going and following along with the blocks going I'm glad we're good you would have seen there's a couple things that they did initially for Recon because there was that massive exploitation attempt so a lot of places got to chill not many of them were actually exploited past that website and then some people put up honey pots including Other Nation States and then they got web shell and realized oh hey here's how I can use the ones already installed by you know my adversary and then just from there about a month or two afterwards there's a mass amount of Recon so what commands are being run no snake who am I net commands and all that would have been triggered by a detection just like this so again I'm not going to necessarily harp on this too much he's a big scene Windows event logs alternative so you can repeat with me open source offerings or web with Powershell unscheduled past now I'm going to take a quick pause break here because anyone have any questions about the last and full size we just went through okay so one thing that is often brought up is a lot of these exclusions