Tools Don't Hunt. Tools Find. Analysts Hunt.

hi welcome to b-side charlotte my name is rick rutledge i'm a senior security engineer with polarity and i'm here today to talk about uh hunting in general the title is tools don't hunt tools find analysts hunt so i'm going to kind of walk around the building blocks of a sock this is a out of the building blocks recommendation guide from sands on building your sock as a world-class operations center so i'm going to walk through kind of each of these chunks and give you what we've seen in real life from experience as far as best recommendations what we've seen problems with and how to get around some of those issues so the first section i'm going to

talk about is technology so within the technology component of a sock it's broken up into groups so the technology piece is is often the easiest one that the low-hanging fruit that when you know first building a sock or even updating that the applications the uh software that's used in those hunts and investigations that's usually the first thing uh brought on board um often without a process in place so we're gonna go through each of these categories so can i talk about some of the problems we see of uh socks that are now mature using software that was bought before the process or even through the software cycle and updating so identification protection detection response and recovery

and generally what we see is response and recovery there aren't quite as many tools for those categories and i'll touch on that when we get there so the first section identification this would be anything for recognizing the threats anything that logs or you know your sim risk analysis and then of course user and asset management user and asset management being a big one to maintain when there's changes users changing roles updates of new servers coming online so these pieces are very important to keep up to date as much as possible obviously and then log management we've got a lot to say about log management as well missing logs things that aren't uh updated in that realm can often leave you blind

to threats the protection category this dives into the is referring to the proxy firewall endpoint protections any usually the stuff that goes in first for the first line of defense and security when you're thinking about just keeping people out um as well as the ssl and encryption layer um but one of the more often ones that i've been working with lately especially with uh you know financial and and other sectors where where private information is important is the dlp or data loss prevention component and then within the detection group we have all of our threat intelligence internal and external products and that's you know there's a wide variety for all of these categories of tools

and it's usually a matter of looking between budget and what fits into your mode best often with thread intel you've got the option for open source or or not um community driven uh is a is a big one for thread intel and we'll dive into some more of those uh in a bit but um as well as on the detection side we have soar which is often one of the newer categories um often used for enhancing detections logs that have come in to understand if there's a threat there or not and then of course traffic analysis understanding patterns um and then putting a little bit more of the intelligence around the logs that you're already gathering like the

machine learning with ai rf uba and uh just correlation of rules to understand when something means when an alert means something or when it doesn't and then of course your basic log monitoring and alerting when a happens alert on this or when when this event occurs we want to know about it every time and so the detection phase is often where most socks that i work with spend a majority of their time in focus and then on to the response category this is where we're talking about you know ddos protection endpoint detection and response and this is generally less less emphasis on the on the response more people are usually focused on detection and finding the problems um

response is is usually for the more severe ones like like the ones mentioned here but this is this is one category where the vendors are a little bit behind as well and then of course recovery uh several stages of recovery you've got your vulnerability remediation products that are just you know going to clean your endpoints and within the virtualization environments you've got the ability to refer back snapshots go back in time uh take configs back to the way they were when things were working and not infected and then of course ransomware remediation has been a big topic over the past few years so all of these categories combined in subcategories of different products and tools that companies have

that's a lot of tools so 25 categories here we've usually got more than one tool in a category most customers that i've worked with in the past uh a lot of times they have several um and so this quote from the cso of palo alto he's you know on the high end he's looking at 130 tool average for large enterprises um and i have been to custom companies wherever that's the case and then there's uh usually a team dedicated to figuring out which tools they need to get rid of but that team is not generally working with the sock directly or in the weeds so you know that that takes some tuning it takes work to get those right

um but with that amount of tools um there's often a gap between the analysts and and the end game that the the finding the the hunting for the iocs and the bad actors um so that's where we the one of the best fixes we can get to with that is focusing on the people in the training and then aligning process to that this next one is just a quick snapshot of just a handful of the tools available so it can be overwhelming um to say the least that's where when you get your process defined first and then bring the tools in behind it that's that's uh going to give you a much finer scope of

targeting those those threats so the the biggest problems with accessing all that data obviously the the time consumption involved in flipping between products so when you have five or six tools to do one thing you generally come down to one or two that you like the best and each analyst ends up doing something different so we get what we always see a lot in the field we're different analysts three different four different analysts on the same team have three or four different ways of doing the same thing so pivoting between tools is a big one that also requires more training so if you have those you know ten tools to do one thing uh that's

ten different uh things that everybody has to be trained on you generally do have a smee for each product but uh then again you have if you have people using different products you have different results and then process deterioration comes out of this with the fact that they're using different things getting different results taking shortcuts because there's so much data to sift through if you know you're going about this manually you often will get burnt out looking at all that information and then there's obviously a need for better correlation between those separate tools to help with all this um obviously multiple monitors help and most people go that route but uh there's only so much you can do i've

you know i've had up to four monitors i generally end up ignoring one so so you know you can only do so much with what you put in front of the person you need to make that more efficient less to look at and and you're going to get better results so you know the first piece the first slide i was showing you the different pieces of the sock so the technology stack again that's usually where socks start their invest investment in building out their sock and the people in the process come next but over a time once the stock matures you see more focus on process and better use of those tools aligning those tools with the established

processes will eliminate unnecessary tools as well as focus the tools you know that can do 200 things you may only need it for 50 of those things then you can focus in by your license for what you need and use it for that specific task and then building out an easy method for collaboration so you know there's stock analysts that are using different tools one guy finds something bad somebody else finds you know no no result if they're sharing that information you're less likely to miss something plus you're less likely to duplicate work so collaboration is very important in any sock and the biggest issue with that is making it easy for the users to do

if you have a product that's cumbersome to use when collaborating analysts are less likely to do it and you're going to have more gaps and then consistency between analysts again that's refining the process and tool set giving everybody a very tight uh not not too narrow but a very tight set of processes so that they can focus on finding the the proper things that they need the proper data that they need the next component in this cycle i'm going to talk about is is the process so looking at the different processes i've seen around different socks we've come up with some some good ideas and bad ideas that we've seen so i'm kind of trying to put these

together and highlight the problems and challenges and and how we can work around those so preparing for the hunt one of the quotes we've highlighted highlighted here is uh successful threat hunting depends on a thorough understanding of the operating environment meaning you know the targets where they're at what threat actors want understanding the environment's a big part of it um knowing the system's uh values is number one so understanding if there's some data on a system that's that's not sensitive or it's not as important um why waste time hunting that uh area of the network or looking at logs for that system but knowing the value of those systems to outside actors is key also understanding the orientation of

their system so if you know you have a system that's unimportant that you're not focused on can it reach a system that is important so looking for lateral communication things that you know ports open that shouldn't be configuration items that uh you know could be changed behind your back and you not realize it because you don't know the configuration so knowing the configuration that's intended for that device is another big one so that when there is a breach you don't let it sit for six months while they do what they want with it companies often prioritize logging over training so you get a new system new a new network of servers stood up in production um the first

thing they do is turn logging on so it's taken care of and it's alerting so done right the problem is that analysts are getting this information they see an alert that says it's a red flag go look at this something bad's going on without context around what that system is there for and the responsibility of that server um it's not as valuable so they could get an alert that might mean something on one system and nothing on another depending on the job of those those servers also knowing what ports should be open when they get there so giving those security teams information more information and context around the assets they're protecting is key for them to be able to quickly

respond and then forming a hypothesis um the next phase in the process would be figuring out okay we've got this information about threats potential threats and known threats coming in um you have to formulate what what you think is is happening the hypothesis you know figures out what the goal of that threat actor is what they're trying to gain um access to and why um this is usually the hardest part of the process is to try and figure out what someone else is thinking when you when you see that there's there's something going on you can use risk-based approaches is a very common way to do this to figure out okay this this risk exists here's the exploit for it

it's meant to do this one thing so that's probably what they're trying to get that's where the threat intel comes in to feed you that information so using those tools together can help you format a hypothesis um other challenges with this is making sure your thread intel is accurate and current and relevant um you know 90 of thread intel a good thread intel feed might not be relevant for my network i may not have certain services running so so you have to sift through that information make sure you're looking at what's what matters to your organization and this also you know the challenge challenge with this one is also a historical information matching up what you've seen in the past

is not always as quick as searching for a ticket that had this ip in it before sometimes things change ips or names change there's there's a lot of variables in there that can cause the historical data to not be as helpful as it should be so normalizing that and making it searchable and relevant and attached to the current incident um that that's usually a big challenge in formatting this hypothesis and then of course identification of security controls knowing which firewalls they should be going through which proxy server should be serving up that web page and understanding the flow of the network behind behind everything you're looking at and then log collection once you've got your hypothesis

you've figured out what you're looking for and what the threat actor is looking for now it's time to go gather the evidence to figure out um what else he did or what is all the what was all the data that that leaked out everything that you um can put together from the logs is going to help in understanding what happened um so the log reliability and availability is usually uh one of the issues but that that's got to be verified now you got to make sure you've got logs from the server for the time frame in question um and if not you've got to take more extreme actions grabbing network traffic that sort of thing

um and logs for you know any investigation are usually coming from multiple sources with most customers i work with every now and then you get someone that has one centralized log server that's a lot a lot simpler if you can if you can manage to get there but it's not always the case generally you have different departments that have different budgets and have different products so piecing those things together is key to understand the whole picture some challenges with putting that information together is that as i mentioned many log sources don't easily port into a central management system so you may not be able to cross certain network borders so it has to live somewhere else

many high value systems legacy systems they often don't have a normalized log format so you may have to do some manual work to sift through those logs and figure out you know what's relevant what's not in extreme cases you can't get the logs so your only option is to go do live forensics captures on the on the endpoints looking at traffic network traffic that sort of thing and then for the next phase of our process segment of the sock is the investigation so once you've formed a hypothesis collected your logs and your evidence it's time to investigate and try to figure out exactly what happened first thing to do is to confirm was this threat realized was was this successful

uh an attempt to get data or whatever the target was trying to do to what extent did they get the information they were trying to get or did they succeed or not and then establishing a timeline of events that's obviously key for logs and this may be where you have to pivot back to some of the other steps maybe you need more logs maybe you need data from another system that you've discovered was involved so often you go during the investigation phase back and forth and capturing more information and then determining the impact and figuring out what this this breach or attempted breach meant for the company biggest challenges on the investigative phase the budgetary limits

along with you know your mandate to operate a risk-based approach often limit the scope so operating with a log management system that you've only purchased a certain amount of logging for you maybe you couldn't store logs for the firewalls but just the servers and so you don't have all of the connectivity attempts in between that's an extreme case but but i have seen similar situations and then evaluating information from a narrowed viewer or bias can can cause you to miss data so you know looking at that narrower scope of information shorter time frames than than you otherwise would be able to um can just cause you to not be able to see and have blind spots

uh along the investigative route and then reviewing logs in isolation and manually um just any manual task in general is going to be cumbersome and mind numbing and cause cause you to miss things people start uh glossing over when they get to page 50 of the log for example and you're not going to catch everything manually that way so that was the technology piece and the process piece of the stack the next component i want to talk more about is a little bit more about the people so talking about some of the training with the actual individuals doing the job the training as well as um focusing on the the parts of the process that they need

to focus on so this is where we get into actually trying to avoid those mind-numbing blind spots so the focus of this section is going to be on the blind spots themselves um enhancing the hunt one of the things that we can do is or one example i found here is a it's a where's waldo graphic and there's actually an algorithm that someone wrote and i've got the link in this uh slideshow that i'll show at the end that um someone wrote it's probably like 10 lines of code it's pretty pretty uh brief but it eliminates all the colors but rad or dims them finds certain patterns looks flips it to black and white and looks for a certain

pattern and then uses the the simple code and it finds wall to 90 of the time to 95 percent of the time so using code to speed things up and help you get somewhere quicker um you know making use of your tools it's kind of the highlight here this is enhancing the hunt with the tools but also knowing what you're looking for so the challenges the biggest challenge i'm going to highlight today are the blind spots uh decimating disseminating that historical information among the team and getting that information out to everyone um that's that's uh key as well and then real-time collaboration between all the analysts working on an incident uh generally is something that uh most

socks you know can do 90 of the time 75 percent is probably more of the average i see that they they're happy with that um and that's where it comes down to more of a process and figuring out what tools work for your group for real-time collaboration so this is just kind of an interesting fact about about blind spots in general so the optic nerve routes before the nervous fibers are out before the optic nerve in each in every human's eye that creates a blind spot so everyone's at least one blind spot in each eye the way that our brain synthesizes both of those images together and overlaps them it is able to ignore the blind spots in

both eyes creating a full image so this is kind of a synonymous with how we're trying to look at the network and figure out when when something's not visible can we see it from another angle hunting where you're blind can give you better vision tomorrow so so figuring out and looking in those spots where you don't have visibility helps build out that uh that visibility for the future so an example of this is where you're not capturing logging data for example um your hunt might lead you there and figure out another way that you're going to have to capture information from that system so that you can see when there are incidents or issues with that with that

particular part of the network and this is based on the diamond model which for each thread actor you you've got four points four key points that you want to try to collect so who are they what data are they after and what are their capabilities and methods and of course what infrastructure do they use so the the idea here is that if you have all four of these pieces of information you could lose one and still have visibility to what that threat actor is doing if you only have one piece of this data then then you're single threaded and you're basically cut off if he changes his ip and that's all you had for example

so so the key is to get as much about each actor as you can from all of the different sources that you have and make use of the technology to gather that information and don't just stop when you see the ip address keep going um and get as much information as you can so something about you know the model does change you've still got eyes on that on that actor so this uh graphic i'm gonna switch to kind of some some graphic uh demonstrations here that kind of walk through some different ways to observe uh the information so the unknown threats everybody's got unknown threats that can affect their organization and then there's realized and observed

threat actors so you know that information it's coming through your threat feeds you've had attacks in the past that you've identified what was going on you blocked those there are no knowns known unknowns but there are also unknown unknowns so that makes sense you don't know what you don't know it's actually donald rumsfeld quote so that's basically telling you you you know some of the information you know what you know but you don't know some of the things that that are not in front of you and that's where hiding those are discovering those blind spots becomes more important so filling in the gaps i know this is a busy graphic but uh kind of walk through it here so in

the last one we had what we had already observed what we've already seen and we've we've captured that we know that information so the way to fill in the gaps of the unknown information is with your standard red team analysis looking at what could happen uh community sharing is a big one so talking to your peers other people in the same vertical as your company figuring out what they've seen that could potentially overlap and understanding and sharing information a lot of times i work with a lot of different you know energy vertical financial vertical they all have different specific to their segment communities and once you put in you get out and then of course collection of data

hunting in general um gut feeling is actually a part of that um and then there's the different threat feeds so you've got bad threat feeds which you don't there's no point in bothering with that it doesn't overlap with any of your organizational data if it's a bad threat feed that's that's what that means it doesn't have any relevancy to you but even in the good threat feed information now the subscription services that tell you all these cves and everything that that's potentially hitting the networks right now you can still eliminate a lot of that that has no relevancy to you things that don't matter to your network that you shouldn't waste time looking at so

you wipe those out you've got a narrowed focus and just a few small gaps so you take that knowledge of the that knowledge of your unknowns down quite a bit um and so then we want to look at the data it's the data itself that that you're trying to protect you have your organization's data and then data that is sensitive to your organization often those are those are two separate things so you don't want the bad guys getting any of it but but you want to start with the sensitive stuff so then there's the data that the threat actors are after usually it looks you know similar like this you'll have a small segment of

organizational data that is not as sensitive that they're going to get regardless trying to get to that sensitive data but where you want to start is is protecting that sensitive stuff that you know the threat actors are after often the most sensitive stuff would be the trade secrets and customer data that's usually what has monetary value or ransomware value uh the most valuable data is often what the threat actors are after um and then stuff that that falls into that uh that white space you know like account names passwords network addresses they can use that to get to the sensitive data sometimes but not always and then when it falls outside of the organizational data

the the larger part of that circle um that's old data usually that you could just delete so if you wipe that out delete it get rid of it things that that are still in your network that are not necessary for your business um that takes away a lot of the scope that you have to focus on um you can also often use that for honey pots so you can build you know a segment of the network where it attack tracks the threat actors to and they think they're getting somewhere so cutting out that information you don't need narrows down the scope quite a bit taking out just the sensitive data gives you a much smaller sliver to work with so now

we'll focus on that for starting our hunt this is an area where we know that we are vulnerable potentially that's where the threat actors are targeting that's what they can reach so that's where we want to start our hunt now overlay that with vulnerabilities so we have our unknown vulnerabilities and our cbes things that have been caught and picked up on before so we've got to narrow this scope down now so with your cves you've got what was discovered by the red team there's known and unknown there good pen testing obviously can find a lot of those automation both good and bad you have good bad pen tests as well that's what we're cutting off out of this circle

and then adversary reported reported information on you know direct web stuff like uh dark reading that you would find some of those cves and and exploits for those so that shaves down our known or actually increases our known vulnerabilities and shaves down our unknown vulnerabilities so we want to overlay that and what we're using here is a mathematical union is everything and we're actually one of looking at want to look at the intersection just the data that overlaps so putting our targeted data over the sensitive data brings down the sliver even further so we've got a much smaller segment that we're working with now once we figured out the vulnerabilities patched them gotten them out of the way and now we

can focus on what's still unknown now we look at uh known capabilities and methods that we've already collected so these are the the things that the threat actors are using um overlay that with our unknown segment of the network to take that chunk down even further so we have a much smaller segment of unknown data maybe where we need to start looking at logging and things like that in this particular environment and that's where our hunting is going to be focused and that's where we we come back around to looking at the tools usually the tools of configuring logging configuring the devices um to talk on the right networks to be able to access them

updating os is that sort of thing so all of the perimeter internal network uh database hosts applications that's that's what we're going to focus on for understanding in that unknown network what do we have there that we're that we're not monitoring today um and this is obviously the tools uh ngt uh we're talking about the next great thing so whatever whatever comes next after sim dlp antivirus that's that's basically the the stuff we're going to focus on for this information so now overlaying our detectability what we know we can capture what's visible in the network so what we can see not only what we can see but then what we can detect on top of what we can see

overlay that with the vulnerable portion of the network that we know the threat actors are trying to get to takes us down to the smallest sliver and we can see where we have no visibility which is where you're going to start your hunt highlighted in red is anything you can't you can't see you're not getting logs you're not getting detectability um we have we have we're blind to that section that's obviously where you need to start your hunt because that's where you're most vulnerable and usually hunting enables detection so you know starting to look at those systems you're not only seeing what this current state is but you're observing why it's not detectable or why it's not

visible and that hunt process alone can help you resolve some of these missing components in your network um so starting the hunt there and as well as in the where you can't detect anything about the network is critical to understanding you know whether there's a threat actor there yet or not um there very well could be in the near future thank you i am rick again i'm rick rutledge with polarity senior security engineer and enjoy the rest of the show