So You Want to Build a Threat Hunt Program

ah here we go good afternoon brian oh brian disappeared as we uh wait for brian to get back

there we go apologies are we good now brian i think we're good all right thank you thank you um so here we are this afternoon with uh brian bowie he will present be presenting so you want to build a threat hunting program so i guess we have the man necessary here um brian has worked approximately 15 years in security on a very diverse background starting with his eight years in the us army to getting his feet wet in cyber security by becoming an application security engineer finally leading teams and being a senior threat hunter brian has a passion for knowledge and security a desire to make an impact commitment to challenging and rewarding work utilizing the ability

to collaborate across teams to solve hard problems brian has two little girls and a little boy when he is not building legos or having tea time with his daughters he is reading the latest white papers or tinkering building destroying something in his lab um he is the i've never heard of that work the kung connoisseur thank you he is a connoisseur of spicy peppers and dabbles and growing his own brian is eager and exciting to work side by side with the best and the brightest in the industry with that being said greatly appreciate it ernest thank you very much i hope everyone's having a great conference so far and good afternoon good day wherever you are um

as introduced my name is brian bowie and i i wanted to throw out a talk about building that threat hunt program there's so many resources out there there's so many things that you can you can dabble on whether it be a web blog or a vendor or whomever and they'll tell you about threat hunting and some hypotheses and things that you can run with but no one ever talks about how you get to stage one so uh that's that's essentially where this talk is is going so as as mentioned my name is brian i've been doing this for just shy of about two decades or so um that started in the army from uh intelligence work and background

also doing infantry and then as well as working through a number of organizations eventually leading to where i am now as one of the um one of the senior incident responders and i moonlight inside of the organization as the hunt lead to organize together teams as we go out and be doing threat hunting missions throughout the organization so how to make sure i get legal to sign off on this i don't represent the company that i work for nor do i represent any of our customers everything that i say here is my opinion and my my position so what we're going to go through really quick is we'll talk about threat hunting because you can never

have a talk about threat hunting without going a little bit of an overview of it um we'll talk about how we're going to build a mission statement and where we go with that mission statement and we'll talk about the the lanes of how we begin hunting um there's so many different models there there's so many ways of feeding hunting but there's really two primary avenues of how you begin to do it um we'll we'll talk in in part about the period of the pyramid of pain and how it applies as well as dealing with maturity maturity is a huge part of every threat program and we need to make sure that we pay attention to it along the entire life

cycle and then finally we'll talk about a process overview that we've developed internally inside of our organization and tweaked it a little bit available for everyone else's consumption and then we'll give some resources so you can go forward and you can do what you need to do inside of your organization or wherever and begin to build your hump program and then obviously hopefully we'll have some questions and answers at the very end so when we think about threat hunting everyone really has their own definition especially security vendors that obviously they're trying to sell a product they're trying to sell a service and they all have their own mission statements and their own definitions some go as far as making it seem like

you're some ninja in the night others will create this monolithic definition and will try to say that you need to be some uh cyber like expert top of your field in order to be able to do threat hunting and we also have mssps obviously that are trying to sell those services available to customers that don't have the the capability or the manpower to be able to do hunting or really any security services and of course they're going to have their own definitions it's really all over the place and then for anyone that isn't part of them uh that they're going to have their own even mom and pop shops that do have a threat hunting capability

they're gonna have their own definition and while yes they're technically there is the nist control for threat hunting that that has now been released uh as part of ra 10. um nobody really aligns to it directly so what we want to do is we want to kind of take a step back and and define what threat hunting means to us but what also tends to happen is when we look at all those other organizations and all of those definitions they they really all harp on one thing whatever hump program that you do it's gonna find the baddies and you're gonna be walking away with apts and cuffs every time and it couldn't be furthest from the

truth um that's great for the more senior organizations and the ones that are at the tippy top of their maturity curve but for those of us that are just getting uh just getting started and just starting off and building that program there's a number of questions that immediately come to mind and it really all boils down to where do i start how do i even begin doing this what do i need to be successful so first we want to build a definition a mission statement something of what we define as a stake in the ground of what threat hunting means to our organization uh to our team and what we're going to rally behind whenever somebody

asks what what are you threat hunting and what does that mean to you we particularly like this definition i'm very partial to it i'm from creating it but all of the words here kind of create a picture of a of a target board and i'll discuss this in a little bit later but essentially what we're doing is we're creating a larger target versus a bull's-eye of going right after the malicious actor every time so each of the words and phrasing here is really important uh at least from the way that i've designed it is threat hunting is that human driven as in it's not a blinky box it's not something that's racked and stacked inside of a data

center and claims to do threat hunting it's a human-driven approach that's proactive as in we're not responding to alerts from a sim and it's also iterative it's a it's that search that you're able to rinse and repeat every single time that you're able to always be able to do no matter all the variables that that are included we're going to look through our endpoints we're going to look through networks we're going to look through data sets sql servers network traffic you name it we want to be able to look through everything and the reason why is because threat actors are varied in skill across the board and all of the different types of hunts that

we do we need to be able to account for those different types of data sets that we need obviously we're going to look and we're going to try to detect militias of course that's what our role is inside of security we're out to find the bad guy however what we also tend to forget about is trying to locate those suspicious and risky activities there's some activities inside of organizations that could be more damaging than a breach could ever especially organizations that deal with compliance that are beholden to laws and regulations you could experience fines significantly more than what gdpr could ever do so we want to make sure we identify not only those malicious activities but

we're looking at those suspicious and risky ones as well and we're we're doing this in a way that we're not again kind of going back to the the beginning we're doing this proactively by looking at those ones that have already evaded our current tool set we have the latest and greatest edr platform we have the latest and greatest sim we have all this awesome networking capabilities and all this out of the box stuff that's created we want to look in between those lines and we want to make sure that we're finding items that aren't already discovered so now we've designed our our threath mission but i want to make sure that i really harp on this point

we're not just out to catch evil we need to make sure that we grab that suspicious and risky activity and we always are accounting for it within our hunt program we're not just going to be looking for that malicious actor we're grabbing suspicious things that are occurring in our network that has fallen out of scope and we want to make sure that we chase those down eventually or maybe pass them to another team to do that we want to make sure we're grabbing those risky activities and we are accounting for and changing that behavior with our environment if we have dev teams creating back doors within applications that's a very very risky activity while not necessarily against the law

or against policy it may it needs to be a behavior that needs to get addressed so while giving this talk and giving this presentation inside of other organizations i i'm often combated with isn't just threat hunting something uh whether it be red teaming purple teaming some blinky box inside of a inside of a data center what what may have you i want to make sure that we we kind of keep this in the back of our heads threat hunting isn't all of the other stuff it's very much a blue team activity it's not red team red team is very much offensive operations we're talking about penetration testing social engineering how to attack something and while yes um we do

skirt the line on purple teaming activities by maybe doing an emulation based hypothesis where we then act like an actor and we try to find those particular breadcrumbs that an actor may do we're not doing that as a dedicated role that may just be part of a hypothesis we fall very much inside of the blue team but we are not incident response we are the other side of the sock ir coin if you will we're also again going back to that mission statement we're not automatic we're we're not just a blinky box that just always turns out some type of an alert or some type of notifications we're going to find things and other times we're not

so we need to make sure that we account for those and then as always results aren't guaranteed we need to make sure that we let our leadership know that when we do go out for a hunt we aren't always going to come home with something it just happens to be the way of the way of the beast

so as we begin going down that that process

can't see the slides i am so sorry

all right hopefully y'all can see the slides now ah all right so uh going back from the the last slide to this one here my apologies uh for um for missing that front half when we begin our hunting um program we we really need to settle on two different paths we have an unstructured path and we have a structured path and unstructured is really just a fancy way of saying ad hoc doing ad hoc hunting and ad hoc hunting is really simple it's very agile very flexible scope we can say we're only looking in this part of the network and then we can change that as we need to as we begin looking around there's normally no no dedicated team

oftentimes we're finding a news article or a blog or something and we're saying oh have we seen this before and then you go and you perform an on an ad hoc hunt you're looking for evidence thereof that an attack has occurred however there are some major drawbacks it's normally undocumented uh when somebody goes and they they read that blog post and they go and they they begin hunting they don't really uh document what they found did they find anything what were the queries that they used if they were using a sim um how long did it take and really what happens is the success and failure rate is really unknown so we don't really have a level of

effort measured on how successful those ad hoc hunts are what we also tend to find is that when we look at hunts historically we don't know if somebody performed an ad hoc hunt because of that undocumented nature how do we know if somebody looked at a dns desync last month or a year ago how often are we looking at those so we need to make sure that we kind of account for the lack of documentation that comes with ad hoc hunting and from a structured side it's it's a little bit more straightforward we're organizing a team of individuals and for those of you that that may have dedicated staff you already have a number of individuals

assigned to you and you're able to perform your your hunts woodstead team hopefully that team is cross-functional in a way of expertise they understand intelligence they understand endpoints network forensics and point forensics etc for those of you that don't have dedicated teams you may need to pull on other bodies across your organization and hopefully you'll be able to pull on a number of teams and get a number of different individuals that specialize in those roles those would be really really helpful as you begin to build out your hypotheses and go and look you're going to get different viewpoints you're going to get different understandings and different levels of maturity of those particular individuals everything from

the most green junior soccer analyst all the way to your most amazing incident responder and digital forensics person we're going to make sure that all of our processes are task and ticket driven we want to make sure we are doing an iterative approach we can rinse and repeat it every single time and while the details may change the overall structure and the framework is there available for us as we need it we're going to make sure that we focus our scope and each of those hypothesis themes or each of those hunts if you will they're going to be dedicated to a very specific area or very specific target anything that falls outside of it will

obviously should be documented it should be followed up on however we want to make sure that we don't have scope creep what will happen is it'll end up blowing out your hunts and you'll you'll start going off in a million different directions we want to avoid that that's where the structured nature comes in what will happen is when you do finalize your your end of your hunts you're able to build solid metrics you're able to provide those details back to your senior management and show the success and show where things can be improved and where those items that are available to be measured are and you're able to ultimately give that back to your management team obviously there are

drawbacks uh when we talk about when we talk about structured hunting it takes a lot of effort and there's a lot of resources that are really needed um everything from tooling to expertise to everything else in between and then we also really need that approval for management if you don't have your managers signing off and saying that yes we're going to be doing structured hunting what's going to happen is they're going to start asking questions around what are you doing with your time and why aren't you doing what you're initially hired for so we want to make sure that we're being transparent and completely visible to our management team and letting them know upfront what we plan on doing and

what we plan on delivering so they can be successful in their future reportings up the rest of the chain i do want to make sure i make a big note here just because you have a structured program never ever should you have to worry about dissuading someone from doing ad hoc hunting everyone should always be curious if you're interested in wanting to look whether you're on a dedicated hunt team or not you should always feel inspired and empowered to be able to look within your environment and to find items in an ad hoc way just keep in mind that we do have some uh some downsides when we do that because it's normally not documented if

you can find a way to document it and reduce those impacts by all means it's great absolutely helpful all across the board so for those of you that are aware of the pyramid of pain awesome for those of you that aren't it was created by david bianco back in 2013 and it essentially measures the different pain points if you will of all of the different artifacts of all of the different items within the cyber world when we begin to look at how we want to monitor and how we want to try to combat those items everything from gathering hashes which is trivially easy as well as blocking them within our tool sets all the way to going against our tactics

techniques and procedures ttps which is more behavioral and what's really important here is we have to kind of set the tone of where we're going with hunting it's really easy to grab a hash list and then look across your network that's relatively trivial but we want to do is we want to try to avoid that where possible we want to make that as part of our sim let the sim do the heavy work we want to try to divide and conquer here if you will turn that into an automated capability get rid of that out of your scope for your hunts make it to where it's not something that you need to worry about it then starts shrinking down the need

of what exactly you're trying to deliver so what we are going to pay attention to is the upper half of the pyramid of pain we're looking at network and host artifacts which really give us the history of a breach those are the items that tell us there was something here we need to dig further we need to be looking at those ever-evolving tool sets changing code is relatively easy so trying to hash them and look for hashes isn't really going to work however we can only profile so much so we need to make sure that we just have a basic understanding of what those tools are and try to design content around them to such a way that it's not

monolithic in nature we need to constantly evolve and then lastly we where we really want to be paying attention where we really want to try to drive the knife home is at the ttp layer and the reason why is really primarily because we want the attacker to do one of two things we either a we want them to leave we don't want them to be bothering us anymore or b we want them to be changing the way that they act within our network to such a degree that we can find them and then kick them out forcefully and this is probably the hardest part of threat hunting is combating at this layer we're combating directly at the

adversary but where possible this is where we want to focus our attention so we want to kind of focus more so at the top half of the pyramid of pain so it's great and all we've we talked about how we're going to be starting our our program where we're going to be going uh with it and where we're combating our adversary but we need to make sure that we're paying attention to the maturity of the program we can say that we're doing all of these wonderful things but do we have a way to measure that do we know where we're going by knowing where we were and for those of you that are familiar with this quote yes it is from maui

the shapeshifter demigod of the wind and sea hero to all but on a more serious note squirrel has done an awesome work before being bought by amazon on measuring the way that these stepping stones are for a maturity curve now there is a little bit of a problem here um it really lacks detail we can see initial and we see a couple things here it doesn't really give us a whole lot so instead we should probably focus on a matrix this way we can actually check box our way along and gradually measure how we're doing our hunt program how are we moving from traditional ad hoc hunting to leading and being the best in breed

across across the industry we need to make sure that we step from each of these particular stones along the way and become the best at what we do and it's really relatively easily we can take each of these stones from traditional or initial all the way to leading and we measure them against ppts the people process and technologies we need to mature each of these to make sure that we're kind of moving that bar along so we just plot it out in the spreadsheet and we begin marking them off where do we want to actually improve how do we begin improving those items and essentially it becomes a checklist we don't need to be so hard on ourselves

that we're not marking something off if we're not achieving 24 7 we can change that around that's okay but what we want to make sure that we're doing is we're not fudging the facts we don't want to we don't want to lie and we don't want to disprove ourselves we need to be honest and we need to show hey we haven't measured up to this particular this particular ppt we haven't moved beyond this particular part of the matrix and that's okay because that shows you areas where you're going to need to improve and where you can begin to move that bar along the way and for those areas that you just don't have anything just mark them off

eventually you're gonna get there no one says that you have to be able to turn over the world's leading hunt program overnight so start small and grab bite-sized chunks and move along the way so great now we got all of this information but how do we begin plotting this out well let's take a look at a ten thousand foot view and we can break down threat hunting really into four main phases again we're looking for that iterative approach so we need to make sure that we document our way of doing processes if we're moving from pre-hunt to hunting to post hunting and reporting we need to make sure that we show each of these phases along the way

and taking those four phases we can kind of break them down a little bit further at least for us what we've done is we've taken pre-hunting and we've essentially established a way of gathering the resources that we need before we can actually really start churning and burning through a bunch of data we want to make sure that everyone's up to speed we want to make sure everyone has the appropriate resources needed do they have access to that tool over there do they know where that particular log source is do they need additional training and things of that nature we want to make sure that we have everyone as close as they can to the same bar

before we really get started within our hunt program and then finally we want to make sure that we're we're gathering around a number of hypotheses around what exactly our particular target is for us we we use the grand theme approach where we decide we're going to pick a particular theme and then we generate hypotheses around that theme and then begin to hunt it when we move into the hunt phase as you'll notice this particular bar is a bit different from its brother above and it's not a solid bar it's because these three items all happen simultaneously as you begin hunting you'll need to modify that hypothesis you'll need to change the way that you look at the particular data sets you're

going to need to look at false positives differently than what you look at real data and you're going to need to change those queries appropriately and you'll make sure that you keep notes along the way this is going to really help you help the rest of the program looking forward as we begin documenting everything excuse me when we move into the final wrap up it's really the the post hunt we want to make sure that we pull everything together and we kind of give everyone that sigh of relief we're done we're no longer out in the woods let's go ahead and pack everything up and start heading home we're going to wrap up any of our

tickets any of our final notes we want to make sure we do a gap analysis what what roadblocks were in place that prevented us from doing a particular hunt are we missing content detections or are we seeing log sources do we just not see something what it what particular gap occurred that prevented us from being more successful than what we were and then obviously we need to make sure that we raise those to the appropriate teams and get those items addressed i'm a big fan of lessons learned so as always i strongly believe that lessons learned should always be a part of any particular item whether it be incident response or a particular stock alert

there's always some lesson to be learned from some type of activity what anything that's gone good anything that could have been improved and things that went uh that went south we want to make sure that we keep those documentations we're constantly evolving we're constantly changing we don't want to become so static in our process that it then starts collecting dust and we have no way of evolving over time and then finally because remember we're management approved we need to make sure that we give back the management team exactly what they need we need to give them back those those static statistics and metrics around what exactly happened inside of our hunt and then any findings that came along

the way if happens to be more applicable to you creating an executive summary and giving that over to your executive leadership or your senior leadership we want to make sure that we close out any master tracking that we're doing whether it being in the master ticket that houses all of your hunt data or whether it happens to be a particular capture group of where you're putting all your documentation and then finally if you found any content any queries anything that you can say we can alert on that in the future and we'll we will make sure that we find bad we want to bring that over to the the content teams or or the sock teams

and make sure that you're constantly building out those alerts that you're constantly taking those easy detection wins and pushing them further down the pyramid of pain and then finally for those of you that lead you're going to need to prepare for the next hunt it's going to take time so we need to make sure that we account for that so how does this look over a timeline some individuals in some companies will say that they do week long or two week long sprints and that's great but maturity is really going to dictate what how long your particular time schedule is sometimes it could take a month to really get all of the parties that you need

inside if you happen to have dedicated people that will shrink down accordingly for others they may need longer so depending upon what your cycles are in your maturity really dictates how your time schedule is really going to be set up what's really most important here is that we have a start time and then we have an end time and then that way we can make sure we collapse everything in between and we can measure everything else that that has occurred here and that everything has been documented so wrapping this up we have um we we talk about the why we're out to find the malicious we're out to find risky in suspicious activities within that network

we're looking in every place that we can and we're making sure that we're focusing on the upper half of the pyramid of pain hopefully we're trying to attack the behavior of an adversary versus their tools or finding artifacts later on and we want to make sure that we have a diverse team whether that team happens to be specialized in one capability or whether they happen to be pulled across different parts of the organization different views is going to give us different results on each of our hunts and then we need to make sure that we understand how long that particular hunt is going to be whatever your maturity is is really going to dictate whether or not you go

from four month long or four week hunts that take a month down to two week sprints it's all it's all dictated on how fast you can move and what resources are available to you and then lastly we have the maturity matrix we now know how we can move that bar and mature that program and continuously evolve over time and for those of us that are just getting started i don't want you to walk away without having some type of resource available to you so i want to make sure that i disclaim this i do not endorse any of these particular vendors or products these just happen to be the ones that i know and that i i suggest um

only only because they fill a particular need obviously if your vendor or product of choice isn't here it's whatever fills the need for you if you use something different and it particularly fills that need great use that it's really whatever is available to you you're going to need a documentation store where we're structured or even in in ad hoc mode we need to make sure that we're keeping some type of documentation down we need to make sure we have a ticketing system and that we're leveraging all of the ticketing systems across across the entire organization we want to make sure that we raise tickets to the appropriate teams and we're keeping documentation ourselves on each of our hunts i critically

believe that communication is very key so having a chat client is really going to be helpful especially now in this time of covid we want to make sure everyone's able to communicate and we obviously we're building reports so we're going to need some type of document processing and no hunt can ever be complete without some type of event log or a log manager if you don't have some type of central place of where you're storing all of your logs that's a problem and that needs to be addressed first and foremost you need to be able to centralize all of your logs available to you into one particular place and even if you're doing particular historical hunts where you're grabbing

really fringe data data sets you should still be able to have most of your other data your edr data your network data your host level data all funneling into your one big data lake and then finally we need to make sure that we have management buy-in again we need to make sure that they are endorsing the program to be ultimately successful and then finally before i turn it over to questions i want people to keep in mind hey this is a fluid concept we are constantly changing we are constantly adapting adversaries are doing things significantly different than what they were doing last year let alone five ten years ago so our program needs to constantly be able to

keep up with change documentation processes tools it's whatever works for you and whatever works for your organization if you need to change something change it that's what the lessons learned is therefore and if you need to raise issues make sure that your management team is receptive if you need to push items lean on that management team and have them push those items for you to help bring about change and with that if anyone wants this particular presentation it's available on my github page and i open the floor to any questions

it looks like we have one uh so far bryan um how would you convince a smaller organization without a dedicated security team to implement some form of threat hunting into their business strategy i would say that your best way of convincing them is to pull on available resources we we all have strict budgets we all have limited personnel where possible even if you only have one person that is only going out and doing hunting it's better than none so having a dedicated team is obviously the best route however even if you have one person that takes a couple hours every day that goes in and documents i went and i looked for dc desyncing i went and looked for mimi

cats i went and looked for scheduled task abuse having those particular items will help move the bar better than not doing anything at all

thank you for that response brian um does anybody else have any questions you can also post in general chat if you uh don't know where the q a is

all right brian it looks like that was the only question we had for today mom is there words

can you hear me bro yeah no um i'm sorry um no really i greatly appreciate the um the chance to be able to talk here today and thank you everyone for for coming out and listening to me ramble uh obviously if you ever have any questions feel free to to reach out i'll be more than happy to answer them wherever i can and uh hopefully i'll see you guys again thank you hold on brian don't don't leave us yet we actually had a question at the last minute um baseline skills to build a threat team or what is what are the baseline skills to build a threat team curiosity that is the absolute minimum requirement for any type of hunt team

you can have senior individuals and they're going to be the ones that really make the program ultimately successful and can really bring it far along the way however really it's curiosity you can have very oftentimes we often have very junior analysts as part of our hunt teams it's because they're very curious that we want to bring them in they will learn the most during those particular campaigns because they're learning and thinking outside of the box outside of their normal day-to-day so i would say that the baseline requirement is just being curious kind of sticking outside of the status quo don't don't just assume that everything's a checkbox or this is the way that the adversary has been documented to work

things are going to change so being curious and saying that looks strange and then chasing that thread i would say that would probably be the most key and baseline requirement everything else is just a plus awesome thank you for that response brian um and of course you said your last minute words already so with that being said if no one else has any more questions um and you still are intrigued on what he had to speak about today he can be reached out on github.com forward slash polar bear god forward slide presentations if you want the presentation here but you can also reach out to him that being said thank you brian for today's talk everyone else have a

fabulous day today and that concludes this presentation