Find, Fix, Finish: Generating Competitive Advantage With Threat Hunting

but [Music] without any more waiting we have uh Brody Nisbet here he's going to be speaking about fine fix finish generating competitive Advantage with threat hunting so big round of applause for [Applause] Brody all right how's the sound excellent thank you all right hello my name is bro isbit and I would like to thank the organizers of bides for uh allowing me to be up here today um it's a privilege I've been coming for many years and uh this is my first public appearance so this is my presentation fine fix finish I'm generating competitive Advantage with threat hunting and as some of you might know I work at crowd strike you can't tell from the media and I've been there for almost

11 years so I started out at crowd strike as an intrusion analyst and today I am the director of OverWatch I'm responsible for leading our Global team of threat Hunters across the various missions we have um so we divide up our threat hunting based on Mission and the purpose for this sort of thing is something I'll talk about a bit later but we have Espionage e crime inside of threat uh Counter Intelligence and private sector offensive um what else about me so there's only two more things I'd like you to know and that is I have two Professional Pet Peeves one of them is revealing tradecraft in public and the other one is public speaking so I'm

crossing both of my red lines today um and today I'm going to talk about two concepts Central to the way I think about thread hunting first is a simple workflow that we use and the second is the concept of competitive Advantage now I don't me against other vendors um that's a problem for the business but I actually mean against our customers adversaries where we're expected to um to bring effects against their adversaries year in year out and we got a lot of customers who have a lot of problems that don't go away um and I want to talk about these Concepts because I think a successful and effective threat hunting program starts well above the technical details

that we usually focus on and I think our simple approach is worth sharing even if it's a bit mundane and it's probably a bit esoteric to where I've been working for so long but I'm hoping there's takeaways for anybody that's on the blue team and uh Last Thing Before I Go on is I do apologize but after July 19 there's no jokes or memes in this slide deck so let's tackle the term thread hunting first in my experience most discussions about threat hunting quickly narrow down to a very low tactical individual level where becomes about tasks tasks like hypothesis based searching or looking for IAS or ioc's writing detections or even the sort of work that yous find

during an incident response or remediation and I think that's because of the way threat hunting as a term has got as many definitions as there are practitioners and businesses and for some people it's a part of a job some people say that they're threat Hunters but they're actually doing DFI it's a bit of a mess but because of that I find there's rarely informed public discussion that goes beyond that low level um and without that sort of abstract thinking it's kind of hard to set up Direction and determined purpose for a threat hunting team so let me say first and foremost that I view it as an operational art and I've picked that word specifically uh because with this view

threat hunting the way we look at it is the combined effort of the many organized to harness and multiply individual task level activities such that they smoothly achieve a much larger strategic security outcome right so for me specifically we've arranged our threat hunting workflows in such a way that the primary output of that is actionable threat intelligence so if I'd asked most of you before I started talking right now I said what is threat hunting you would have said something like it's detect something and then remediate it it's to find things right detections or whatever um I actually view it as the set of tasks that are needed to to uh produce actionable threat Intelligence on the

day that it's needed so that intelligence output becomes the critical input necessary to achieve our strategic goals and that is like I said earlier to maintain that competitive advantage against um our adversaries so let's look what I mean let's talk about competition in this very specific context so when somebody attempts to carry out their mission objectives on your network what in chues is a contest for control of that network uh it's probably not cyber war and it's definitely not chess um and too often you will hear or read unhelpful metaphors that make comparisons to these things as if the real world relationship between an aggressor and the blue team is some sort of rules-based symmetric game I can

assure you it is not and it is not contained neatly by scopes of work either similarly one might compare hacking to a battlefield and call it a War I'm trying to understand it through a lens best suited for kinetic effects in geopolitics now while I am actually going to introduce a workflow from the intelligence Community later I found it far more constructive to understand the relationship between a threat hunting blue team and an aggressor a real world aggressor as a competition and that competition has three primary drivers uh the first driver is capability which is really nothing new Under the Sun so I'm talking about the the people the processors and the tools I mean The Operators on both sides

so that and I for this dot point I mean their technical skill sets and capabilities the procedures that those operators have to follow uh cuz they have workflows and playbooks as well and constraints and the software and Hardware available to them next up is the intelligence contest or you could also phrase this as just knowledge as well doesn't have to be intelligence and within this um I we have terrain so that's uh the the wear and that's the contested Network in all its detail actually knowing about a network that you're defending might sound obvious but uh they're getting very very complicated and complex and uh certain vendors are now talking about cross domain threat hunting which is

necessitated by um adversar being able to move very smoothly across we across boundaries that we would draw an organizational responsibility like endpoint identity Cloud that sort of thing and they don't care about that they'll just go wherever they can and you've got to follow them so the next is actions or activity and this is the what when and how and that's what your opponent actually doing and profile which is The Who and the why and this is not strictly the same thing as the people and their technical capability um motive and capability not the same thing the final one and equally important and I'm glad we've just had chumpy's talk on exploit development life cycles and cost cost cost benefit

assessments but uh we also need to consider the cost to achieve a goal and the return on investment so for me this is the gradient between the cost to detect something versus the cost to compromise relative to the spending capacity of each party there are edge cases of course but um as we've just seen near all hacking is subject to the demands of money eventually and so that means you can um raise cost on your opponent so every part of the criminal ecosystem needs a return on investment to justify the risks they're taking uh government collection programs need to justify results with collection for their customers uh private sector offensive actors have Yachts to buy and regular

commercial red teams uh as we've just seen you know it's expensive to develop these exploits and um actually can't pay many bills with hacker mentality alone so uh in the long run resourcing matters so how does any of that affect what our operational design well if threat hunting is going to be effective truly effective against an adversary well then the output of that needs to address those competitive elements um and so if you remember that I said the purpose is to generate actionable intelligence well this is what we need to be writing about to make a difference these are the drivers and elements that we want to tip in our favor we want to tell Defenders about

the processes and tools being used by an adversary we want to tell them where in the contested Network the intrusion is taking place where it's been where it's going how fast it's moving and um and what it looks like of course and if we know about the adversary we want to share that profile information too and we want to do this or we need to do this actually in such a way that the all-in cost to disrupt an intrusion gets lower the more often we do it and on the flip side as we detect and degrade offensive capabilities and share intelligence with partners and customers and competitors sometimes um then we want to widen the

gap between the cost to compromise the network and the cost to defend it the purpose of thread hunting really has to be much more than to detect intrusions and remediate them which is the definition you will find on most websites for SEC for security companies the real purpose is to change this very equation in our favor and uh this is how we're doing it um and I think it's worth sharing because I think it can be emulated by much smaller shops as well the F3 EA Loop find fix finish exploit Analyze This is a target acquisition Loop I've borrowed it directly from military Doctrine uh with one exception I have not modified it uh from its source material

because it fit our problem just fine and I had no intention of Reinventing it as a cyber thing or something that I came up with uh if you're familiar with the framework as some of you might well be you or you Google the term later uh you may realize that the six stage disseminate is missing it's not a mistake in our case doctrines made room for operational reality from day one I've omitted dissemination of finished intelligence product because in our specific implementation um that responsibility falls directly across an organizational boundary with our intelligence team however we've not ignored that and threat intelligence and threat hunting really are very closely aligned um in my mind they're more closely aligned than

threat hunting and sock work so for us the flow of data information intelligence and requirements back and forth between a CTI and a threat hunting team really should be considered one of the more important data flows in an organization now what really helps f3a fit a threat hunting use case is that the workflow is as much about Target acquisition and rapidly actioning new findings as it is about carrying out a response and so even in its original text it makes great allowance for a range of non-kinetic effects and outcomes so it's a targeting methodology purposely designed to identify track and and engage a Target among an otherwise civilian environment and in that sense I realized if you think about it it's kind

of Ideal for the task of identifying an adversary that starts off their day with privileged or employee level access to a network and that's actually the point at which um our threat hunting team and I think threat hunting in general really came into play as looking for the Hands-On keyboard right the back stopping of security products and uh it's one of our original missions to be part of a defensive strategy and f3a suits that this distinction also differentiates it nicely from other threat hunting definitions where the stated purpose is usually a fairly straightforward to detect this thing and then to remediate it f3a is a bit different because it uh places significant emphasis actually on the

last two stages which is to exploit and analyze these are the stages that really provide insight into an adversary and allow for the efficient collection and processing of findings that start the targeting cycle all over again elsewhere and to help you learn about about who you're actually trying to stop it's also how you get out of playing repetitive games of whacka against an adversary that's well resourced and has a day job to be where you're trying to protect you will find it one might find it very difficult to sustain any long-term advantage over that sort of adversary by simply responding to them time and again without learning or improving anything so let's start with find

if you've got a Thro hunting effort or you want to start one I'm sorry but you need to start with a with Mission priorities you need to establish a clear and purposeful mission for your hunters and it can't be the same as the sock if you've got a security Operation Center you need to delineate between the two because they need to be doing different things without this without clear purpose and guidance your Hunters are going to find it very difficult to exercise any sort of disciplined initiative in their day-to-day work and and that's going to be crucial if you expect them to react quickly and effectively to intrusions that are unfolding in in uh in real

time next up if you want to hunt for specific people groups or organizations or or whatever else you've purposefully assigned to them then you need timely accurate and relevant threat intelligence about those targets questions like who are they and what do they want are far more than Idol philosophy having the answers can be crucial to understanding the next part like well if they're already on my network where would they be what would they look like when would they operate when would they be active actionable means timely accurate and relevant and you need the thread intelligence to drive a thread hunting mission after that we get to a probably more comfortable ground which is the detections engineering side of

things uh and for today I just want to call it h I'll break it into two groups groups of detections one of them being high fidelity and the other being building block So High Fidelity detections are the ones you probably always want to find an answer for and there should be fewer of these um building block is the one are the type of detections that are allowed to be noisy weak or experimental so that you can layer them up on one another to find new things and you need to combine those two to create a an effective hunting and monitoring program whatever you call those though in your own environment it helps you to build on the next part which is

establishing patterns and uh patterns and baselines so that you can then go on to identify anomalies context is one of the most important Tools in a thread Hunter's Arsenal and it is almost always overlooked whenever I'm I'm I'm reading about guidelines on how to do threat hunting context really is everything and to give you an example of that um there are things that you cannot that products cannot prevent obviously like who am I with an unexpected ancestor or specific FL flags on a 7 zip command line perfectly benign things that happen in large environments probably shouldn't happen in the environment of a think tank that doesn't have a uh you know a very large cardr of software Engineers

so understanding context is super important all of those things go into generating a hunting lead for your team and a good lead helps us transition smoothly to the next phase which is fix now in this context I obviously don't mean to repair anything it means to locate identify and verify what you're looking at as being valid for follow on actions here you want threat Hunters to confirm and identify what they've found okay the systems that you create for your threat Hunters the ecosystem that they're working in really should allow them to take those observations and compare them against known indicators behaviors tools and infrastructure and um and to scope out the entire landscape you're defending to see where else it is

first you find something then you need to figure out if it's a problem or not um and if it's a problem you need to figure out where it is this is the scoping process now how long this takes is highly subjective but ideally you want it to take minutes hours is okay too days and weeks it's far too long uh far too long if you want to be competitive on the day it's got to be minutes to hours um so once you find something you reach some sort of Confirmation and have it scoped and situated the next is the Finish stage which is kind of odd cuz it's not actually the finish at all um

what it means for us is well as thread Hunters the effects we can bring are actually limited and we're not a DFI team um if your threat Hunters are doing DFI probably not doing as much threat hunting as they could be um so the effects we can bring are kind of limited uh in this Arrangement what we want to do is provide a very early assessment for the team tasks with remediation enough information to get them moving very quickly in their in their job and in parallel to this we want to be preparing the entirety of what we know into actionable intelligence for later that is a longer more detailed knowledge transfer that balances timeliness and relevance and we

want to do this without ever forgetting that the immediate goal the goal on the day is uh is not to show the responders how we are and all the cool things we've found but to help them carry out their remediation detailed information that arrives too late is absolutely worthless um and so spending a lot of time writing up you know spending 4 hours writing up a brilliant report about something that ended 3 hours ago it's now too late no good so the Nuance here starts to get very specific very quickly according to where and how you're actually implementing this and uh and what the mission for this threat hunting team actually is let's just say in my world

at least the goal of fine fixed finish the first three part stages here is to put that actionable intelligence in the hands of responders uh this also actually brings us to the end of the average shred hunting definition I looked at like seven or eight from various vendors and they all sort of ended around here they all had many pages on sort of getting from the finding to the remediating but what comes next is how you actually generate and sustain advantage over a persistent adversary the next phase is exploit so that's to identify and capture important information relative to your adversary and your mission and of course we've already done a lot of this of this

during the transition from the um find to the fix phase like a lot of stuff gets written down part of part of the point of this stage is to capture it properly uh and we want to do that now because we want to triage these findings for consumers other than our sock or DFI team we want to take that data information and intelligence and sort store and share it where it needs to go so I'm talking about getting this information to your CTI team to detection Engineers the security Operation Center as I said um and uh maybe even depending on the nature of the threat and what's going on it needs to go to the security and it

departments and of course management and Leadership probably want to hear about it at some point too there are a lot of stakeholders for act for for threat intelligence and the output of threat hunting so if threat hunting sits in a privileged niche in your security organization then the capture sorting storing and sharing of these findings needs to be as seamless as possible and that's what actually makes this an important workflow it's all very well and good to say oh yeah we got to share it but you I mean you must you all have a lived experience of how difficult sharing information actually is sometimes sitting down and for us it's jur no one can get away from jur but um

it's about making sure those people know where to go to get that information the minute it's available right there's no slack or email write them a letter to let them know the workflows we have that you follow the threat Hunters are following you want them to seamlessly transition that knowledge to where it needs to go in the business uh you might have just realized though that proper case exploitation is just a fancy word for doing your documentation properly and that's all it is document this also sets the groundwork for the last part of this Loop which is also the slowest which is analysis I don't want to dwell on what analysis means cuz it's going to be

extremely different in every organization but the things I'd consider is that defensive RNG needs to focus on understanding your adversary improving your detections and tooling working on those workflows um building up context for your Defenders to better understand what's going on broaden and broadening or deepening your visibility making sure you've got a good handle on the on your uh Tex stack and the data that you're expected to hunt across so research and Analysis is obviously the slow part you can take so up until now fine fix finish that's sort of um the first few hours exploitation should happen very shortly after that but this stage can take days or weeks uh or if you're trying to change

organizational workflows it can take months uh months and meetings and so together with uh find we also see that the this targeting life cycle can travel at different speeds so as it might take a long time to analyze the outcomes of an incident after the fact could also take a long time to find something so this doesn't move at a constant Tempo it's constantly shifting and the other thing I'll point out at this point is that while data and information about the intr an intrusion and the work of threat hunters will at least flow in that direction in practice actually a lot of the nodes connect to each other as well often during the fixed phase you'll find

a new intrusion somewhere else right that starts off another loop during the exploitation phase when you're um sharing data someone might come back to you and go oh well it's actually over here as well start a whole new loop again okay and sometimes it goes backwards in very rare cases so that's actually it really a 40-year-old uh counter terrorism counter Insurgency targeting cycle applied for about four or 5 years now in practice to a defensive cyber security use case um in in summary what are we doing the F3 EA Loop is an effective way to arrange otherwise fairly disperate threat hunting activity um you can use it to address imbalances in the competitive drivers of

network control namely the capability knowledge and resourcing and when you apply this over a long over the long term and its scale that's when you start to learn about your adversaries and simultaneously start to lower the cost of of detecting them while raising the the cost of compromising your network so in this way threat hunting helps generate a sustainable competitive advantage that you can use to maintain maintain control of your networks over the long term and that's it thank you