The Threat Hunting Solution You Might Not Have Expected

okay so welcome everyone uh good afternoon thank you for lasting this long especially to make it to the last speech um i hope everyone's enjoying this and i want to thank the organizers i i don't know if they knew it but the last two speeches really set the stage for me really well so that like you'll have a better understanding of where i'm going with this so thank you thank um you also first time speaker first time attendees so this should be fun um and the last 50 minutes of talking and death by powerpoint all right so my obligatory about me slide my name is lee arkham i started my i.t career as a junior network admin in

the us army for six years with the 75th ranger regiment then i transitioned to the civilian life where i spent the next five years as a security operations center analyst this is the time where i really found my love for sims vlogging specifically sysmon absolutely love it and but i was in a position where i was responding to a ton of alerts that was just ant annotated and just really never really went anywhere so i was like i really want to change this i want to make this better i'm going to be effective and that's when i really came into i think what's called detection engineering now but i was like i love logs i love log

analysis let's go with this that's when i stumbled across cyborg security where i am currently and basically i'm a threat hunter content developer and i'm part of the customer success team and have been for almost two years so all my my daily work is a lot of logs it's building queries uh gaining context of the attacks so that's what i'm going to be talking about in my in my speech so a quick overview um i'm going to be discussing what proactive thread hunting is explaining the threat hunting methodology and process where ioc's behaviors alerts and static detections fit into this process because there's a lot of moving parts when it comes to being a blue teamer

and you'll have some people that are like no static detections are the best or now they're the worst i kind of just want to paint a bigger picture that they all play a part just like all of us um also when we talk about dwell time what it is why we want to reduce it and i'm going to introduce you to the uh my spin on the known unknown matrix and i'm going to build in some use cases there so threat hunting methodology what is proactive threat hunting productive threat hunting is a hypothesis driven process that we conduct to try to find maliciousness in an environment in it is the act of searching through logs or tools or anything in a network

looking for possible infections compromise or anomalous behavior proactive threat hunting is focused on behaviors and ttps as alex and curtin nicely put iocs are the easy wins but the behaviors and the human aspect of it is what we're really after searching for things like different ways of lateral movement privilege escalation or other tactics techniques sub techniques or behaviors you can find on the miter attack framework once again thank you gentlemen for covering that in such detail but these are things that are commonly used by attackers we base our hypothesis off of all these different items i emulate the malicious activity and put the hypothesis to the test the main goal is to find maliciousness before alerts or static detections do if

these alerts or static detections are triggered then normally i think it was said earlier as well um which is i'm super happy about being last a lot i can repeat a lot of things they've already said um but it's normally too late right if something pops or if ransomware you get the ransomware notice it's normally too late um but i also found a common theme that showed plenty of these steps leading up to the payload executing there were opportunities for detections of finding that compromise and finding the indicators that the uh attackers in your network so indicators of compromise or iocs are artifacts that are normally found during the ir or threat hunting process that

helps these analysts trace back the events to determine root cause or where this all started these effects or these artifacts can be hashes ip addresses or file names and plenty other things as well but iocs like this really shine when it comes to static detections um we're using them in a table looking for recently reported known bad indicators kind of like how av works with a hash table or if you have a sim that you collect a watch list for ips you know you throw an ip in there and alerts on it whenever it sees in your network um but these are things that iocs are really strong with but they have in common is that normally

they are very easy to change a sophisticated actor may change their infrastructure between or even during an attack to make it harder to attribute the attack back to them now that being said i would not completely dismiss iocs but i would not consider scanning iocs as proactive threat hunting these reports take time to be published and the incident takes time to be fully investigated so we can't always consider iocs after the cart standard or complete accurate relevant or um or timing if the uh the organization that was attacked if it took them a couple months or even possibly a year to investigate and they found all these iocs in the first month are those still relevant i mean that's

not even close to the entire right but if we get this you know if we get it early and we run a scan through our environment real quick we may have an easy win to say you know hey if we're in the same industry as someone and they got attacked and we take their indicators if we're the same indicator it makes it relevant it makes it timely it could you know be an easy win for us so why should you proactively threaten the main goal is to find the enemy before they reach their target and ultimately reduce their dwell time dewalt time is the the time from initial compromise to when the threat is detected

which usually means the sooner they're caught the better the better position the organization is in to handle the situation this detection could look many different ways from discovering tools being dropped in suspicious locations strange or anomalous command line arguments being executed or spelling errors in files that are created there is also the benefit of addressing visibility and validation a threat emulation if that exists in your environment when you when you are testing hypothesis you may realize that you don't have the logs required for some of the hunts that you're conducting which could create uh be used as a use case in your organization to say hey we're investigating this threat we've emulated it we're looking for certain

laws they don't exist in our environment they're not local which means they're never going to make it to the sid then you could possibly talk to the sock engineer or whoever needs to hear it say we need to ingest these logs now because this threat is super relevant to us so before i dive into um anything else i wanted to actually run through what the threat hunting process looks like from my side anyways so it contains six steps i like to create a hypothesis conduct research create a shell query and remember i'm a blue teamer so i'm dealing with sims i'm dealing with edrs mdrs so i'm using a lot of different uh language or

query syntax if you will um so that's when i'm talking about shell query that's what i mean you'll see it a little sooner we emulate the attack or pieces of the attack we actually use the invoke atomics thank you and then we perform log analysis and then we tune our query to reduce the amount of false positives as much as possible um an aspect of this process is to take all the research and intel reports that we have available um out on the internet and we really want to operationalize that intel not just the ircs so taking a little deep dive into each step for the hypothesis i created my hypothesis based on a ttp or behavior of

my choice or whatever is trending uh when it comes to emerging threats in this example i'll be using a word document spawning powershell behavior that's attributed to many pieces of malware uh like hemotec which has resurrect resurrected and you know it's back from the ashes yay but you know fishing someone and getting them to enable macros is is an old tactic and there's a reason they keep using it it's because it works right you trick someone in you get the shell you're good to go but we'll be looking at a word spawning powershell so the research um so i chose my idea and now i need to conduct research on it how do i get a macro created when it

runs for my word document and you know open up powershell that's the easy part because i asked my colleague which we'll call tyler who's a scripting wizard and i can just sit back and wait um but what other but when it comes to research what other conditions need to be present for it to work um you know is this a or is word fully patched is it the latest version is it unpatched um what are we talking about here so there's a lot of different aspects you got to think about once again building queries to be relevant to your organization you want to build a query on something that you're not using anymore right so what does my shell query look like

like i said i'm a lover of sysmon so that's what i naturally went to um but i'll create my shell query uh that i will use to detect this activity this can start as an extreme general query that you see right now looking for the relationship between two processes if you look at it um looking at sysmon log source i have the parent image which is what uh windward.exe and the uh child image or the child process which is imaging sysmon uh just being powershell.exe now i always like to put those asterisks there just in case the the attacker likes to move it around so it might not be in its uh native position um now you could always say you know

what if they rename it that's a whole different story you know takes a little more work but eventually we'll find them um but at the fun stuff of this is as you work with log sources as you find different log sources and get to know your logs you can start reading an intel report and say all right that's going to live here this is the process this goes into that field and really just start working things already in your brain and then by the time you're done with your research search you could have four or five or six different queries built in your head the next the next step would just be to test it

just a shout out i love the d4 report they come out like every two every two mondays actually next monday is the next one love it so emulation um when i simulate these uh simulate these attacks i do and i want to stress this in a controlled environment um as should you if you plan on doing these types of things in the future um i can't stress this enough uh the last thing i want anyone to do is go home or become a customer of cyborg security and say hey so lee talk to b-sides it's cool we're just going to detonate ransomware in production call it a day and we're going to check out the logs no

please don't do that but make sure it's controlled make sure you're letting people know what you're doing in your organization so that they expect if something does happen or um you know and i do want to say this is a caveat this isn't like red team purple team stuff um that i'm doing it's solely researched to say hey we're threat you know we're looking at these threats we're looking at the logs they produce so you might want to let your sock know though because in case they do have alerts that are looking for this type of activity if they start going off and they think hey we're actually under attack you could get ahead of that and be like

no just research testing being done over here um but the goal is to get the attack to look at not the entire attack but at peace the attacks look as realistic as possible um so in the example that i'm using again um with windward opening power or spawning powershell i don't really have to set up you know c2 infrastructure i don't need to start phishing my users i can simply create a word document open it from explorer click it and have the script run and then powershell spawns because right then and there i have windword as the parent process and powershell is a child process so like i said as realistic as possible without having to do

red team purple team stuff now the fun part is log analysis um i'm a log junkie i love digging through like a lot of data um yet i'm not good at machine learning and ai which is yeah i'm not i'm not a data scientist i'll say that um but now that's complete the next step would be to make sure the logs have been created uh first stop is event viewer normally because i'm normally dealing with windows events um or different log sources or but definitely endpoints being window boxes so i'll go straight to event viewer i want to see if the logs are being produced in the environment because like i said earlier if they're

not produced locally there's no way they're going to get to the sin and then how's my sock or anyone else or my threat hunters or instant responders going to be able to figure out what's going on

so tuning um our goal and my goal is always to try and reduce as many false positives as possible um if i have my query and it does become a common uh or if i discover that's a common thing in my environment which hopefully it's not but i have to go back to the research i have to reassess i have to look at the tool that i'm using figure all right what are the other flags what or how is powershell used by my users how does it look different in the attack and then i want to build something that's a little more specific so that it filters out normal user traffic and then you know once again

soft analysts incident responders threat hunters can point to it directly and say this is bad because in my example i use a parent image is windward and then i added the command line um which is tac enc which um normally says hey i'm going to be running an encoded command um and hopefully once again i'll know your environment but hopefully that's not happening a lot and you can use this as a quick you know um a quick win one thing i didn't mention is that threat hunting is not a linear process neither is content development like i'm talking the one thing i omitted on my graph was arrows just like a scientific process you can always go back a step or

two say all right i may have messed up or what what's different and keep testing hypothesis keep you know formulating queries keep doing all these different things it's never just a one and done so i just want to say that because it does get long it does get hard um sometimes tiring and exhausting but back to dwell time um i don't want to get into two of any metrics of dwell time but uh some of my favorite metrics came from the mandi and m trends report today once again dwell time is the initial time or initial the time from initial access is gained to when the activity is detected now the m trans report said that the

global median dwell time was down 21 days in 2021 from 24 days in 2020 the american median dwell time was 17 days an interesting point though is that ransomware not blue team or security people getting better is dragging that number down technically a big ransomware notice is a detection or it's a form of detection that you have been compromised and invest investigations that deal with ransomware normally have a dwell time of four days in those four days though there are opportunities for detections and cap and capture or capturing the activity needed to find the threat actors investigations that weren't involving ransomware had a dwell time of 32 days now before i continue on i want to talk

about uh my humans versus machine side ramp and every time i say that i don't know if you have any fans of how i met your mother but there's a specific episode called uh subway wars where they're all racing to a bar and they're taking different public transportation and marshall erickson decides to take uh his feet so and then he starts singing this jingle of marshall vs the machine but cracks me up every time so what do machines bring to the table uh in our industry we have terms like edr nbr xdr nbr and my favorite ddr um which i got really good at that game back in high school i'm glad i got last i was hoping that or

i was worried that i'd say that no one would understand what i was talking about um but these products can be considered black box uh black box solutions which means customers can't really mess with the configuration of it but they are fine-tuned machines to capture more malicious activity using algorithms machine learning neural networks and all that other data science i don't really understand now if you're improving your security or if you're starting here and you have no security and you want to start with one of these solutions that's a great start um because it gives you log visibility right it gets you started um you know versus not having anything but know that there are some limitations

like only providing visibility into certain certain hives in the window windows registry that have been determined to be important by the creators of the tool for a specific example is windows run registry keys i think i've seen that in almost every edr that i've experienced with not they are they aren't great tools because the experience of a hat had been positive but i also i also know that not all environments are the same and there really is no easy button in cyber security there's no one-size-fits-all also these tools really report their findings and leave it up to the analyst to determine if the activity is truly malicious a thing that we do with threat hunting

especially a cyborg and that's what makes us shine is that we build a context not only do we create a detection or a hunt package say hey here's the query go look for maliciousness this way but we say here's why it's relevant here's the attacks that we've seen it in and then of course we map it to the miter attack matrix so that you know or have a better idea of what's going on so what about the humans um well we're the crucial part of the puzzle as well we are creative and we all have different perspectives and so meant so we may approach problems in a different way this is a strength that we can see

and it's even driving change within larger organizations through like diversity and inclusion programs because they they understand that the more people you have with diverse backgrounds they may be approaching different problems different ways you might may find the most efficient solution in that pool also we are curious we aren't programmed like computers are in a linear fashion but we like to run experiments we like to test hypothesis and we like to be wrong sometimes in my opinion being wrong or making mistakes is our best teacher i can't tell you how much i've learned in my life from being wrong but i can tell you i've learned very little from being right um lastly the malware and

threats that we are facing are human and the artifacts that that they leave behind can be very human in nature as well this can be seen when attackers leave hidden messages in source code or in the notes that they create to the organizations for example uh this was in a 2016 dub black shades the threat actor group they attacked um the receipt a researcher was taking a look at the source code and they found this this screen or this string and it's meant to be read you cannot crack this algorithm you idiot um computers don't have what we have in the ego wise we are very proud of cocky people especially the top tier talent and and they want to let you know

that you've been compromised by leaving stuff like this i don't i don't think machines will ever do that i mean should malcolm would but we're not there yet so reducing that kids while time uh getting back on track um but what reduces uh dwell time um there are a couple things uh machine learning and our artificial intelligence detections once like i mentioned in the humans versus machines we got static alerts and detentions ransomware gives a shout out and a proactive threat hunting so when it comes to ml and ai detections they can be very powerful because they can alert us to anomalies like file a file meeting a criteria that for being malicious based on how it

is acting or even how it's compiled looking at the different processes it calls and even detect process injection based on processes that are spawned after which is actually kind of hard whenever you deal with your normal log sources like windows event logs and system on finding process injection is rather tricky um they do have the ability to look at a large sets of data and create relationships that may have been missed by us humans were then or us to analyze it using the naked eye a limitation being that they report their findings and then it is up to the humans to take the next step and determine hey this is malicious or it is not malicious for example if you have a

ml ai detection fire that says power shell scene used for the first time is that malicious i don't know what's the context behind it is it uh a new network admin that was just hired and he's performing his duties for the very first time or is it you know someone from hr that has never used powershell in the first place or doesn't even know what it is yeah that would be malicious um so things like that those are the limitations of you know machine learning so when it comes to static alerts and detections these are normally based on activity that have been seen in the wild or reported reported on or based off of ttps and

behaviors they prove beneficial when they've been tailored to the environment that they are living in um remember that not all environments are the same so not all detections are the same either i wouldn't take a a detection built for an energy company and throw it in say banking because they have different you know threats and different threat landscapes so sometimes it doesn't make sense um tuning these to fit in an organization takes time and a lot of it but could play a critical role when it comes to an incident so if you do take the time to weed out all those false positives find the detections that work really well then you can really stop the

threat actors before they even begin and whenever i run the goal the end goal is to take my hunt and that shell query that i've created and to tune it and improve it and improve it until i can say hey i'm no longer hunting for this i'm detecting now ransomware gets the shout out again um like i said uh there's a lot going on we are never safe for it it affects everyone but in the ransomware investigations or research that i've found there are opportunities for detections like i said in those first three days before the payload executes for example uac bypass which is user account control bypass there's a lot of ways to elevate your

privileges using that and you know copying to file shares you know how do they propagate the ransomware and that's normally the easiest so thanks to our number one uh reducement of dwell time uh proactive thread hunting uh so it's easy to say but hard to implement um not a lot or organizations are starting to implement threat hunt teams but it's hard in this industry where everything's go go go go to actually set a time you know start picking people from the stock start setting people from you know intel and say you now have another responsibility now not that we don't do that because i know a lot of people that have wear a lot of

hats but you know sometimes it just doesn't work um the fun thing is though i found a statistic from crowdstrike this year that they said 68 of detections were not malware based and they were um they involved living off the land binaries or programs or executables that already exist whenever you install the operating system for example command prompt is a living off the lan binary because i don't know what windows box doesn't have it

so what does it all mean what am i going to do with all this information and how can i you know how can it help my security posture

all right sweet um so i came up where i found the known unknown matrix online and i was like wow yeah i could probably do something with this um because when i'm talking about um relationships between artifacts and these artifacts existing in an environment it kind of just fell into place um so i'm going to take the things i've been mentioning i'm going to throw them in there for you um so the four quadrants are known known unknown unknown known and unknown unknown and here you can see where i've placed those those items from the threat hunting process such as external reports static detections ifc scanning and proactive threat hunting but i'm going to take time to explain them to you

show you my dark and twisted mind but so when it comes to no knowns this is when i'm talking about you know it's the situation where we know what artifacts are and that they exist for example the intel reports and the threat reports that i read we know the attack happened because they're reporting on it and you know they're involved or they're adding a lot of indicators of compromise to prove you know here this is how we found it also triggered static detections and the reason i want to mention that they're triggered is because the static detections at rest fall into a different quadrant but once that alert triggers you know that the artifacts are going to be there

and you know that there's maliciousness on your network uh the known unknown quadrant this quadrant contains artifacts that we know such as iocs but we're unsure if they exist for example if we said uh said threaten intel report that listed a lot of iocs for it we're going to take you know extract them all from the intel report create a search in our environment the artif or the artifacts are known because we have our ircs but we're searching for it in our environment now once again not the best thing but a critical process of the uh instant response process because you got to find out you know does does this relate to us now don't all

i know i'm guilty of this in my early years but just because iocs don't come back and you haven't had any hits don't say hey we're not compromised by this attack like i said and as i stated earlier they can change them real quick you know a simple comma a different character that'll change the whole hash and you know you want to keep digging and always assuming that you're compromised um the dome unknowns now this this one's a little tricky because it actually might overlap into the unknowns unknowns but this is where i put static detections snort sim rules that have not triggered now the reason for this is because we know that have we built a query that's looking for

maliciousness we know the behaviors that we're looking for we know they're out there we know threat actors perform there but we don't know the artifacts exist in our environment for example what computer is creating that activity which computers compromise what are the command line arguments that they're executing we don't know those things so this was a little tricky but once again these type of detections are important into the for the process and to increase your security posture and the unknown unknowns this is the quadrant that i live in um both the artifacts and behaviors or activity and the existence of either is unknown um this is where proactive threat hunting falls being hypothesis driven we're really

just guessing which ttp or behavior that we will find and we have no clue what the artifacts are as well we're really just kind of taking the dart and throwing it seeing where it sticks and seeing what we find but you know we have a general idea of the output what it will be um and because the tuning if it comes back and it's not expected you know we have to rethink the process re uh create a hypothesis and continue on i'm really tearing through this you're welcome i know we all want to watch hackers so uh your takeaways um so now i've thrown all this information from metrics to matrix i hope you have a better idea of what a

holistic approach to threat hunting and having good cyber security posture from a blue team side of things all the factors that i've mentioned play a role in finding maliciousness they just do it in different ways you know also as good as ml and ai algorithms are we shouldn't just set it and forget it and rely solely on those um once again they're lacking the context these are all more tools to increase our capabilities they're not the final solution to fix everything anything you can be an ioc it doesn't have to come from a certified or well-known company it could be artifacts that you found in your environment so never you know discredit yourself so sophisticated actors will tailor

their attacks per company because just like the defense side of the house no attacks are one size fits all so no detections are one size fits all trust in your humans don't be that company or organization that continues to take on tool after tool after tool um all you're doing is increasing your technical debt and you're probably burning out your analysts even faster because hey now you have another tool to look in hey you have another log source to look for let the humans be the ones to drive the change let them be the ones that say hey this needs fixed we need to ingest this you know because the creativity plays a big role into what we what we're

doing and for the uh climax i know this has been burning and everyone's wondering what the answer is but the solution you might not have expected it's a balance between tools machines and humans if you can leverage their strengths or their relative strengths then you're really succeeding as an organization if you talk in ultimatums you say this tool this tool this tool or this person this person this person what if that tool is no longer supported what if that person finds a better job and leaves i mean you're you're kind of in trouble there um but if you lean on the creativity of the people trust in them train them when needed and then bring in some tools and

say what can you do with this then you'll you'll be successful um thank you all for taking time to listen to this um i really enjoy being a part of this conference especially being my first time especially being a speaker um i do want to thank the organizers of organization buffalo thank you for uh choosing me also to the people that made this happen my uh wonderful marketing department at cyborg security because what i wrote didn't look like this it was a little rougher um also my wife i'd probably get in trouble if i didn't mention her but she sat there and listened to me non-stop repeat this over and over um and also uh to a

colleague and a friend of mine happy birthday chad all right questions

oh and if you want to find out more about cyborg security by all means follow us on our platforms [Applause] scheduling your analytics how often do you find yourself in a position where you have your analytics right but then you don't realize that you have you know you're not collecting the logs required for the analytics to fire off and how how do you go about identifying that in the process so the question was i'm correct if i'm wrong um what do i do when i'm in a situation where everything's right but like emulation hypothesis but the log sources aren't there so luckily i live in a lab environment so i can talk to my security engineer

and say hey you know we need these logs and this comes up a lot especially when it does come to register keys registry keys they're very powerful they hold all the settings in your windows box that you can configure from you know dark theme to light theme to group policies um but there's not a lot of auditing on them because they can be so noisy so whenever it comes time to that i talked to my security engineer if i were in a corporation or a larger organization i would have to build a use case like um stated earlier and say here's my threat i don't have these logs you know and here's how this could add value to our

network now after creating that i hope that the cfo agrees because it's probably going to bump up my license um but that's how i would have focused that process

so the question was what do i do vlog retention um me i'm lucky i don't have to deal with that i'm just my threat hunter um but um once again i live in a lab environment so we can keep larger um larger loads because our lab environment is full of maliciousness and you know us testing i i couldn't i couldn't answer that question because i've never sat in a engineer's position so i apologize anyone else all right thank you very much [Applause]