Jacqueline Stokes, Danny Akacki and Stephen Hinick - Hunting Defense Against The Dark Arts

you're good talks in it besides it doesn't go have fun he did it wounds on their necks or you know last hour I heard that was absolutely packed it was about as crowded in here for the for the competing talk these have been it's really been a good day so thank you everyone i get for coming out they really appreciate it I really have a little bit of fun of announcing this next room so when they submitted we've ever seen a teen life without an equal right for Eastside Augusta we did rcmp like quite a while ago and the four of us all work together we all worked roughly around the same product line at

mania par I and at that point it was you know there's going to be pretty cool Erie to put in this talk and it was all me something all four of us four or five of us and we all for now work reported for companies but Jackie is still in is and I made it guys I can hear Gloria Gaynor I will survive anyway Jackie is mine evil on Twitter so no monthly call her and then we've got also seen any steven is now at Oracle and we have any nikakih who is a former Indian as well it starts to that Bank of America two days from now on their hunting so really good to hear about that so I give you

these three guys to talk about hunting and final cool hi everybody as he said I'm Jackie Stokes own principal consultant / immediate been with many for about three years part of that I spend most of my life and hot dusty dangerous places as to his stuff for the Defense Department Steve I Steven Inca with Oracle now I've done the pedal whole range of blue team stuff soxer running them building them involved in on doing it response check offs you know deploying firewalls and stuff like that previously ideal for that big nasty uh technically bankamerica Monday morning so nice to work with as Phil settles guys maybe a complete run teams so stodgy capital on everything as well so I'm doing it for a

while for that general purpose I team Slattery so just never sought to oh yeah primarily one is kind of thing like red blood and do the same thing that's a nice so excited for it super gravity here if you gotta do cashgate with other just go check out doors awesome all right so uh we talk about hunting or actually let's back up a little bit we talked about security operations and we talked about the problem set that hunting could potentially solve right we're looking at finding evil we're looking at ways for evil to do evil things which are risky potentially stupid things that are happening in the environment things your users are doing ways systems are configure the way the

architecture is laid out that create wider attack surfaces that you would really want and we want to use the data that we already have data that we can get and the intent is to make things better I want to improve things it's not just hey guys we have a lot of bad things and we have a lot of incidents as a result want to feed this information back into security operations that improve the security program overall so that's what we're here today to talk to you about so our solution to that right honey hydras on the edge crack and threaten honey news is really using the data we have to to look for ways that we

can improve our posture whether that's kicking bad guys out or preventing them from getting in through different means and we want to use use what we find as Jackie mentioned to bring it back into the sock so that we can continue to improve both on the sock and as the on the security program is all yeah so I guess the main point i hunger is hunting is a process climatology your tools will notably fall down he be able to translate your process an internet value and your have the thinking to any cool but you might have at your disposal so it is not tools were you here say as body all the time decisive actually

risking a beta render to the Jackie we are fully nothing so use what works for you it's not no worse you're alert is telling you something happen that's really right so in the context of a solid contact commenting in general we think of it as it's not already it's actively going out there to kind of find ways or equal doing things and the endgame is the endgame location right so if you're going to try to find new honey mythology come on you don't have to do it over and over again the Indian should be our nation but as it's not about having a machine telling you something from it is about the human is about

using early experiences to have people so he's talking about building a i guess i should use a Michael we talked about building hub program there are different foundational aspects of a hub program or of a security program that you need to look at before you start thinking about building a hot program on top of that right so the very first thing however looking at here is a formalized security program right somebody cares about security within your organization someone is providing some executive sponsorship there you have some type of budget some type of resourcing available to tackle this problem within the organization right most of you here hopefully have this in place the next thing we want to talk about is a

functioning security operations capability and a functioning instead of response capability right you've got to be able to you've got to be able to handle the incidents that you have they're coming in you've got to be able to detect these incidents you've got to be able to feed those back into security operations and kind of IT across the enterprise to reiterate that information and make it useful to the rest of the organization to drive a risk down that's what we're trying to do here right is drive a risk down and your daddy mentioned technology and tools but the way we look at it at Mandy in any way is kind of a trifecta right tools technology process and people

right these are three distinct things that come together join together to build a functioning security program so tools and technology you need to have them but they're not the end-all be-all and they're based on the people and the processes that you have right and use cases and playbooks a lot of people don't have these this is getting up into the detection maturity right see that arrow trending upward there must be this tall to ride right you've got to have a security program going to have security operations capability it's the response capability you've got to have the technology and tools that support those functions and once you start thinking about how to mature your processes you

start working on detection use cases and response playbooks how do you detect the problems the issues the risks the vulnerabilities how do you how do you bring in the data sources that are required to make these detections and how do you respond to these in a methodical manner that incorporates every business unit or individual within your organization that has some kind of relationship to the incident response process at the very top of this pyramid we're looking at a hot program I said a lot of things that need to be in place hopefully optimally for you to begin hunting now if you have analysts bench time and you want to start hunting for fun in your spare time that's a great

way to start dipping your toes in and getting some experience they're getting some exposure to hunting so um I draw hands who's heard or read him yonko very very good looking out right in your hands and you shouldn't I think he's the guy oh so he likes these work wasn't it he never asked were also going to using a lot of you we stand on the shoulders of giants Danny yeah so I'm a lot of time is because I can get mired in words but to give you kind of an overall normal idea hunting spree new right well for some people do for a lot of a long time just not working out a little bit so to kind

of illustrate this you know David wrote this on a scale from level zero three four of what kind of organizations are which level and might be great on core athlon star level zero you're pretty much is putting our fire all right you have a 90 s babii signature bass it's okay start but not really prepared to do any kind of immature wanting level one which is minimal um now you're like okay we might have apparently had a deist you are so tuned now listen Italian and maybe start actually collecting some of our own dad which is suitors who are important up there l three levels 2 & 3 now you're talking critical processes those procedures you're not worried

about your tools as much because you get really good at them and then your data collection goes way way up all the way up so for which is your kind of leading the charge there and there's not only companies that are really at that level or maturity like they're the ones everybody else kind of looking you talking to some of the guys could do this for a while one of the good examples is the G search the guys been doing this work barrier in a long time if you have a chance to meet anybody's the GS or pick their brains and plasma shoot movies so that's kind of the injury model that you look at that

alright cool so now we want to do this right so how are we going to build this right basically we're going to go back through that pyramid first thing executive sponsorships your management needs to know what you're doing what you're going to find what it means when you find it because they want to know they want a heads up that you're going to find something bad and you're going to tell them about it and they may have to take that public for compliance reasons for I you know disclosure reasons what have you so they need to know you're doing it and support you next you need to have a blogging strategy you need to have those

logs coming into the right spot they need to be normalized arsed d dubed all that stuff right it's very important that you're getting stuff to the right place so that you can actually use it all right saint-saëns what i just said ryce Algar eyes it take it in one place so that you can actually start digging through that data if it's all over the place you're going to have a bad time make it available within one spot make it searchable right it's just an assist log server sitting on disk and all you got is crap again you're going to have a bad time you got to be able to do something more powerful than that faster

than that and then lastly you gotta drive maturity right you've got some stuff you've got tools you can search you can hunt but now you need to start document you need to start making these use cases to start taking these hunts feeding them off to becoming alerts through automation so develop use cases right check your data are you getting the right stuff do you have a long do you need to find what you think you shouldn't be able to find if you're trying to look for for suspicious domains and you're not getting DNS traffic right there's a great spot you need to get your DNS logs for figure to link and associated requirements you know you're going to get you're going to

start you're going to go cool security onion here we go I got it on an endpoint security I ain't got another one well okay Scott is working great for us well we've expanded now we're a team of you know 15 people security onion working for us anymore I don't know it might be you might need a tuning you might need a tool he might need some more enterprise level you know not as much of the open source but it's going to depend on you your program your team what you're comfortable with what your capabilities are and then lastly to harp on it again is it's very important what you find needs to go back into the rest of your

program if you find you know malicious actors how they get in how can you prevent them getting again if you if you find someone doing something silly that they should be doing then you know how can you stop that from happening in the future how can you help you prevent it is a policy visit education of you know how can you improve the posture of your overall organization okay this is another good good illustration of the overview of what your mission is with on it so a lot is going to be kind of narsil sila right and I depend on what you're doing you can just be in a little apricot or groom and never talk to other departments and

that sucks and it's never going to get anywhere just kind of illustrates about what that angle is from going from your on going on missions feeding that into your IRA so it might have enough to get already but if you find something that maybe isn't it on it's important to build a bill that road with them and of course that out the IR output effects second trait that affects your saw your guys with their Island laughs right they have to benefit from the fruit of your Labor's from I are on them and then this really is all about lessons learned I don't know if you eat yourself right now nobody wants to do it nobody wants to

write it but it's super important and you don't want to have to make say the sex twice so let's learn after-action reports or super super important and then build that back into your overall cashgate village right so they can go back and find new or older stuff but that all of those have to feed into each other so there are a couple arrows here thank you Dan you a couple arrows here right yeah yeah evil uvula non-evil right so we can get into this and a little bit there are two ways to look at the hunt output right evil and waste for evil to do evil things so things that negatively affect our security posture

and that widen our attack surface so when we talk about the output of the hunt when it comes to evil we pass that's an incident response right these are the guys that need to handle the issue triage and go through that process and a non evil risk something that is just stupid or risky needs to go into security operations so they can handle that vulnerability in the in the way that makes sense for the organization and so on that note right there's three things you're going to find in the hunt you're going to find nothing that's not bad right if you phrase your hunt correctly and you have the right bosses you don't find anything that means that whatever

you are hunting for is not there right that that's a good thing now eventually you may need to review this because you might have the repond right that might not have the right data or maybe it's not relevant to you your environment right if you have no sequel in your environment and looking for sequel injection you're just not going to find it right it's not relevant to what you got you're going to find non-malicious the stupid the ways we're able to do evil things right this is good right you can take this you can apply new policies and procedures in order to mitigate that risk or you're going to find malicious stuff you're going to feed that to your

eye our team that can remediate it find those lessons learned bring them back in again to try and prevent it from happening again excellent so when we talk about sorting out data already talked about data sources data sources are critical to the task of hunting if you don't have the right data you might as well not hunt so this is a kind of a set of examples of different types of data that we can integrate through our hunt missions and you'll notice a lot of pretty typical traditional things by one of our tour VPN connections and we want to monitor DNS and want to monitor threat intelligence whether that comes from inside the organization or it was

developed internally internally or externally right you can have information feed in from multiple places from outside the organization that are going to be really useful the place where I don't see organizations focusing from a threat Intel perspective is within the organization right there are people performing reconnaissance against your enterprise this is useful information there are people talking about you on IRC that's useful information right your organization is targeted it behooves you to understand how you're being targeted and what the objectives of these threat actors are but their motivations are and how you can integrate that information in ear hunt operations same goes for HR information we don't see this a lot right guy gets fired guy or gal gets

fired right now they may have shared their credentials of someone else they may be coming back into the network to steal some intellectual property get some stuff off of their machine or the case may be if you're not tied into the HR termination workflow from a side perspective you may not know about this you may not be able to detect this very well so there are a lot of different sources of information that you can use to generate some really useful detection xand you want to think very broadly about this information and how it can work for you in your environment I want to talk about events there are two types of events there are observed events

these are events that came directly from a system that process the data that you want to look at right you have a router that passed a connection right it generated a log that is a observable events observed event came directly from the source he have things like synthetic events these were generated through some type of analytic process that was automated if you have a let's say you have a sandbox at your egress point your sandbox popped and Lauren said hey we analyze this data and there's something that you should look at you can look at it as an alert or you can look as a synthetic event and frankly guys there isn't any kind of standardization across

the industry right now for a lot of these hunt terms and so these are suggestions for how to look at these some folks in the back over here would say that an event is an alert is not a synthetic event but it could be looked at that way depending on what the source is well you're seeing here up on the screen that's an Apache log right that log on the very top and these screenshots are from the Parker analytics platform if any of you ever heard of it excellent hot platform don't want a pork product but it is really cool guys product agnostic I should say that because it is true if you're not product agnostic you need to

be it's important up here I you see the circle right Patchi HTTP server that long originated from the server that process that client connection and so that that came from the server and so it's an observed event that's the betting event down here this is an analytic log or in analytic events a synthetic event basically stating that the ratio of sac data to receive via for a particular set of events didn't see right so it's saying maybe some kind of big ass filtration of hurting here to Denali something that concerned about so that's a synthetic event that's what we mean here just want to do some definition of terms so that we can understand we're talking about here so

how we have a prologue up here on the screen if you don't know bro network security monitor you got to know it's a great great open source tool we want to use original source data whenever possible right just like we said an observed event something that came directly from the system that processed that that data so this is metadata coming from the system that process the original data we want to make sure that the right fields are there so you see all these fields are split up right the client variables we have a connection ID that allows us to pick a few different data sources not specific to the fair analytics platform but some of the other

pivot points here are just standard pivot points right for an HDTV perspective you need to know what a SAS code is for a particular session or connection and you're going to want to potentially stack those and see whether maybe you happen to have a high incidence of 40 fours right maybe someones trying to do something squirrelly and they're triggering a bunch of 404 s that's useful information but you wouldn't know unless you are able to pick that piece of information out of the event right and use it as you see fit observed events are better than synthetic events unless you're using synthetic events for context alright synthetic events are great for context so when you use a hot platform you have

varying capabilities in terms of enrichment you may be able to tag an event you may taga said of ip's you may be able to add contacts to specific events I mean it really varies in terms of the technology and tools that are out there but the point is is you generally want to rely on the information that came directly from the source close to the source as possible from from a data perspective again you've got to be able to pick those extra those extractable data points out that that makes sense for you and that are important for you ready the spells so the balls sure yeah so there's really three main points that on your

own you're depending on your organization even as formal way up to the large one like your back person oh it's uh opinion network you have to really focus your energies which is not everybody you have so much data coming in and the other data is not a solitary go home oh crap doesn't it plus you have to recognize your glasses and also think think more think more broadly right just don't think about your own interactive think about how what your constant going to do got a benefit you know I our compliance audit the tutoring words or messy operations and also you're going to find you might find a lot of parsing parsing issues do look at that you

really have to ferret that stuff out we came to the crap all day right that's not what this without so those are those are always two two and two rich and contextual enter data because it's all about context I never want to see somebody coming mean like hey I found this journey I theaters right halton yeah why flashing blog why the dirty oh because no she's a dog even true story i had another organization come to me they had a enthalpy and they said just block everything i am saying ok why ok this is though it's not this no it's not how this works so this is that's kind of how your computer so then

the right you talking about contact right we want to enrich this data we want to rinse rinse it is coming in right maybe we can tag this do we know we have our management we know what our star database servers are what our web servers are do we have a set of fees Friday Intel fees are they coming you know external internal things that we're building or that we've paid for this pathetic volatile me oh there we go you know whatever information do we have that we can feed into the system right can we figure out where these peas are right somebody talking to an IV in California and in Kuwait this might be interesting it may not be or you know

maybe they're coming from California and toys maybe more interesting but putting context to this information that's coming in allows you to make additional informed decisions and having it there in your platform is very very helpful to be able especially to say you know I've got this guy he's hitting this type here is over and over and over again instead having to go into yourself and say who is IP address if it already will tell you you know with some level of confidence what the who is report is for the IP that's really really helpful to know immediately oh it's you know CloudFlare site for amazon and it's just refreshing a page is not a big deal so

tools right we've talked a lot about data we mentioned tools tools next all right thank you that is in my country like Danny just go ahead and this with Google's do so we didn't talk y0 not really done so what are I mean to our minds this is probably the three most important things you don't want to be late for search to the run all day anybody familiar promoting that weird obstacle alright cool so you know sometimes you have a higher time students to get breakfast awesome dinner going depending on how could they your search headed functioning forwarders means that with every day so ideally very officer to come back is is amazing we've gotten

spoiled by being an annular because the surgeon really went very very quickly but I know the struggle of what happened be slow stacking which is really just fancy word for casting right you want to say I don't want to see what this thing did how many times a bid in who is it worth giving being able to move laterally through on that right so it's not enough just for me to have possible I want to hopefully I have fireball grace I get into that ideally in pony lodge now closed down once I yes so you can form a larger nature and he seems that earlier about nothing is still something right it don't find something

if you don't go right logs words this is exactly why hunting is important so and then as you mature being able to get a great intel lot of me stuff we're already talking about that time so this is these are the keys to really building a function operative TQ sweet dance moves too so obviously tools tools cost money maybe right there's a lot of good open source stuff out there you gotta you gotta drive these by operational requirements well as I mentioned before one you and your team need because it's going to be different for every organization you have different stuff coming in you have different needs different requirements and so it's very important that when you go through a

selection process for your tool whether it's something you're paying more from a vendor or your open source with a you know elasticsearch or Elsa or you know whatever the case may be that then you take into account what you actually ate what's going to work for you so we put a list here these is absolutely not all-inclusive there's some great stuff out there if it's open source it's free it will scale very well you just need to decide and work with it correctly and so it's something to take into account when you are when you're doing this going through this process all right we could talk about analysis we talked about the threat hunting loop this is another

wonderful graphic from the team over at squirrel David Bianco he's going to hate me for this but he's a thought leader in the hunting space he used to work over with us at mandiant on the threat analytics platform team and came up with a lot of really good stuff with us and then took a lot of that went over to squirrel stole it just kidding so we start with hypotheses right things that we assume about our environment right for example we have a policy to not plug USB drives into critical servers well how do we bear that out how do we make sure that we aren't actually plugging USB drives into critical servers right we form a hypothesis and then we seek to

prove that either wrong or right once we develop these hypotheses whether they're very broads broadly scoped or more narrowly scoped which you'll you'll end up trending that way over time you'll start with these really broad hypotheses and then start narrowing the bat and we'll talk about what those look like here in a minute start investigating using your tools you start developing new automations for these and then you weren't those back into the to your platform to your hunt methodology to security operations overall this is a wall of text we are not going to go over all of this this is for you to take home we'll put up the slides after we get done here today and hopefully you'll be

able to get some ideas from this list that will help you develop a base set of hypotheses about your environment and start thinking about what data sources you need to bring in to detect those and what tools you can use to detect and investigate those so so well known on the hypotheses this is kind of lie as kind of well when I first started doing hunting I like to phrase a question do I see suspicious outbound data transfers all right and it's a good way to think about it I like I've changed a little bit i think this is mostly a functionary stuff but young though is I like to phrase them as a statement and i want to say

the statement is we're doing something right right we don't plug USBs into critical servers we you know pick one of these off the list right we don't have inbound traffic from our DMZ to our internal network and I want to prove ideally I want to prove it right right i don't want to find that stuff chances aren't probably going to prove that wrong but i want to start with a statement of i'm going to find that we're doing this right we're making the right decisions we're doing the right things so that's some way that's how I've started to phrase those its kind of to you but I just want to make a note on them so those Dhabi and so now we're

going to talk a bit about about some of the stuff that we found doing this at a whole different country different companies and you want to kind of introduce you sure yeah um so if you're greener safer inside already ways for evil New York a person actual holy call I heart actually evil and this is was this is file my favorite kind of falls should I are people say oh my god you came to your guys help desk don't worry about it you know because exact nineties and a more engineering and this is there cuz we see this all the time our TV is tommy was all gone okay so and excellence this actually is to the other

fun story that we have of during our hunting expeditions own so one of the biggest very nice dish is a day slumber right shooting patch management / / management we went to a client one time and said hey look at all the stuff you have your job all you have a team-building rings of fire box straight face he's on a business to run he kind of said it like that though he was very he was very like emotional about you know we're showing him like you know Firefox going back to like version one across the whole business like just totally crazy steps like flash java Firefox super older explorer from like you know 10 15 years ago we're like

how could this be this is like a humongous company we've all heard of them and they're supposed to be doing the right thing and as you were conversation that you're going to have with your negative real a lot of times they all know we go and you have to try to get it across to them that this is a guest drama on App Engine Java big deal going a little later across the entire enterprise that so you have to really try and talk to them whether in next slide so each of these examples we're going to give this is all stuff we found real live environments different companies whether it be our own others consulted with what have you they're

going to have a couple different pieces they're gonna have hypotheses rights same phrasing what we found and then we put some remediation stuff in there that's kind of a war you know read at home but wanted to try and have an opportunity to put some information there about you know how we would recommend you know fixing some of these things right just as a to try and give some more information that everybody can take home and learn from so got it sure so this is one of the cat roles that will always sit down and we're for so um and three essence of it we can talk all day the process and mythology occupation but the invaders the different keyboard

on all right but now helmet look look cool and the end game is to be able to fit any Alice down through there some experience inside here's my mantra cool stuff you can look for our TV remote access always always the crap later so the hypothesis is as Tina said earlier all of our ultimate access is you can approve beans and that's him being an optimist so it can be anything from your find vse to the production network which is awesome Arnie peas or domain controllers from the DMZ evidence of logmein gotomypc yeah Tom's at it again and then from there you have to liberalize recommendation so you don't have to you don't have to go over the

regulations because we're going to be short on time but you'll notice here guys that we didn't put a disposition in here right in terms of like evil or a way for an evil to do an evil thing right you guys know your environment you guys know what's acceptable in your environment you guys know what the risk profile hopefully more or less of your organization what it looks like and so you know what you can risk accept and you know what you simply can't risk accepted you know what enclaves are more important than others or what Dad is more important than other and what users are more important than others and so you'll you'll want to make your own

determinations of course as to what the disposition is on these but we left it pretty vague on these slides data storage so the hypothesis is that we're only storing data in places where it should live I see smiles yeah yeah it's pretty crazy right but this is a very broad hypothesis again we're worse we're having a very broad start point and then we start to funnel those into kind of some more detailed thoughts about specific detections that we can enact in our environment so some of the things that we found we found a really sensitive corporate data on USB sticks and we're infected so users coming over to the machine he's plugging it in McAfee is blocking it we don't care

because it got blocked so nobody's looking at that event and their sensitive data on there and they comes back the next day and he plugs it in again and McAfee blocks it again and it's like the same thing over and over again oh no one cares about this because McAfee said it was blocked well now the guy goes over to another machine that somebody filled with cell avion and now you have a problem right so you gotta pay attention to stuff like that cloud providers right if you just run a list of the top 10 or 20 cloud providers out on the internet and run that through your proxy data or just straight network you're going to be really upset there

are a lot of people uploading a lot of weird stuff that you can't quite identify to places where you don't want and to organizations that you don't have an SLA with and code repositories of something that we see a lot developers they kind of operate with our mentality they need to do things the way that they want to do them and they store data where they want to store it so we do end up finding that a lot and that is a risky proposition for an organization that depends on intellectual property to survive all right proxy interfere shirt hypothesis our proxies are configured correctly right I get very broad you you'll narrow these down right because

because something much more interesting is going to be our proxy don't allow categorize malicious traffic private that's probably a little bit more narrowly scope probably a little more realistic but all in the same light right how to client we went to him and said why are you allowing all this malicious known malicious traffic what do you mean we paid millions of dollars for you know software you know firewall vendor X and it's supposed to be blocking and doing gateway antivirus hi you might want to go check him out because I can tell you they're not right this is being allowed like you know and that was one nice thing about in this case putting putting a different sensor

in right we put bro in because their firewall logs came back and said you know it said alert but they just took that to mean well it was blocked either that or they weren't looking at him which is more likely a freely easy way of identifying something like that your proxy logs it's just to look at the ratio of like denied to allows yeah that was that was how he originally found it is is it was like a one-percent chance or one percent denied traffic have a firewall you know out of a web filter like that's not going to happen in any user environment ever they're like but we brought a consultant in from that

company from that vendor to help us implement this and they spent hundreds of thousands of dollars on proxy infrastructure that was wasted for a full 12 months as money that could have gone back into the security program right so some other things right not blocking stuff you station whether it's executables or or you know whatever the policy says or maybe the things we found is not logging enough information right there's one big major firewall vendor that doesn't log user agent strings on HTTP traffic right user agent is incredibly important to identify malicious activity in HTTP traffic but it doesn't log it period does not log it I have sent requests than many times and sorry I don't have

good news report okay go back so that that last one there that last discovery bullet the proxy is not logging the necessary metadata this is important not just from a hunt perspective but from a forensics perspective if you have to investigate an incident six months later and it was something really bad and you really need to know what happened if you're not logging the right field you might as well not even log and save the data because it's going to be useless to you when it comes down to the to the critical times shorter make that point all hurdle or autumn is here as review um so what have we found telnet FTP plane test place xpress go across to you

know SNMP version to IRC yes only but a goodie still happening and this could be something as innocuous and just kind of an air for being consultants we don't always get full visibility we don't know what all the processes armies have you best guesses when I see IRC in 2016 and over network why it could just be some security nerd I just want great winter games where is he or it can be an RC channeling you connect two images a 2030 boss talking to each other so that's kind of risky tour you guys see it all the time you see movie downloads gigs encasement movies downloaded you know that's not really supposed to happen and

so yeah that's one of the the muses proof clients right internet access achieved knowing using no good software right this is question of patch management all right we've seen suspicious stuff that you're like this is related to know browser or you know app or anything like that I've ever heard of or you know as we mentioned before you're going back Firefox versions you know 10 years old that you should have no place on an enterprise network now sometimes this stuff's on guest wireless and you kind of kind of how about you're stuck but a lot of times it won't be but hunting you know is a good way you're going to find this not malicious stuff but it's a good way

to validate other things right you you now validated hey patch management may not be working as well as maybe we thought it was right and so it's a good way again feedback into the program to improve that to take back to your your Ops teams to say hey look I thought we were crashing firefox you know everyone should be up to you know whatever 35 or whatever the latest one is outside of the traditional audit cycle right which always comes back with the check in the box magic it's magic exactly those boxes always get checked I don't know how so privileged management I put usernames up here but it should probably read accounts right you want to make sure

that our accounts are stored the right way are our standardized the right way that there's a taxonomy that makes us for our organization so that we can pick out the things that don't make sense all right if everyone in your organization is first initial last name for the user accounts and you see like you know I don't know give me an example elite guy 56 like you know that's not a legitimate user name right there there are different implications here right when you start noticing user names that are kind of outside aspect there are a lot of different locations right if you have service accounts that are shared by different users or used by non automated

processes right that's what a service accounts for it's not for a human to you so you want to make sure that's not happening you want make sure the service accounts were only used for one type of service and that you're not reusing the service accounts across infrastructure you've got to have some type of specificity to your account taxonomy so regardless of whether you identify people by a number and say RS which is the department 546 19 and that's a specific user and you know how to track that back in your privilege management or you simply use a more standard first out last or first initial last name you've got to be able to tie accounts

back to individual people you've got to be able to tie them back to individual processes or services and so that's really important you can identify that during hunts if different groups within your organization business units are not complying with the organizational standard this is really good information to pass back to the security operations when I make sure that folks are using the proper privilege escalation techniques and that they're not logging directly into machines as root right now the best practice kid could you know make things easier for a threat actor and it all of this kind of like stacking usernames in a threat hunting platform or just some type of data analytics process allows you to pick out the

things that don't make sense in that that seem a little weird that you want to dig into a little bit more and then that last bullet there going back to that HR termination workflow I love using that example because we had a client who was able to take their their provisioning process and take all the users that were terminated from the network and feed that right back into their hunting platform on a regular basis and a relationship with IR and they were able to find some insider threat activity that would have been disastrous to the survivability of the business so I try to drive that home there are so many different data points that you get integrate to hunting and

kind of come up with some kind of a fusion model that make sense for your organization our laws are going to tell us about movies know every time to dolly our edges so yeah we have non security specific Liza's a lot of times any event called Fitness is only only audience up that I think is supposed to be doing security and out there right so we have you know even stuff like this is going as a scan detection you know disabled yes it's very very noisy but should it be disabled you know to to level or it's not going to give you what you need to know and probably the biggest one that I want to get here and we've seen it but

sometimes is having your sensors in the wrong places I've seen so many times but yes we know that the site they wanted to the resource that they got the e boulders there at the vons game dished only the toxin I can't trace it back ok they actual reform now ok so let's try to go get it through some other things so that's that one of the most important ones we're going to get one more as get the last punch to the essence of time process execution right this one's great if you can get your host logs system on snare windows logging pride hessle notations in points only xq process is required for business functions all

right obviously this is really tough to do on a user environment in a server environment much more simple because things shouldn't change all that much right we found all these cated powershell not generally legitimate activity all right mimi cats all right one time we went in and we're like on a hunch like gaming cats keyword searches why did this hit like this shouldn't even be showing up in logs right we could went on a joke the background on that was the organization that hired us to perform this hunt assessment also had a pen test running at the same time and the pen testers had far exceeded their scope and its old Mimi castle a controller oops so I was I was

interesting you know suspicious file names you get users installing random crap where you know the level to which you can do this depends on your level of access controls on your users if they have admin rights on your box you know don't even try under using environments servers but still good one I don't really want to skip over DNS there was a pretty good talk earlier from from when I guys from the NSA I talked about a lot of really cool stuff that you find there so I'd go pick up his slides he had some cool stuff that was like you know like I need to go find out it check out some of the stuff we

can find a lot of good stuff 93 DNS logs sure so thinking ahead what are our goals what was once your justification to your management resulting do matters as to why you want to spend cycles spitting up a on program you want to be able to part your environment you want to run further or all detects rahman irr obviously you want to find stuff that you didn't see before I'm actually just going to get down to metrics because it's one of the files in portable especially for monday because it's really difficult to go to your boss and say and find anything awesome right not good theater like but nothing is still something another important one is the

better you get at this those instance are going to go up initially yes that's something you have to explain to them as like we're getting really do this we have some more isn't it doesn't sound right at first blush but they're going to go up before they go down and it's a good day as well as giving other metrics like how much time did it take us to bond and stuff cool and then we talked about kind of maturing hype methodology and what that's going to look like in the future today we rely on system one thinking alright this is really intuitive stuff this is an analyst who happens to have a network engineering background and notice is that something

is off about a particular set of packets hey there's been some kind of header or manipulation here that I'm not comfortable with let me dig into that right or you might have an admin that notices something former admin that knows there's something on a host that that seems quite a nautilus for this individual based on their experience right for the most part we are in system one thinking today most organizations that are building a hot program or have some type of operational hunt program don't necessarily have very formalized methodology for how they go about doing this right this isn't quantitative this is a empirical there are systematic process is applied to how we go about

doing this we simply look at the data we manipulate the data in a that makes it more clear and easy to understand as a human and then we just apply our experience a judgment and reasoning to it you know through the through the lens of our own experiences and then we show that with our team and say hey do you see what I'm seeing here is this kind of weird we follow that that down a rabbit hole we follow the amelie down the rabbit hole until we can either determine it is a false positive or something evil or a way for able to do an evil thing right to mature this methodology and get to a point where

we're doing this in a much more systematic a repeatable way that allows us to scale teams and do this with a lot more people than we are doing it with today we need to get to system to thinking and this terminology is borrowed from you know Central Intelligence Agency analytic tradecraft right this is the way they go about thinking about how to apply analytic techniques to intelligence data we can apply this type of methodology to hunting as well we want to investigate the ways that make the most sense for us that are repeatable that are consistent and that allow us to continue with both system when thinking and system to thinking right they feed into each other

so some to thinking helps eliminate some of the bias that we all of us here we all know our environments right we're hopefully right to a certain extent we know our environments we have biases about our environments right if we enact a policy that says we don't plug USB into critical servers we might be under this belief that we really don't do this and so it's kind of a blind spot right when you apply more structured analytic techniques to this problem you develop the ability to reduce bias and to eliminate the risk or reduce the risk that is that that comes through bias analysis from a more intuitive standpoint so any questions that's actually the end of our presentation

hopefully this has been useful for you guys I'm actually first of all can I have a show of hands for anyone in the audience that hunts today or has hunted in a way that you know kind of matches up with what we talked about here today awesome awesome awesome awesome and the second question was yeah so how many of you feel like you got something new from this presentation that you didn't know before so here's E accent go down it feels really good there was somebody bandra 42 it was like all right so really cool questions get cool sweat so who's got cold questions yeah so you're starting out yes when you're when you're working on validating

your data sources right so say you're building your enterprise long strategy and you think about say you start with critical systems right that's how you're doing your face plan for making sure that you're getting over daddy you're supposed to get you are going to want to perform some type of validation on that data and so you have a starter set of use cases right why are you bringing in this data how do I validate that I'm receiving the right information and so you would go in with that type of bias to say I'm looking for something specific I want to be sure that the data set includes the right information to perform that detection work does that

answer your question questions

dependent on analyst experience depending on how close the hunt operation is to security operations and high integrated it is within security operations right these organizations have a close relationship and you know the analysts are more experienced or less experienced they know they can lean on each other yeah different people all be hiring I think that I think that when you start getting up into the into higher levels of maturity right Barney built a hug program and you have quit resourcing from a personal perspective you can't start specializing right when you have a smaller security shop everybody wears every half the bigger your group gets their large in your organization the more hosts or end points if you have to be concerned with

or critical data the more you can specialize and start splitting those out at mandya we absolutely have those specializations we have folks that are very network oriented I'm very network oriented very much event log oriented and we have folks that are in the host all the time so what you what you want to do when you have that type of specialization is that you never want to silo those people you want them to work side by side and you also want to have them validating each other's findings because they see things from a completely different perspective all right cool I think we're done thank you guys so much for sticking around really appreciate it