Human Hunting

awesome so uh we're going to go ahead and start PR uh presenting Shan Gillespie with human hunting so a quick little intro on Sean Sean is just this guy you know Sean's career in infos SEC began as a network Defender for the US Air Force where he later transitioned to an attacker role with an aggressor Squadron after leaving the air force he has spent most of his career developing tools and techniques for intrusion detection for both uh Department of Defense and private companies he moved to the Bay Area as an early member of mandiant Redwood City uh s so focusing on Advanced detection methods and now works at Yahoo working on projects such as grr and for Effective intrusion

detection and response another well-known fact about Shawn is that he has a cat that looks just like a killer

[Applause] whale

um

okay clearly don't touch the cables all right uh my name is Sean gusby U like I said I work at Yahoo uh primarily working on G um go get into what we're talking about uh the first piece is a little bit on General Automation and tools uh a little bit about who the adversary is then getting into preparations to go actually hunting for the adversary and then some real examples with Q&A at the end if there's any time so automation is great um we've done a lot of cool things with automation particularly in Safety and Security spaces uh ID badges where you scan the badge it goes back to a back-end system tells the guard this is

what the badge is supposed to say he validates the badge says that lets the person in um complex Logistics and Battlefield management was a big thing in the Air Force we've got a lot of planes a lot of logistics a lot of people a lot of fuel a lot of weapons all this stuff has to get moved around and so putting all these into automation systems allow Battlefield commanders to know what resources are available and where and where they need to go uh some of the stuff that I worked on previously I actually worked at a hospital for a couple years we did a lot of really interesting stuff with u modernizing a lot of the technology getting a lot of

things online for them one of the really interesting ones was um Wireless carts that have um barcode scanners on them so previously it' take two nurses to go into the room with the patient they'd have to validate um the correct blood or the correct medications that there is the patient and that both nurses have to agree that this is the correct thing to do and they administer whatever blood or medications so the system we built only required one nurse so they able to go in do the first validation the system would then they would scan the patient scan whatever meds or Bloods the uh the system would actually validate this is the correct pairing and then additionally it would

go further with medications and make sure that are is this patient been given anything else that might interact with the drugs that you're about to give them uh so we can do a lot of really good stuff with medication or with um automation um critical factors are really that they're skilled humans at the decision points in these chains so that's great except for when it is isn't uh some examples recently with the Air France Asiana Airlines U these were um incidents where either the the pilots didn't understand what the system was telling them there was a system failure something had happened that caused uh these planes to go down and a lot of it was over Reliance and automation uh the

2003 Northeast blackout was another really good example they actually had people in the field calling in saying there's things going wrong there's things going wrong we have to go deal with this but because the the system wasn't actually raising the alerts that they were expecting The Operators didn't respond they said no everything's fine the system would tell us if there was a problem um really this is a critical one there's f-22s that were flying from Hawaii to Japan and everything was going great clear skies life is good they crossed the International Date Line and all the computers shut down in the air um they actually had to use visual on their tanker to turn around and fly back

to hium so this actually be became a really critical piece in in what I'm talking about here in that the reason that that one didn't turn into a much worse disaster than what it was is that the people in charge of that were um highly trained experts they were able to actually recover and deal with the fact that their automation systems went away and they're able to still control that plane and get back to safety so um with the airline stuff Airbus has taken an approach to automate basically everything and it's the goal is to increase alation to make flying more safe and they've targeted more the pilot is going to be there to serve the

machine where Boeing has a a slightly different history but their engineering angle tends to be more the machines will be there to serve the pilot and so their training um is pretty different for these different types of systems and it fundamentally Alters how pilots and the planes interact um there's a lot of training about levels of automation where if the autopilot isn't doing what you want it to do you need to lower the level of automation so that you're taking more control and if the plane still isn't doing what you want it to do you need to turn off your Automation and fly the plane yourself and that really relies on Experts being at the helm of

these things when the automation starts having problems so there's a lot of talk about how automation is going to solve all our problems we have all these pieces that we can automate and we can make machines do all the work for us and IBM had this big sales pitch for a long while where they're going to build these giant chess playing machines at pretty ridiculous cost and go around the world saying look we can beat the best human chess players in the world and they did really really good with this um the problem is we still can't build machines that play go very well go can um go machines can build or beat human adversaries at

moderate skill levels but expert players thoroughly destroy these machines like they just can't keep up and what winds up happening is that the more complex um the the game is the more variables there are the more tactics there are the less performant the machines are when trying to play against humans um so moral is automation is great for repetitive tasks it's great for exact sorting and calculations um it's great for doing accurate calculations and it handles large data sets very very well humans not so much at this we get tired we get fatigued we make errors automation is bad at replacing highly skilled humans uh humans are very good at pattern recognition we're very good at making

sense of data when when data comes to us we can actually build a story out of what happened and why it happened and how these data points relate to each other much better than machines usually can um we're also very good at changing how we're looking at how how we're categorizing data on the fly so if you're pulling marbles out of a bag and you start sorting them by color and then you pull out a cube you can decide wait I'm no longer sorting sting by color I'm sorting by shape and you start sorting by shape instead and so it's very easy for humans to quickly shift how we're we're viewing data um the other big thing we're really good

at is misdirection particularly in the security world we have a denial of service attack over here all your engineers are responding to that I'm going to go hit your sensitive data while your engineers are panicking about the N service attack the machines are not going to be as aware of that kind of misdirection and bluffing that humans are capable of doing and humans are going to be much more capable of detecting that sort of thing so where does that leave us now we have cyber War now where we have to defend our cyber bases against the Cyber missiles of our cyber enemies so we have to teach our cyber Warriors how to use our cyber

weapons so a little bit on the state of where these cyber weapons are we have uh a heavy vendor focus on these Magic Black Box boxes we want um a vendor solution that replaces all the humans we don't want to have to pay expensive analyst to do a difficult job we want a solution that just takes care of it all for us and this is what's getting sold very few of these systems that actually do do some of the more advanced statistical analysis machine learning these types of things very few of them are actually targeted analysts they're very much targeted as the checkbox and the compliance folks of you built a system it will magically detect and it

will just email you when there's a problem um and it's it's really trying to cut the humans out so we wind up in the same problem with the airline issues is we wind up with an overreliance and automation just like you don't want your pilot to have his first experience flying a plane at 35,000 ft you don't want your analyst to have a first experience of like what security looks like when the guys are running off with your data we wind up chasing magic unicorns so it's not all the vendors uh the community has issues as well we build a lot of really cool tools uh a lot of incident response type stuff is built on open source tools sluth kit

volatility recall these are very very useful tools but they're often very complex and they very have are very light on documentation or if the documentation is there it's very spread out um and these tools rarely scale well if you've got five machines that you're working with that's great if you've got 5,000 machines that you have to look at you're going to be there a while trying to deal with this so who is the adversary it's really easy to lose track of who the advisary is when your day-to-day is dealing with computer systems you're dealing with malware you're dealing with pcap you're dealing with network traffic netf flow all of these other things and it's very

easy to lose sight that behind all of that there's a human attacker um humans are very creative we are very adaptive um the attackers are tend to be very high skilled they're I mean there's script kitties and there's different levels of attacker but compared to the average computer user all of them are pretty highly skilled attackers and they have a reason for doing what they're doing they're very focused on achieving whatever goals they have it may be notoriety it may be your banking credentials it may be you know whatever secret source code you have that's powering your organization but they have goals and and things that they are trying to achieve so to hunt these guys there's a

number of systems that I C oriz into um a handful of different capabilities that you really need to be able to hunt these guys host interrogation is being able to interact with the host pull files pull data that sort of thing host sensors are things that are more like event logging and pulling data back uh Central logging is kind of the all-encompassing like Splunk and things like that where you going to put all this data um Network sensors the standard Fair um bro surcot snort like any type of network sensor stuff and then alert Management systems that are really tying this all together so as these things feed and capture binaries you can push them into Mal

analysis and actually have a number of places where you can actually interact with that data in a meaningful fashion not just here's a bunch of line item log things that I'm going to alert on but here's data that I actually have to manipulate um with the alert Management Systems some of the key components that you need to to have in that capability is severity classification so you know what's important what's not important how to triage things you'll have resolution status so that you can say this is a false positive this was handled this was part of incident X or whatever the case may be and then you need to have notes particularly for false positives so when one analyst

comes along does all the work to say this is a false positive this is why we identified what's happening here the next time that alert shows up the next analyst doesn't have to redo all that work they can actually see what happened last time and make the same determination and move on to bigger and better things but in all of this you we also need the threat research of understanding what to go looking for and how to interpret that data and you have books sandbox analysis threat and tell these types of things above all that you need humans this is a human adversary and you need a human Defender you need humans for alert validation when these

automated systems generate alerts they don't know what's going on they don't have that picture of of really what's happening you need humans to look at the alert and go validate this is what really happened was this a false positive is this something we care about is this something expected in our environment effective response after they validate that alert you need a human to go decide what to do about it there's been no shortage of antivirus tools that decide some super critical dll is actually malicious and it shuts down the entire system uh one of the examples i r to was actually with Mac vepo hips it has this really great feature where if it detects a network

attack it'll block all Communications to an IP for some configurable amount of time what we' ran into when I had arrived is they'd been having this problem with the SharePoint servers and the databases where the SharePoint servers would just go away and no one was really sure why and so after I started getting into the host intrusion stuff I built a bunch of dashboards I actually got it set up so we could do real-time management of our alerts and see what was happening and one day the the call came in again and the guys across the way from me said uh we got the server it's the SharePoint thing again and I looked at my dashboard and

it was the same host name and I said okay guys what's the the database server and then we matched all up I was like well what's happening is there's some 1998 pptp alert that macv default deploys and so every so often the SharePoint server would send the packet that triggers this and for the next 10 minutes the database server would quit talking to the SharePoint server so the user would make the call it' make it through the chain it' eventually get up to the guys that are actually going to have to deal with that server they'd call the user and say okay what's the problem user would check and say oh well nothing's the problem everything's fine

and so when you have these automated systems trying to do all your Containment all your response all of these things all by themselves you wind up in a lot of situations like this where you can damage business critical systems without really understanding why any of that's happening um another big piece of why you need a human is the emerging threats as you cut off Avenues of attack in your environment it's going to take humans to really sit down and think where's the next place that I'm going to get hit the machines are not going to know this is the next thing I need to go look for this is the next thing I need to go protect uh an example

of this was about a year or two ago gu about a year ago now um I'd read a ton of our reports at ment I'd read a bunch of books looking for all these points where I could go looking for bad stuff in various environments and I started noticing well none of the attackers are storing their executable code in the registry and that seems like a really good place where they could move their code if they're trying to stay off the file system in some of the traditional like places that we would look and so I started writing indicators for well if the attackers go to store their executable code in the registry what is

that going to look like and so some months later after I left i' had actually found out that these indicators had actually detected an attacker at one of uh the clients and so the machines are not going to be able to do this sort of work it's going to be humans that really have to sit down and think what are the other ways that I can attack this system how can I predict what another human is going to do next so a little on the collection systems the three categories that I basically break collection systems into two types there stream capture and state capture stream capture is the typical things where it's event logging it's

change it's capturing State changes so as something changes on a system it's generating an event and sending that event somewhere it provides you an excellent timeline of events of process uh launches file wres these types of things and it's great for investigations if something happens you really want that full timeline of everything that's happened and it's really good for signatures and searches when you know what you're looking for and you need to find it in real time you can say this is exactly what I'm looking for and find it very quickly the problem it comes with is it generates very large volumes of data especially in large environments you have to be able to transfer that

data you have to be able to store that data and you have to process that data and particularly at very large environments you're not usually going to be able to buy a vendor box that can do all the things that they promise with this type of data um you're going to have to have substantial investment to have the type of processing to get real- time information out of these sorts of data stores State capture systems are more the forensic style it's capturing a snapshot of what a system looks like at that point in time it gives you a further historical reach into what's happened on that system so unless everyone is reinstalling their entire environment every time they install

carbon black or osquery you need a way to look further back into the past if you were compromised 3 years ago and didn't know it that could still be impacting your business today and you just don't understand understand why and so these types of State capture systems let you look further back in the past to try and understand what may have happened before it's also really good for discovery of new uh threats some of the registry stuff I was talking about being able to take the snapshot of the registry across Fleet to see what is there what isn't there what looks what the Baseline looks like and what deviations look like um it's excellent forensics forensics is typically you

want a snapshot of what the system looked like at the time that you you found the intrusion the big problem that you have with State capture is the gaps in Timeline is that if you have two cap two capture points you have to infer everything that happened in between there's potentially a lot of events that were Transit that you're not going to have that data for between those two captures so for alert management a little back to earlier the the events and alerts that you're getting are going to be unique for the types of systems that they're coming from um Network alert data such as surcot bro those types of alerts don't look like file

systems and Registries and those don't look like log entries so you need to be able to handle these types of data in different ways you really want uis that are tailored for the types of data that you're trying to interact with um they really should be relatively simple the everything in the kitchen syn uis really wind up back in the same problem of do you have a flag do you how many different ways do you click on this thing and it's it eats up a lot of time in analysis being able to get to the answers that you need very quickly you need to be able to search and pivot very quickly when I get response back from a

persistence mechanism looking in the registry and it lists a file I don't want to have to go back through another system to go get that I want to be able to immediately pivot off of that file name into the file system of that host and see that then say okay well the file is still there now I need to Pivot into the process list of that host and see is that process still running then I want to be able to Pivot over to the Mal analysis platform say did we capture this what are we doing about it so being able to Pivot through data really hinges on on what kind of data that is and it's

very difficult to do when it's all just flat log line type stuff um for many of these things particularly with State capture machines if you're going to capture the state of the fleet to compare it you're going to need to do stacking you're going to need to do filtering and you're want to be able to tag the interesting items so that as you are dealing with an incident you can actually tag any light item evidence that the process was running that this file was on the system that this registry entry existed that we see this network alert at the same time you want to able to tag all of that data so that you can hit you know the big report

button and have all of those things actually tied together so you all of you can work from the same sheet and see there's here's all the indicators of the attack here's where other analysts can go take that information and go back into other systems and try and PIV it uh some really good examples that I've worked with in the past or similar systems uh snorby is for dealing with network alerts it's actually a really nice UI they even have like iPhone like iOS capabilities so you can actually do incident response on the Fly and at least get a good idea of what's Happening uh EPO is actually really good it's kind of a configuration Beast

they've got this entire dashboard monitoring system that is actually pretty decent they just don't ship with all the queries and dashboards that you need to make it a really useful real-time vent monitoring system so if if anyone's actually interested in that I've got queries somewhere still that are really nice dashboards for that threat research again this goes back to you need to know what kinds of things to look for you are going to have to go to the books you're going to have to go read CIS internals you're going to have to go read I believe to the apple core it's the MacBook that's kind of similar um you need to understand what this the OS level and what these systems

are doing read the forensics books read the malware books understand what types of things can happen and that's also going to help you predict what the next move is going to be you don't have to understand everything in every book but as you read through you'll get an idea of this is how the system operates and here's potential weak points that may other people may have or haven't identified yet um you need an analysis lab you need to be able to do at least some level of malware analysis static analysis um Dynamic analysis you want to be able to examine the protocols that the various tools and um tactics generate whether it's SMB stuff as

they're doing lateral movement or if it's custom binary protocols that a malare sample does and you want to be able to understand the techniques you want to be able to spin up a domain controller and a bunch of systems with all of the software that you're running and do run past the hash and see what your systems detect do lateral movement with the Max and see what kinds of events that generates and validate that your systems are able to pick them up without trying to do all this in the production Network for threat intelligence you need to read the reports if you're skipping to the end for the md5's you're doing it wrong we we talk about a lot about how

advanced these attackers are and how threatening these attackers are and their techniques and the kinds of things that they're doing and then we give out lists of md5's and say now go find them these guys have recompiled their tools they've changed the binaries trying to find them by md5 is not going to help you which gets into malware databases take those MD fs and go back to virus total and some of these other malware sample places and go get the samples yourself when you read the report look at the samples find the things that that malware is doing learn about what the attackers were actually targeting learn about their behavior as attackers where they're favored working

directories are how they exfiltrate data and understand what the attackers are actually doing when you're looking at lists of ips and domains from some of the thread Intel feeds a lot of that ages out really quick there's a lot of overlap there's very little context but you can start to get a picture of how these attackers use domains what kind of fake domains do they like to use what kind of naming conventions do they stick with and then a big one that uh I actually had a lot of problem with in the aggressor world is the red versus blue practice there's a lot of exercises where we say okay we're going to go red team this network and the red team comes

in kicks the door down steals the crown jewels goes home writes a report and says this is how we kicked your ass this doesn't really help anyone uh what really should be happening for this to be an effective cycle is the red team needs to be going through step by step and ensuring okay we got a foothold did you see us no this is why you didn't see us this is how you need to change your environment and change your tools to catch us now do you see us okay great now we're moving on and the blue team needs to understand that they can't just squash them and say we won the game because they need to know well okay you

made it past the first step we're going to have to ignore that now and let you do the next step so that we can learn how to detect the next phase of the attack and so red versus blue practice the idea that we should hide what's happening from The Blue Team while we're doing it really doesn't make any sense because it doesn't actually make the Defenders any better they don't get to interact with the red team as the attacks are happening and get a solid understanding of how to detect those attacks and in a lot of cases the red team if they haven't had a lot of Defender side practice they don't understand what footprint their tools

are really generating they don't understand some of the systems that might be in play and how they could be getting caught to improve their tactics as a real attacker would so the big project that I've been working on with Yahoo for uh almost the last year now is working on G it's a open- source uh rapid response um tool for dealing with live response data it's not meant to be the Dead the typical deadbox forensics of there's been a compromise I'm going to take a machine and I'm going to go do analysis on it instead it's meant to say there's a compromise that machine's online I can go start interrogating it and start pivoting to other machines without

tipping off the attacker that I'm on to them and it lets me get much better into on what the attacker is doing and actually respond and be able to remediate all at once to kick them all out of the network at the same time uh it's all open source there it runs on a number of Open Source projects it uses recall for the memory forensics component he uses SLO kit for dis forensics and there's actually a forensics artifacts component that's similar to various ioc stuff it's a way to define things that you want ger to go check so a little more details on G it's like I said it's a state capture system uh it's state machine based

so the there's a component of flows and we'll get into a little bit more in a little bit the the flows actually are the unit of work for G in terms of like go capture a file go list processes but because it's state machine operated it's not strictly go do this task and finish it's go do this task and based on the result do this next task and so you can actually chain together flows to get much more complex usage out of it it's crossplatform the agent runs on Mac uh Linux and windows it's very scriptable it's all open source it's python um most of the parsing of data happens on the server side the downside is there's

limited client side filtering uh some of this is stuff I'm working on for a couple capabilities but the idea is that you want a really lightweight agent particularly in heavy developer environments um the Google folks started this project and we're in a similar boat at Yahoo with developers everywhere uh if you start taking over their machines and start doing too much work they just start ripping things off so you want a lightweight agent that doesn't disturb the user so like I said uh flows are the unit of work there's a concept called hunts so rather than interacting with just one host at a time a hunt will let you run a flow against the entire fleet

you can give it different criteria such as how fast you want it to run across the fleet uh how many clients total do you want to run um looking for labels you can label your clients to say only run against these uh clients uh and then there's artifacts which are used with a particular flow called the artifact collector and they're a yaml defined way to define points of interest on a host say registry run Keys uh the location of KRON tab files the location of etsy password and you can start building out all of these artifacts so instead of an incident responder needing to remember every possible thing that they need to go check you can actually have them

predefined in the system so you can run the flow that says I need to go collect all of these artifacts and then pivot and do more artifact collection on different systems so the common flows more interesting ones there's an interrogate flow that's Gathering system information a hardware Mac address IP users uh there's an artifact collector flow which I talked about filefinder is just searching it's searching the fleet or searching a a host for a file which you can then run in a hunt to search the fleet registry finder same kind of thing analyze client memories for running uh recall plugins on hosts uh memory collector will actually let you capture full memory captures over the network

net stat list processes uh launch binaries pretty self-explanatory so to actually go hunting uh some of the things that I like to do with this is create an artifact that's going to collect all your live response data so you can chain artifacts together so one artifact can Define that it actually needs to go collect these other five artifacts to be complete itself so you can define an artifact that says these five things for Windows these 10 things for Mac these seven things for Linux and then use that one artifact to run a a hunt against an LR label so you have this hunt with a flow running artifact collector for your live response that's only going to run

on clients that you've labeled LR so then create hunts for um actually hunting what you're looking for persistence mechanism strange looking processes things like that and then when you find a host that looks like it might be compromised or you want to do further validation on all you have to do is apply the LR label the next time the host checks in the hunt will see that it has that label and it will go do all of its live response collection for you when it actually completes that it can actually email you and tell you I have all the data that you need and an analyst can then come back and start doing analysis after the

system has collected everything so the types of things that you want to hunt uh processes there's a lot of interesting things here there's bad parent processes if service host is running under Explorer or IE like these are not where this normally belongs um smss always has a a pit of four or eight so if that Pro if you see that process without the correct PID that's an interesting uh thing that you need to go check out unique processes if the process only exists on a few hosts in the entired fleet it's worth checking out it might be an edge case or it might be something malicious open connections is another really good one if you see

Elsas connecting to the internet you probably have a problem so these are things that you can do with G is sell I'm going to go search the fleet looking for any elsass process that has a connection to the internet registry is full of persistence mechanisms there's too many to list uh authentication manipulation there's a lot of different things you can do you can do authentication bypass with debugger Keys you can do uh hooking with the some of the LSS key or LSA keys so anytime you change a password a dll gets loaded that records your password and sends that to the attacker uh services and Damons look for wrong configuration it's pointing at the wrong binary

pointing at the wrong uh service dll It's Got U strange options that aren't normally configured uh look for Unique Services services that only show up on a few hosts Opus skated files uh if you see C.D in s system 32 but file magic reports that as a raar file there's probably something going on there you need to go check out uh suspicious file placement C program.exe uh Windows helpfully takes away the need for escaping spaces but that also means that c program.exe will get executed on boot uh scheduled tasks and cron jobs especially in Windows scheduled tasks are a really good lateral movement trick um cron jobs really good persistence these are all areas you can go looking

especially with a state collection system go capture the state of your Fleet and then go do stacking so think about your threat research think about what you see think about what the attackers are doing think about what the attackers are doing to try and evade your systems and think about what evidence they're going to be leaving behind when they do that where they're going to put files if they're doing time stomping you might have PE files that have a compile time that are later than their written to disk time so these types of things will let you go go find where attackers might be that other systems maybe didn't pick up and then try and generalize when you see the

specific attacks in these reports don't just search for the file name don't just search for the md5 you need to think about how they're actually doing it if you see multiple reports where the attackers use win SSH or win SSL and you can infer their naming convention you can start looking for their naming conventions instead of just I'm going to blindly search for a bunch of file names so use bolt collection where possible if you're doing launch agents or launch stons you can pull everything back pretty easily pist files aren't that big uh G actually understands how to dup that so it'll only download a file from the first host that sees it on for every

other host that it needs to download that file it will check to see if it already has it if it already has that file it won't go download it again it'll just set the pointer in the database to say here's another system that has a copy of this file filtered collection we impracticable if if you're trying to do looking for time stomping like uh with on PE headers it's probably not going to make sense in most environments to download every PE file across the entire environment to go analyze that uh so getting some filtering in place on the the G client will be a good thing that to expand the types of things you can

search for reliably um use artifacts with parsers were possible I actually wrote parsers for the launch Damons and launch agents so instead of just collecting the pist files and putting them in the ger database for you to go look at it actually collects the files puts them in the database but then parses out the interesting bits like what is what program is it actually running so when you're looking at your results from G you're not just getting a list of these are all the files I collected you're actually getting a list of these are all the programs that are being executed by these files um the API there's a HTTP API for pulling out data in Json format and then

there's a g console that's basically an interactive python that loads all the G Library so you can actually interact with the internals of G and the ACT actual objects as they are in the database um this goes back to you want to inter you want to use systems that understand the data that you're working with so that you can easily pivot you don't want to just flatten it all the time because if if you're dealing with a directory structure you can actually walk that directory structure rather than just having a list of files then when you bring this data back stacking and filtering comes into play if you are dealing with the more flattened structure you can start to

stack components of that um how many of these registry keys are referencing the same binary how many of these launch D are referencing the same binary and then if it doesn't stack well you can start doing filtering to quickly filter out these are known false postive these are known things that we use in our environment where's the interesting stuff so we're going to go through a couple quick examples these are things that I did while doing a lot of development and testing with G rolling it out uh these were not strictly dedicated hunting efforts uh a lot of the examples data I'm going to show is kind of cut and pasted because the tools that I were was using to do this were

very raw command line messy stuff uh it was mostly just to verify that the system was actually up and running and working so we're going to go through some common file placement attacks we're going to go through uh some run key stuff and we're going to go through OSX launch stons so this is actually what an artifact looks like uh it has a name a document string it defines a collector type so it could be file registry command and it can have arguments in this case a file collector type has a path list so you can actually go through and Define all the paths that you want to collect you can see you can define environment

System Route so you don't have to know it doesn't have to be absolute paths and it also has the ability to do users home directory users Sid things like that so here's some example results hold this back did quick stacking um if you haven't seen the set HC util man those are uh easy authentication bypass tricks for file placement you can see there's a whole bunch that are exactly the same size so those are probably fine that makes sense for a fleet of Windows hosts as you get into the lower numbers it gets interesting there need to go validate why do I only have two of this particular version um chased all those down it was just odds and ends servers

um patch levels things like that uh nothing particularly interesting ocid dll is actually pretty interesting there's a well-known persistence mechanism if you drop oci dll in system 32 when the msdtc service starts Microsoft is hey put the registry keys in saying oh you're probably an oracle customer so we're going to go ahead and load this uh dll when we start if it's not there the dll fails to load no big deal if the dll is there it loads it up as system level Privileges and so I actually tracked this one down and it was the first time I actually saw the Oracle dll installed in system 32 this was the legitimate uh Real Deal um and

it kind of comes back to developers developers do weird stuff um the Crypt based one however is a privilege escalation trick and what I found or a search or a hijacking trick what the file actually was was a UAC bypass it was a well-known code I ran strings on it found some obvious strings and found the actual code for the UAC bypass in GitHub um not a whole lot came out of that again developer tricks getting irritated with systems this is they go download GitHub they compile things and they do stuff on their hosts so uh for the Run key example some of the format has actually changed they the G artifacts repository just changed

some of the formatting for how you define these but it's basically the same you're still defining a path to a run key um so we get results like this every user has the same thing in their local directory this this is not super useful stacking is not going to work well on a key like this now you can do some data manipulation you can carve off the user directory and that sort of stuff but again just quick looking at the results this doesn't help much so instead I decided to get some rexes out that I've dealt with in the past um this is a Rex looking for registry keys that are referencing ads strings so if you see a

run key or something like that referencing an ads string there's probably something uh not right poison ivy did this a lot uh when it was still easy to execute out of ads so in this case because some of the execute out of ads stuff has been disabled in later versions of Windows there's still ways around it in this case they're using red server 32 to load this dll and so I went to go validate these two hosts obviously run SVC host is pretty sketchy run key uh and it turns out a had already gotten these things but they were two Bitcoin miners similar trick here I'm just going to grip the results came back with uh NJ

rat Houdini ninja worm whatever you want to call it variations of that again it's pretty obviously not okay stuff but all you have to do is really simple GP search through your results and it all falls out and so for Mac uh this is one of the examples I had with a parser where instead of just grabbing the files I was actually pulling the files back and parsing the content of those files and came back with a similar problem where if I stacked it I had a bunch of oneoff systems but I noticed in the oneoff systems that there was a com. apple and in a fleet of Macs there shouldn't be one Mac that has a com.

apple P list there's something wrong with this picture and pointing at user bin Global update does not seem like a thing that apple is going to be doing so I pulled all these files back and the global update file was missing the watch sh file was uh actually there and it tied those two files together so those two pists were obviously related and then I pulled the Mac hook file back threw it in VT or through the md5 in VT and it was wire lurker so I was actually able to pick this up before any of the other systems were able to using G to actually go actively hunting for the attackers so coming soon I'm working on

a lot more parsers for things like at jobs uh Chron jobs stuff like that to make it easier to get data out quicker uh better searching for file type doing file magic and things like that so I can say show me all the zip files and show me all the raar files in system 32 uh better filtering for artifacts and flows the idea is to have at least some level of client level filtering so that in searches that we're going to be bringing too much data back we can filter it down a little bit on the client side before we bring it back for processing uh and then the big one is the the better tools

for working with G data um outside of uh Google there's not a lot of tools for effectively dealing with a lot of the data that comes out of G um particularly with some of the past experience of dealing with similar systems I want to build a lot of the similar UI to actually be able to quickly analyze the type of data that's coming out of G to be able to tag it to be able to move around in it to be able to stack it provide comments provide history when you see the same thing in the next month when you run your hunting scripts you already know that you've already evaluated what that was um if you already have some of this

set up you've already got a platform to do these things to get a little bit further with gur you can actually start reading your process execution logs your AV logs because it is so scriptable when you see these events happen you can actually pass these events on back to G to go schedule LR capture to go capture that binary and feed it to your malware analysis platform and to do all these other automation bits uh another possibility is taking your network alerts as you get network alerts or interesting Network flows you can actually have it go back to G and say well I need to go find the host that I think is associated with this and then

pass that into a flow and say this is the IP that it was connecting to do the process list when it finds a process that was connected to that IP you can have the flow take the next step of go capture that process take that process put it into our malware analysis platform then schedule like live response collection to for full invest netive data and then go capture that by the time all this said and done when a network alert is happened emails start flying by the time the analyst gets to the system to deal with it there all the data they need should already be there it should have been able to capture the

binary capture live response data and give them a really good starting point so they're not wasting time getting further into the investigation so here at the end uh what we need is we need more automation that enhances cup human capabilities rather than trying to replace them um we need more focus on hunting and alert validation skills actually being able to respond to this stuff in a timely fashion um we need higher quality threat data and sharing giant lists of ips and hashes and domains are not super useful we need more information about what the attackers are doing how they're doing it what their preferred methods are and things that we can actually turn into actionable intelligence and we need to

stop looking for magic unicorns um another big one again with G for me is that we have an industry that's built a a huge business process on things like Apache on Linux all of these open source tools and I think it's time that as an industry we really need to start building highquality professional uh open source tools that are at the level of Apache that are the go-to answer for the industry because we have the capability to build these tools and make this better without having to pay protection money so any

questions so for dealing with hashes the the hashing piece with G happens on file acquisition when you actually tell it to acquire a file it'll do the hash and send the hash back to determine if it already has a copy of it uh at that stage it's up to really the analyst to figure out how do you want to build your process do you want to check all those against virus total do you want to check them against nsrl do you automatically feed them into your malware analysis platform and see what comes out um so it's more based on that it's not based on searching for

hashes so the way the way I would do that just like yeah sorry uh to find to find the Trojan sshds across your Fleet um just the quick thinking on stage the way I'd probably handle that is I would just tell it go collect sshd from every system so it's only going to collect the file from the first one it finds so it's only going to have one copy so if you have a large Fleet you're not going to be doing a lot of file transfer because it's only going to have to do it the first time when you get the results back you can stack it quickly say you know X th000 of my fleet have this hash this

percentage only has this hash and then you can start doing analysis on the the few and far between

numbers so we to get in a little bit how yaho used it started spinning it up and we got I'm sorry uh the the question was how we do the analysis uh once data's back um the short version is no we use the actual G either the the HTTP API or the ger console um in our case it's mostly the HTTP API because the HTTP part part um flows through the access control mechanisms that the admin UI uses whereas the G console piece is using just the authentication like if I'm on this the a server that has console all I have to do is read the config file to get the passwords to the database and so because we're doing a

lot of auditing we have that flow through the HTTP piece so that we can actually do the user level auditing of who's using who's pulling what data with ger console right now it's mostly for us like as the admins of the system to go in for maintenance or dealing with problems because it bypasses that that user level Au and we're just authenticating directly to the database um hopefully that answers what you're after okay we actually there's in the latest release of G there's a MySQL Advanced Data store the previous MySQL data store did not scale very well and so you see some nods um we we got to a couple thousand clients on the thing caught on fire and

then it was much scrambling to start restructuring the database redoing some of the tables and we currently have it running uh on stable and run 15,000 hosts um and there's no signs that we can't go further than that so we're probably going to start testing further than that soon to see how far we can push it before we hit another world block with our data store any other questions all right thank

you