BSidesAugusta 2014 Tim Crothers - Techniques for Fast Windows Investigations

well good afternoon everybody did everybody enjoy lunch is everybody having a good time at b-sides augusta awesome good stuff glad to hear it uh i wanted to very quickly say thank you again to all of our sponsors uh first and foremost is georgia ridge university whole college of business for the beautiful facilities and everything else they provided for us today so around applause we're very appreciative of how they work with us and um for all of our other sponsors mandiant edts sands trusted sex sophos security solutions um startup augusta rsi southward tenable alpha networks google com technology hack five cinegrass no starch press real technology fund fail and oh by the way uh i want to say a special thank you to mr

adrian crenshaw who is like the world's best infosec videographer

we said this year we want to have the world's best videos we have to have this guy whatever it takes so we're very thankful to have him here today i'm very privileged to introduce our next speaker he is truly an expert when it comes to advanced persistent threat he has dealt with apt for a long long time and he is truly an expert when it comes to incident response so please join me in welcoming mr tim cruthers well thank you for those uh those kind words so um they gave me the enviable after lunch session so my task today is going to be keeping you awake so i do walk around a little bit i'm warning about that right

now uh and they gave me two things to give away so um you know i gotta come up with something to give those away one of those is uh richard bailey's uh signed book so those are are not easy to come by so it's a pretty good treat so thanks for joining me everybody this afternoon just a real quick on on who i am i'm director for ed mandiant um i've been in infosec 20 years actually this year got into 94 which basically meant you were the firewall guys all right back in 94 and i've had the privilege of uh working for uh working in the space quite quite some time uh and seeing it come along and along

the way worked with some really really talented folks so uh a lot of what i'm going to talk about is more my observations the fun part of my job is my team is customer facing so we work with customers who bought mandiant's products and so these are security teams that pretty much uh who's who of companies out there uh and i've been with mandy in just about three years now and uh so i've had the privilege to to work with some some folks that are very very skilled uh at what they do and some folks that maybe aren't quite so skilled at what they do right um because what we do is hard and i i also want to uh give

props to those of you who who are here in the blue team room uh because you know red team gets all the you know people thinks it's sexy but from my perspective having spent long time on both sides uh so at that fortune 5 company uh part of what i had to do for i reported the richard baitlich there part of what i had to do was run the red theme there uh and so red team is definitely very near and dear to me my heart but the thing is blue teams the hard work little fool can break stuff i'm just saying right the hard stuff is defending and and of course that's why i'm mostly working in

the incident response space because at a certain point the security is going to fail the reason it fails isn't necessarily because of your efforts in defending that organization but because at the heart of it what we're dealing with is people right i don't care how good our technical measures are i don't care how good our education is etcetera you know that fortune 5 company they had a fantastic uh user education awareness every six months every employee got fished so the first time they ran this so this is an environment with right about 600 000 employees so pretty big place right so the first time they send out this fish you know and if you use or clicked on it

they go to this site and says hey you've been fished and here's what you should be looking for etcetera it was like 64 percent of people clicked on it i mean it was just stop seeing right every six months though it steadily improved got to 86 and then just locked you could not budget from that 86 so 14 we're talking 14 percent of 600 000 people so that's a fairly big number right and um so of course we dug into it what the world is going on why in the world are so many people clicking on this well about seven percent of them were new employees who hadn't been through it all right that makes sense all right now

they've been educated the other seven percent just didn't give a name that's really what it was right you could send them anything they would click on anything and so inherently at the end of the day no matter how good your defenses are at some point they are going to break down and fail okay and so that's where incident response comes in and there have been some really great talks hopefully you guys have been catching uh chris's talk at the beginning with with the uh biases and analysis and stuff great great stuff um mike i enjoyed mike's uh uh talk quite a little bit and in specific uh or specifically rather mike talked about nsl network security monitoring and

network security monitoring is great but if you stop at nsm you're screwed okay because at the end of the day from the network layer we can only infer things okay so like let's take mike's example and this is not in any way please don't take this as criticizing mike because mike had a great example of the difference between like just ids and nsm so we take nsm and we've got a lot of you know we got flow data we got packet data we've got so uh mike gave a really quick just example story where we see you know some malware on the wire we pull it out of the caps we run it through cuckoo or sandbox of choice

right obviously we would recommend using the firewise sandboxes but uh so run it through your sandbox of choice and sandboxes are great but at the end of the day a sandbox can't do everything for you right um a sandbox can only tell you what it observes a piece of software doing so if when i ran that say that piece of malware through the sandbox it reached out to site x to pull down the second stage right so we can load that up and finish pulling that box if the sandbox succeeded at it or maybe it detected it was in the sandbox it may have stopped there so we've got no further intel to go on to

know that oh but when it ran on the user station it got that 404 error because it tried to reach out so then it went to a second site that we didn't catch in our monitoring so we think that it failed and we've got no problem when indeed we've got a problem so at the end of the day what we're talking about with with uh you know i guess the focus of my talk i guess is what i'm trying to say here is how do we layer in host activity to that nsm because nsm is fantastic if you don't have nsm if you're not doing that that full combination then you have no idea what's going on frankly

i'll just people on um but if we stop at nsm and we don't layer in some hosts we're still missing pieces of the puzzle right so back to chris's talk where he's talking about perception versus reality the only way we can determine reality is by having as much of the story and as much of the picture as possible so if we don't have those other pieces then we're trying to make an analysis based upon limited information so if we stop at just ids and we don't have full-on sound we got essentially nothing to work off of with nsm we're much much closer we're probably 80 percent of the way there and probably 80 percent of the time that

might be completely enough well what about that other twenty percent right and so why we wanna layer in i wanna talk about for just a second one is it's much more decisive at the end of the day if i see a 404 i can make an assumption that that malware failed right but i don't know unless i go check the host okay the other thing that's important is severity if i don't know what it actually did on the host again i can infer some of this stuff right so i can pull that malware off run it through my sandbox that gives me a list of what it's possibly doing but back doors are notoriously bad about

that right because a sandbox can only tell you what it observes if a back door instantiates the sandbox has got no way of recording it being given directions by the bad guy at the other end right so without that big piece of it we're just again making a guess at severity right um ultimately i think resources is a big play of this so i've been in this you know so i've been i t well over 30 years now and and mostly in management for the last several years so i haven't completely let my tech skills atrophy thankfully but uh the uh um the resources right our industry as a whole we as practitioners have got to start

thinking about a big picture right part of the reason you know might give some great suggestions for instance in his talk is how do we justify the expenses for getting sensors to instrument our network why in the world in today's day and age with everything that's going on the media do we have to justify that well the reason is because we're doing a really crappy job of explaining to the business appropriately what it is we're doing and part of that's metrics right which also follows in here one of the things that that fortune five so grady summers was the cso there and grading has these series of metrics he calls drain cover and and it really boils into a couple

key metrics uh one of which is what we call dwell time so dwell time is the point at which the bad guy penetrated our security until when we detected them okay as mike said we find the meantime in 243 days between the now to be fair that's the upper end of bad guys right that's not the the commodity stuff here we're talking about but 243 times 243 days mean dwell time so that means they were in our environment doing whatever the heck they wanted for better part of a year and i can tell you i've been on hundreds of investigations where that number is four years or more and we couldn't tell any farther back because the computers involved had

all been recycled in that in that intervening period right and so dwell time the reason i'm such a fan is it's a great metric for us to be able to explain to management why the investment and what we do day in day out is so worthwhile if i can go to my boss or or the cio or the ceo and say look in 2013 we took dwell time from 243 days to 36 hours is that not a massive easily you know to get your head around that means from 243 days the bad guys are running amok to 36 hours before we detect and get them out but we can't get those if we don't measure and we can only do that at the

host layer if we stop at the network layer we're going to be missing that piece and finally ultimately i think this all boils into maturity right maturity of us as individual practitioners as well as maturity of our programs if if we don't have the ability to to quantify what it is that we're doing through metrics through things like that then ultimately i think we're just doomed to failure we're doomed to be just considered a cost center uh that's a necessary evil right and let's be real how many of us that's that's where we're at so that's why i'm such a big proponent of post forensic so what i want to talk about though is how do we

do that in a way that doesn't break the bank okay because it doesn't need to be that hard so when i was putting this talk together i just popped out pulled the latest start rules right and i'm looking at you know this just a snippet of the start rules right because ultimately it's whether we're using snort or a commercial product or whatever it all boils down to we've got a bunch of rules that we're using to look for malicious activity okay so if i look at those i realize oh let me back up just one more thing i should make sure and clarify what i'm talking about today is a very specialized version of sniper

forensics so if you're not familiar with what sniper forensics is google it please when you leave if you're not doing sniper forensics you're killing yourself for nothing okay what i'm talking about here today is we're dialing in sniper forensics to be more precise okay specifically in the contents context of i'm running a stock and i've got these event sources coming at me which of course are going to be more than just signatures to be sure but i've got these sources coming at me i'm running nsm i'm evaluating them how do i fold that that host layer so if i really boil those signatures down i realized they fall into one or two categories either it's something that's telling me

that an exploit was attempted or it's something telling me that i have an active compromise all those detections that we use boil down to those two things somebody is attempting to bro pop me or somebody has popped me well if we step back and look at it at that higher level broader context then our job actually becomes easier right because all i need to determine in the context of my sock situation then at that host is did it work if it's an exploit or does it exist if it's something that's an active compromise so one of the if you go read the the the materials around sniper forensics so the idea with sniper forensics for

those who may not be aware is we go out and cherry-pick data instead of disk imaging pulling full memory which is great if you're in law enforcement and i was in law enforcement for several years um you know and if i've got to stand up and testify in a court of law i absolutely want a full disk image and a full memory capture right because somebody's going to be potentially sitting behind bars for a long time based upon the conclusions of my analysis but right that's not what we're doing in the sock that's not what we're doing in the sock so with sniper for instance we go out and cherry pick and the key thing to being successful i would argue

with all forensics but especially sniper forensics is asking the right questions and so within the context of us doing quick host investigations we can boil that down to two questions did it work if it's an exploit or does it exist if it's something that's active i can sense so far still i've seen a couple knots okay and if people got questions throughout questions i have my questions i love questions because otherwise i'll have to start picking on people so you don't want that so then let's start at the host layer what do we got to work with and specifically i'm really interested in what can i work with that i can get quickly okay because again i've ideally

what i'm doing is is some people have dubbed enterprise security monitoring right i'm taking that nsm model and i'm just adding a host layer to it right where i'm now looking at that host and going okay did that exploit work does that piece of bad that i'm saying is active in my environment really exists there okay so we've got url history that's an easy one right pulls really quick from house and that'll tell us oh yep they went to that download because no matter how good we monitor and or i'm sorry how good we instrument our networks there's always gaps right that that large environment one of the things we did was one of the devs on our team he

wrote this cool thing that ran every host once a month would have to do this check-in process and uh called health check and so we were able to insert into that so all the end points it wasn't all of course but a substantial portion of the endpoint ran this little tool that just reached out to the internet so and then shot that back to a central db so we could figure out how are all of our internal hosts getting out to the internet so of course we could go out send mike out to instrument them right which is pretty important but of course there were thousands of ways right people are bypassing our proxies they're doing all sorts of things to get

in and out of ground market i'm sure you guys are familiar with the problems with our soft perimeters right so url history at the host layer is going to likely be more definitive to us than at the network layer in a lot of cases all right services what services are running on the host trivial to pull that running processes persistence items right what are the things that are persistent on the host prefetch is a gold mine and really quickly pulled levels sadly largely going away now with ssd drives coming in network connections events registry keys and files all of those things can be snagged from hosts very very quickly and can tell us answer those two questions almost always almost

definitively and generally in five minutes or less okay so then we need a way to go about doing that so how do we go about getting at that data on the host of course there are commercial products to do that um but i'm not a sales guy i'm an engineer so it really boils down to one or two things or ideally a combination of the two we can either pre-record the activity on the host so capture it in some way as is happening on the host so that we can just go look at it or we can go and gather it after the fact right classic forensics types of approach the ideal is a combination of the two

and the cool thing is there's a couple really really strong tools not open source but both free that will do this very very effectively for us so let's start with recording so i've got a couple vms here so this is just where i dig in so if you haven't been keeping up with cis internals shame on you cis mon uh racinovich has been just really doing some cool stuff with the system terminals things cis mon now has the ability to run as a service on our host very very small footprint in our endpoint and what i do in in these situations so i install it i typically go for md5 obviously md5 is not nearly as strong a hash as x1 and

shot 256 but everybody's got piles of md5s that i can look up at so it's just a little more convenient from that perspective notice here there's a couple really cool options uh n dash n log network connections every time an executable fires sysmon will just put a simple little event log entry into the host you know into the event logs on the host with what application process id and all the socket information for us okay the other thing that's fantastic about it here is the uh dash c or h if you modify it so you we specify an algorithm so i usually use like i said md5 every process that runs as the process executes

sysmon just takes a quick md5 of it stamps it in the event log this process ran with this process id oh and here were the command line parameters for it something that's basically not accessible in anywhere else except in 2012 r2 okay and because this all goes into the event log it's trivial to pull it back right we can use a map log uh sorry windows event collector stuff to do it back if we want and we don't have to pull it back real time we can pull it back on demand right so we can use uh event log we can use something like uh nx log i mean there's just a bazillion options for getting those events off of

that host or we can even use the next thing but before i go to the next thing let me just show you what i mean so i installed it earlier and this vm so i've got to do is go out here to the application and service logs microsoft windows and i've got this nifty little system on eventually when i scroll to it there it is so i go out to operational logs for syslog sysmon sorry and of course we've got logging we can do all of the filtering et cetera notice the kind of rich detail we get about this right so here's a process create here's the utc time here's the good for the process

logged on user all the rich detail that we need in order to determine what in the heck happened on that host it's uh you know free solution we just have to install it so there's one option then the other option of course is to go out and gather it so this is another free tool redline for mania company i work for again if you haven't uh if you're not familiar with it shame on you you're just not keeping up like you need to be so let me pull up red line and what i actually want to show you in redline is not the analysis portion but redline is your friend if you want a cheap host forensics gathering tool as

well yeah do a little bit of work on your part but you guys are up to that right so i'm going to create create a comprehensive collector and the screen resolution is a little problematic here let me see if i can get that too there we go because all i want to do is go up here what you may not know about is this edit your script so what you want to do is go out here to edit your script and then do show advanced parameters so what this is building redline calls it a script it's really an xml file okay it's an xml file that tells the m agent that that redline is going to package up

here for you what data you want from that post so look at what we can gather so under memory we can get process listing we can restrict it to particular ones we've got all these options so anything we check off and we don't have remotely enough time to show you what all those mean and do but go out download it's free tools spend a little bit of time with it uh it will be time well spent okay um so we've got memory we've got disc tons of stuff from disk system network other now you really probably don't want to actually gather all that data okay because remember i said we wanted to do this fast

so what we want to do in reality is uncheck a lot of this stuff right i don't really need to verify the digital signatures and all of that i'm not really worried about that i'll go ahead and pull the persistence but i'm going uncheck most of these options so i'm just telling the script pull all of the persistence mechanisms pull me a list of services but just lists because 99 times out of 100 that's all i need to do my work same thing over here reports and this i'll go ahead and pull all of the network notice browser history that'll be the file download and the url history for the four primary browsers chrome internet exploder

um firefox and uh safari sorry i can't resist i've got some buddies that work for the security team over at microsoft and uh they actually call it that there too but you didn't hear me say that oh shoot we're broadcasting anyway key system information that's really useful but also quick uh i wouldn't pull the registry as a full registry event logs uh we can potentially pull but in this case i can actually tell it just pull that sysmound event log for instance if i've got sysmon running on that host and so we just go out we select all of our different options that we want and then we just give it a directory here and then we say okay

and what that builds for us then minimize this is these different directories so notice i've got this redline standard collector so what i did in in uh so i've helped you know a number of organizations that are you know smaller organizations that uh nonprofits stuff like that so what we did was we built a self-extracting and sfx rar of this stuff right and in the raw parameters we told it okay when you run go ahead and just execute whoops there it is run redline on audit we ps exact that with admin privileges across to the host and that'll take you a couple minutes to run then you ps exactly the data files back simple as that

easily be scripted and automated uh to run quite quickly you do need some credentials right to do it there's both uh 64-bit and 32-bit you know versions of the the agent that it's running to gather that looks like a champ and then what we end up with is something like pull one of these open here time

i thought i hit it right there ah there it is i can get the right view sorry computers were so picky that way

i can't quite get to the button

it's right there below you can probably adjust your resolution up ah yeah that's a good idea

i should that might help wouldn't it let's see if we can do a little higher resolution

hey there we go all right helps if i can click on the button there so i open my analysis here and we don't care about that stuff let me open up my bar here over the side notice what i've got that's being cut off let me slide it over a little bit notice what i've got over here i've got a timeline i've got the processes the system information this was a quick one it took a minute and a half for me to pull it if i go out here and click on this timeline let me re-maximize this a little more screen real estate collapse some of these down literally what i've got is a timeline of

all of the activity that i pulled from this house so end to end it's going to take me about 10 minutes to get this from a host okay ah faster if i you know pre-build some of the of the mechanisms for doing that then where did my presentation go on there so what we do

you see there we go so as i said essa ah again if you haven't been keeping up with ps exact there's an option nowadays in ps exact where it will automatically set the flag for the um you know administrator query so you can auto answer the yes it's okay to escalate this to administrator you still have to run it with administrative privileges but the user won't get any kind of a pop-up box saying hey i need to do this and this is all lr stuff so you can run all of this even if i checked all of those options off in redline you can run all of those on that host while the user is running the host

they won't in 99 of time notice that they all right keep on working and that mouse isn't going to be lagging none of that kind of stuff but obviously verify that in your environment okay so in terms of determination then it's it becomes very very straightforward for us okay so in the case of exploit alerts just start with the event time so if my event you know in my you know whatever interface be it squeal be a dark side be it whatever take that event time start there what was going on in that host at that point in time and you're going to see did those processes get created what network activity was occurring all of that will be right there on a

silver platter for you so you can make a fun determination then on whether the exploit fired or not or whether it was just an attack in the case of presence alerts just look for those particulars so if if i've got backdoor x that's supposedly running uh look at the particulars for backdoor x what ports is it using what's the process name you know those sorts of things uh that it runs on go look for those on the host right again if we don't have that information just fall back the time because at whatever point you're alert then if that really is going on then there will be indications of it happening in the host layer

now to be clear this is still not a panacea that solves every single one of our problems right uh root kits so on and so forth although i will tell you the modules that the agent for redline is using to gather processes is walking memory it's not pulling just api very hard to hide from jamie butler the gentleman that that runs our r d group is the b and h b gary which i i don't think too many people would argue that jamie's the king of rupees uh and then we just pivot right we pivot off of that information pivot is just a another way of saying we do analysis right we look at what that's

going on oh that spawned this process okay i go look at that process oh it created those registry entries okay go look at those registry entries right we just pivot out from that data right it's very straightforward uh if we gather the right data that's the key is gathering that right data and luckily with sysmon and redline uh which like i say are our free tools it's it's pretty easy to come to accomplish that so in the end what we end up with is a pretty straightforward way of adding the host layer to nsl right nsm is absolutely essential i do not in any shape or form trying to uh reduce the the you know the strength of what nsm is to

us we've got to have that as a foundation but without adding that host layer in as well then we're still at the end of the day just inferring what happened ultimately which pretty conclusive inferring sometime to be clear but uh you know mike's example i i is uh so spot on you know from security teams it really behooves us to if we're putting out tickets that host such and such is compromise right and they run that av and it's not it just makes us look you know fail sauce right if instead we open up that ticket we say hey it's running it's this process it's on these ports it's installed in this directory how does how does that make us look in

comparison as uh you know contributors to the organization right so questions those questions sir so when you talk about reducing dwell time you're obviously not looking for exploits at that point in time you can assume that your own that's right and you now have the problem of going out and interrogating your host to determine whether or not i've got malware running that i didn't detect before yeah so dwell time after the fact you're gonna have to have some indicator of presence right so you know back to the you know to the earlier right the um that's ultimately going to be almost there an active present a active compromise right a presence fire of some sort so

you know whether it's beacon i know chris he said not to use the gourd beacon but you can still can be found the uh you know whatever that presence indicator is yeah so then is it practical to use something like redline and a collector to do a comprehensive sweep of an enclave or a network that's a little trickier i mean you're gonna have to script it at that point right um but you know if you know python powershell some of those pretty trivial to add a little bit of scripting layer on top of that i mean that's where the commercial products really come into their play of course right is that to take some of that pain uh and effort out

of that but yeah it absolutely can be done um you know that fortune five company that he has a manual lr script that is run manually right so uh it absolutely can be done you have a question sir yeah it sounds like you're the right person maybe to ask this um the question became with redline yeah once they were bought with fire right what's the commitment then to keep it not a free yeah sure it is it is uh commitment is still there um yeah they're uh they're still planning on developing it releasing it 112 i'm sorry 113 112 current route 113 should be out shortly they're pretty committed to about a every other month release cycle

they aggressively uh aggressively uh continue so at least so far there they're definitely continued planning on continuing that so no signs yet of uh that that underlying agent that is underneath uh redline is the same thing used in their commercial products and stuff so lots of you know benefits and and frankly we use it extensively internally as well sir do you have any uh recommendations in those worlds sure yeah great question so what about other hosts besides windows well actually i would say it's easier in those right so in those cases you know all of your x's my mac those have all got syslog in one form or another and so in most cases we can gather a lot of

that really useful information there uh with those and uh and fire it that way that's what i tend to do um beyond that uh there's not really a red line equivalent unfortunately for uh for max and hicks's yet um supposedly we will be doing one at some point but i've been i've been with mandy in almost three years and they've been saying that for two years i'm a little skeptical we'll see that but hopefully eventually hopefully but the process is the same right look for for you know we're looking for one of two things what happened around the time of that event or you know what's going on in the case of the presence is to match it up to our

indicators most cases we've got the data we need to do that host investigation already especially if we're running in themselves good questions other questions sir

so vista up it writes to its own event log right because microsoft overhauled the uh the system then and then with the if you're on 2003 down it puts it into the security level yep any questions other questions all right mike throw out a movie quote for me buddy so we can uh give away our giveaways here looking for mike on the spot i sure weren't him ahead of time while he's thinking that'll be for richard's book anybody got a good use for python for kids first one yours was the first hand i saw there sir since not all of us necessarily have kids that are likely to go but we'll go over that one and then

for the big one what's our quote the quote is uh the rug really tied the room together i saw that hand first sorry was he right yeah i didn't know that one sorry there you go well thank you everybody any other questions another question no no problem

it can use iocs yeah so redline does have the ability to process iocs as well but you can do it with or without isis but you have another tool that's semi-free that does yes yeah yeah there's a separate tool that will also uh search a host for iocs yeah so you know that would be potentially another approach if you're uh using the open ioc format to generate you know indicators around that then you could go look just you know and potentially even automated it's theoretically possible that we have companies out there doing that but i can't name names so did i see another question over here all right thank you for your attention and staying awake after lunch and

another session here